One decision metric per test (and as many guardrails as you like)
A test concludes. Signups: flat. Activation: flat. Click-through on the new banner: up, significant. Time on page: up, significant. The readout deck leads with the two green numbers, the team calls it a win, and the variant ships. Three months later nobody can explain why revenue did not move.
Nothing was faked. The team simply gave itself six ways to win and needed only one. That is not measurement; that is buying six lottery tickets and framing the one that paid out.
Why multiple success metrics multiply false wins
Statistical significance is a false-positive budget. One pre-declared comparison at the conventional threshold means about a one-in-twenty chance of calling noise a win — a risk you accepted on purpose. Add a second success metric and the test now wins if either clears the bar: two draws from the noise lottery. Six metrics, six draws. The chance that something looks significant by luck alone climbs fast, and dashboards do not annotate which green badge is the real one.
The deeper failure is organizational. When every metric is a potential success criterion, the decision gets made after the data arrives — whichever number moved becomes the story of what the test was "really about". The same team that ships on a click-through lift would never have written "we will ship if clicks go up even if signups stay flat" before launch. Writing the rule down first is the entire game. This is the same discipline that makes segment analysis honest: comparisons declared in advance mean something; comparisons found afterwards are stories.
The metric hierarchy that works
Primary: the one that decides
One metric, chosen because it is the closest measurable thing to the value the change is supposed to create. It should be sensitive enough to move within your traffic — "checkout started" often beats "revenue" as a decision metric because it is upstream, higher-volume, and far less noisy — while still meaning something. If your hypothesis cannot name the single metric it should move, the hypothesis is not finished.
Secondary: the ones that explain
Two or three metrics that tell you why the primary moved or did not — step-level conversion inside the funnel, engagement with the changed element. They shape the narrative and the follow-up test. They do not get a vote on shipping.
Guardrails: the ones that protect
Here is the asymmetry that surprises people: while success metrics must be few, guardrails can be plentiful. A guardrail is not another way to win — it is a tripwire that says "this variant is doing damage somewhere you were not looking": error rates, page performance, refunds, unsubscribes, downstream retention. False positives on a tripwire cost you an investigation; false positives on a success metric cost you a bad ship. The asymmetric cost is why the budget math differs. Our guide to guardrail metrics covers choosing them.
| Role | How many | Can it ship the variant? | Can it kill the variant? |
|---|---|---|---|
| Primary | Exactly 1 | Yes — alone | Yes |
| Secondary | 2–3 | No | No — informs only |
| Guardrail | As many as you monitor | No | Yes — damage stops the test |
Pre-registering the decision rule
Before launch, the experiment doc should contain one sentence a stranger could execute: "Ship B if the primary metric shows a credible lift under the stats plan; kill if flat or negative when the test concludes; pause immediately if any guardrail degrades." Write it, date it, and make the readout open with it.
- The rule binds the readout. If clicks went up but the primary stayed flat, the test lost. The click lift is an interesting secondary observation for the next hypothesis.
- "Flat" is a result. A test that fails to move its primary taught you the change does not matter at detectable size. Ship whichever version is simpler and move on — that is a win for the roadmap even when it is a loss for the variant.
- Exceptions get written down too. Overriding the rule is occasionally right — but the override should be a documented judgment call, not a quiet reframing of what success meant.
Making the discipline the default
None of this requires new math — a plain understanding of p-values plus a doc template gets you most of the way. The hard part is that the discipline lives in habits, and habits erode the week the roadmap gets loud.
This is where tool design can carry the weight. Trevo's experiment structure mirrors the hierarchy directly: every experiment it proposes names one primary decision metric argued from your funnel, monitored with mSPRT sequential statistics that stay valid however often you look, and any number of guardrails that auto-pause the test on damage. The decision rule is in the experiment from the start — not reconstructed in the readout meeting after everyone has seen which numbers are green.
Frequently asked questions
How many metrics should an A/B test have?
One primary metric that alone decides ship-or-kill, two or three secondary metrics that explain the result, and as many guardrail metrics as you care to monitor for damage. The cap applies to success criteria, not to observation — every additional way to win multiplies your chances of a false positive.
What is the difference between a primary metric and a guardrail metric?
The primary metric is the one the variant is trying to improve; a credible lift there ships the change. Guardrails are metrics the variant must not harm — errors, latency, refunds, retention. Guardrails cannot make a variant win; they can only stop or kill it, which is why you can afford many of them.
What happens if secondary metrics improve but the primary is flat?
The test lost. A flat primary means the change did not do the job it was designed for; the secondary movement is a hypothesis for a future experiment, not a reason to ship this one. Deciding from whichever metric happened to move is post-hoc reasoning and inflates false wins.
Why pre-register an A/B test decision rule?
Because humans rationalize. After seeing data, any green number can be framed as what the test was about. Writing the ship/kill/pause rule before launch — naming the primary metric and the evidence bar — makes the decision mechanical and keeps twenty dashboard metrics from becoming twenty chances to declare victory.