Statistical significance in A/B testing: what 95% actually means

A team ships a test, the dashboard turns green at "95% significance", and someone says the sentence that launches a thousand bad decisions: "there's a 95% chance the variant is better." It sounds right. It is not what the number means, and the gap between the two readings is where teams get hurt.

Significance is worth understanding properly because it is the one statistical concept every stakeholder will encounter, argue about, and act on. The good news: the honest interpretation fits in a paragraph, and it is enough to make better calls than most dashboards encourage.

What 95% significance actually means

Start from a strange place: assume your change did nothing. Both versions convert identically, and any difference you measure is noise — some users buy, some do not, and random assignment sloshes those users unevenly between groups. Under that assumption, you can compute how often pure noise would produce a gap as large as the one you observed. If the answer is "less than 5% of the time", the result is called significant at the 95% level.

Read that carefully: the 5% is a statement about a world where the variant does nothing. It says noise alone rarely fakes a result this big. It does not say there is a 95% chance your variant wins, and it does not say the lift you measured is the lift you will get. Turning "noise rarely does this" into "the variant is probably better by this much" requires information the test alone does not carry — how plausible your hypothesis was to begin with. The number that gets you closer to that fuller picture is the p-value, and it deserves its own careful reading.

The false-positive budget

The practical way to think about a significance threshold is as a false-alarm budget. Run twenty A/B tests on changes that truly do nothing, at 95% significance, and roughly one will come up "significant" anyway. That is not a flaw in your process; it is the deal you signed. The threshold does not eliminate false positives — it prices them.

This budget framing also explains why some testing habits are corrosive. Peeking daily at fixed-horizon numbers spends the budget many times over without telling you. Slicing results into a dozen post-hoc segments hands each segment its own 5% chance to lie. The threshold only means what it says when the rest of the process respects it — one primary metric, pre-named segments, and either a fixed horizon honored or sequential statistics built for continuous looking.

Why 95% is a convention, not a law

There is no theorem that makes 5% the right false-alarm rate. It is a default that hardened into ritual across a century of scientific practice, designed for contexts — published research, irreversible claims — that look nothing like a product team deciding which onboarding flow to keep. Your test is not a journal submission. It is a decision under uncertainty with known costs on each side, and the threshold should follow the costs.

The pattern: the more reversible the change and the cheaper the mistake, the more false positives you can afford to tolerate in exchange for faster answers. A team that runs everything at 99% is buying certainty it rarely needs with traffic it rarely has; a team that runs pricing tests at 90% is gambling with the wrong stakes. Deciding the threshold per decision, before launch, is a five-minute conversation that prevents the after-the-fact version — where the threshold mysteriously lands wherever the result looks best.

What significance cannot tell you

• Whether the effect matters. With enough traffic, a 0.2% lift will reach significance. Significant and worth shipping are different questions; the second one belongs to your minimum detectable effect.
• The size of the effect. The measured lift on a barely-significant result is often inflated — marginal winners tend to shrink after shipping, because tests that got lucky are overrepresented among early crossers.
• Whether a flat result means "no effect". Non-significance means you failed to detect an effect, which at low power says more about your sample than about the change.
• Whether the test itself was sound. A broken split passes no significance filter. A sample ratio mismatch invalidates a significant result just as thoroughly as a flat one.

Making the honest reading the default

You cannot retrain every stakeholder, but you can change what the dashboard says. Report effect sizes with uncertainty ranges, not just a significance verdict. Set the threshold per test, in the test plan, tied to the cost of being wrong. And use statistics that survive the way people actually behave: Trevo runs every experiment on mSPRT sequential statistics, so the significance guarantee holds no matter how often anyone checks the dashboard — the honest reading and the convenient reading finally agree.