Sequential testing, explained without the math degree

Here is a thing every team does and almost no statistics textbook allows: check the experiment dashboard every morning. The classic t-test that powers most "significance" numbers assumes you chose your sample size in advance, waited, and looked at the result exactly once. Look daily and stop the first time the number crosses 95%, and your real false-positive rate is not 5% — it can climb to several times that, because noise gets a fresh chance to cross the line every single day.

That is the peeking problem, and it is not a moral failing. Checking on a running experiment is the correct instinct — a test that is tanking revenue should be stopped today, not at week six. The failing is in the tool: fixed-horizon statistics were designed for agricultural field trials where you harvest once, not for dashboards that update hourly. Sequential testing is the family of methods designed for how teams actually behave.

What goes wrong when you peek

Picture an A/A test — both groups see the identical page, so any measured "difference" is pure noise. Over a long run the noise wanders: some days A looks ahead, some days B does. If you only check once at a preplanned end date, there is a 5% chance the noise happens to look significant at that moment. But if you check every day for eight weeks and stop the first time it looks significant, you are asking a different question: did the noise ever wander across the line? Over enough peeks, it usually does.

How sequential testing fixes it

Sequential methods change the accounting. Instead of asking "does the difference look big right now?", they track how the evidence accumulates and set decision boundaries that already price in the fact that you are watching continuously. The boundaries start wide — early on, even a large gap between A and B is treated with suspicion, because small samples produce wild swings. As data accumulates, the boundaries tighten. The test ends when the observed effect crosses a boundary: strong enough evidence to call, guaranteed to keep the overall false-positive rate at your chosen level no matter how often you looked.

One popular way to build those boundaries is the mixture sequential probability ratio test, mSPRT. Strip away the name and the idea is a running evidence meter: after every new visitor, it updates the ratio of "how well does the data fit there is a real effect" against "how well does it fit there is no effect". When that ratio gets extreme enough, the test concludes. The "mixture" part means it does not have to guess the effect size in advance — it averages over a range of plausible effects, so it stays sensitive whether the true lift is 3% or 30%.

The user-facing version of this is the always-valid p-value: a number you can read at any moment, as many times as you like, that never overstates the evidence. When an always-valid p-value drops below 0.05, you can stop and act on it — today, tomorrow, or three weeks from now — and the guarantee holds. That single property dissolves the peeking problem entirely. (For what a p-value does and does not mean in the first place, see p-values for product teams.)

When does sequential beat fixed-horizon?

The pattern in that table is the honest trade-off. Sequential tests buy you the right to monitor continuously, and they pay for that right with wider boundaries: a genuinely close result takes longer to call sequentially than it would with a perfectly disciplined fixed-horizon test. In exchange, clear results end early. Since large effects are exactly the ones worth acting on fast — a variant lifting signups 25% should ship this week, and one breaking checkout should die this afternoon — the trade tends to favor sequential for product teams.

The discipline question matters more than the math. A fixed-horizon test is only valid if nobody acts on interim results, ever — no "it's significant, let's just ship it", no quiet early stops. If you have ever watched a stakeholder read a dashboard mid-test, you know how realistic that is. Sequential testing is the version of rigor that survives contact with actual humans. It pairs naturally with guardrail metrics, because a guardrail is only useful if you are allowed to act the moment it degrades.

What this looks like in practice

• You launch the test and watch it. No sample-size ritual up front deciding the exact day you may look, though a rough sizing pass still tells you whether the test is feasible at all — see how to size a test.
• Every look is a legal look. The always-valid p-value or evidence boundary means Tuesday's glance and Friday's glance carry the same guarantee.
• The test stops itself. When the evidence crosses a boundary, the decision is made: ship, kill, or — if boundaries are never crossed within your patience — accept that any effect is too small to matter at your traffic.
• Losers die fast. The largest practical benefit: harmful variants get caught in days, not discovered in a post-mortem.

This is also the approach Trevo uses. Every experiment it runs is monitored with mSPRT sequential statistics, so the dashboard is safe to watch continuously, and guardrail damage triggers an automatic pause instead of waiting for a human to notice. We wrote up the implementation details separately in why Trevo uses mSPRT; this post is the intuition, that one is the engineering.