How long should an A/B test run?

The honest answer to "how long should my test run?" is a calculation, not a constant. But the calculation has only three inputs, and understanding them will tell you more than any generic benchmark: how much traffic hits the tested surface, your baseline conversion rate, and the smallest lift you would actually act on.

The floor: one full business cycle

Whatever the math says, never run a test for less than one complete cycle of your business — for most products, one week. Weekday visitors behave differently from weekend visitors, payday weeks differ from the week before, and B2B traffic on a Tuesday is a different species from B2B traffic on a Saturday. A test that runs Monday to Thursday has only sampled the Monday-to-Thursday version of your audience.

The ceiling: what sample size demands

The real driver of duration is sample size, and the driver of sample size is the size of the effect you want to detect. The relationship is brutally nonlinear: detecting a lift half as big needs roughly four times the sample. That is why "we will just run it longer and detect anything" is not a strategy — halving your minimum detectable effect again quadruples the requirement again.

To turn sample into days, divide by the traffic that actually reaches the tested surface — not your total traffic. A checkout test only sees visitors who reach checkout. If 2,000 people a day hit the surface and you need 78,000 per variant, that test wants roughly 78 days at a 50/50 split. At that point the correct move is usually not to run an 11-week test; it is to test a bolder change whose effect is big enough to detect in two or three.

Why "run until significant" is the one answer that is always wrong

Fixed-horizon statistics — the classic t-test — assume you pick the sample size in advance and evaluate once, at the end. If instead you check the dashboard every morning and stop the first time p dips under 0.05, you are giving random noise dozens of chances to cross the line. Simulations of this behavior routinely show false positive rates several times the advertised 5%. The variant did not win; you caught it on a good day.

There are two honest ways out. The first is discipline: compute the horizon up front, run to it, look once. The second is to use sequential statistics — methods like mSPRT that are built to be checked continuously and remain valid at every look. Sequential tests call clear winners early and let ambiguous results keep collecting evidence, which on average ends tests sooner without the false-positive tax of peeking.

When stopping early is legitimate

• A guardrail is being damaged. If the variant is tanking revenue, crashing pages, or spiking latency, stop immediately. Protecting the business outranks statistical purity.
• A sequential boundary was crossed. That is the method working as designed — an early stop under mSPRT is a valid stop, not a peek.
• The setup was broken. A sample-ratio mismatch or a bug in one variant means the data is unusable. Stop, fix, restart — do not "let it recover".

And when running longer is a trap

Tests that drag past four to six weeks develop their own problems: cookies churn and users switch groups, seasonality drifts under the test, the codebase accumulates conflicting changes, and the team stops believing the result will ever arrive. A test that cannot reach its sample in about a month is usually the wrong test — too small an effect, on too thin a slice of traffic. Redesign the experiment rather than extending it indefinitely.

This whole calculation — surface traffic, baseline, plausible effect, resulting horizon — is mechanical, which is exactly why Trevo automates it. Every experiment Trevo proposes ships with its expected duration up front, runs on sequential statistics so an early call is a safe call, and auto-pauses if a guardrail degrades. You review the pull request; the arithmetic minds itself.