A/B test sample size: how to size a test without a statistics degree

The most useful moment in any A/B test happens before you write a line of variant code: the moment you ask how many users this test needs. Teams that skip the question build tests that were doomed on day zero — tests that could only ever end flat or falsely positive, because the change they measured was too small for their traffic to detect.

You do not need to derive the formula. Calculators exist and every serious platform builds sizing in. What you need is the intuition for the three dials that drive the answer, because they are product decisions wearing statistical costumes.

What actually drives sample size

Dial 1: the baseline rate

A change to a step that converts at 40% is much easier to measure than a change to a step that converts at 2%. At low baselines, conversions are rare events, and rare events are noisy: you need many more visitors before the signal separates from the static. This is why testing upstream — on a high-baseline step like a form completion — often beats testing the final purchase directly.

Dial 2: the minimum detectable effect

The MDE is the smallest lift you would care to detect, and it dominates everything. Sample size scales roughly with the inverse square of the effect: halve the effect you are hunting and you need about four times the users. Hunting a 20% relative lift is a weekend of traffic for some teams; hunting a 2% lift on the same page is a quarter. Choosing the MDE honestly is its own topic — we cover it in minimum detectable effect — but the short version is: pick the smallest effect that would still justify shipping, not the smallest effect you can imagine.

Dial 3: power

Power is the probability your test detects a real effect of the size you specified. The convention is 80%: if the true lift equals your MDE, an 80%-powered test finds it four times out of five — and misses it one time in five. Raising power costs users; lowering it turns your test into a coin flip that occasionally works. The fourth dial, significance level, is usually left at 95% — see what 95% actually means for why that is convention rather than law.

The bars are the story: the relationship is brutally nonlinear. This is why "let's just test it and see" fails as a strategy on modest traffic. Doubling your ambition for the effect cuts the required sample to a quarter — which means bolder changes are not just braver, they are cheaper to measure.

A sizing workflow that takes five minutes

1. Pull the baseline. From your analytics: what fraction of users who reach this step convert today?
2. Name the MDE from business value. What is the smallest lift that would justify the engineering cost and the risk of change? Write the number down before touching a calculator.
3. Run any standard sample-size calculator. Baseline, MDE, 80% power, 95% significance. Get users-per-variant.
4. Divide by your real traffic. Users per variant, doubled, divided by eligible weekly visitors = weeks of runtime. Not sitewide visitors — visitors who actually reach the tested step.
5. Confront the answer. Under ~2 weeks: run it. 2–8 weeks: run it if the decision matters. Beyond that: redesign the test, because you will not have the patience and neither will your roadmap.

What to do when the required sample is impossible

Sooner or later the calculator returns a number your traffic cannot pay. That is not a failed test — it is a successful five-minute experiment that saved you six wasted weeks. You have real options:

• Test a bigger swing. Redesign the whole page instead of the headline. Larger plausible effects need far fewer users, and the sample-size math rewards boldness quadratically.
• Move the metric upstream. Measure signup-form completion instead of paid conversion. Higher baseline, more events, faster answer — at the cost of a weaker link to revenue.
• Pool surface area. Apply the same change across every product page and test it as one experiment, rather than sizing a test on a single page's trickle.
• Use sequential statistics. Sequential testing will not rescue an underpowered test, but it ends clear results early, so you only pay the full runtime when the race is genuinely close.
• Skip the test. Some decisions at low traffic are better made with judgment, qualitative research, and a rollback plan. The full playbook is in A/B testing with low traffic.

Where this fits in the loop

Sizing is the gatekeeper step, and it is also the step most often skipped, because it happens before the fun part. This is one of the places automation earns its keep: when Trevo proposes an experiment from your funnel data, feasibility at your actual traffic is part of the argument — it will not pitch you a test your users cannot pay for. If you size by hand, the discipline is the same: no variant code until the calculator and the calendar have both said yes.