11 A/B testing mistakes that quietly ruin your results
The dirty secret of experimentation is that the statistics are rarely what kills a test. What kills tests is process: a peek at the wrong moment, a broken assignment nobody checked, a metric chosen to flatter the variant. The result looks like science — there is a p-value on it, after all — but the conclusion is fiction.
These are the eleven mistakes we see most, each with its fix. None of them requires a statistics degree. Most require nothing more than a checklist and the willingness to throw away a result you liked.
The statistical mistakes
1. Peeking at fixed-horizon stats and stopping early
You check the dashboard daily and stop the first day the test hits 95% significance. With classic statistics, that inflates your false-positive rate badly — noise gets a fresh chance to cross the line every day. Fix: either commit to a sample size and genuinely wait, or use sequential statistics that stay valid at every peek. Do not split the difference.
2. Ignoring sample ratio mismatch
You split 50/50 but the groups arrive 52/48. That small imbalance is a smoke alarm: something — a redirect, a bot filter, a caching layer — is assigning users non-randomly, and every downstream number is suspect. Fix: check the actual ratio against the intended one on every test, and treat a sample ratio mismatch as an invalidated experiment, not a footnote.
3. Calling a one-day test
Monday's visitors are not Saturday's visitors. A test that runs less than one full business cycle measures your traffic mix, not your change. Fix: run at least one full week regardless of how the numbers look, and let how long to run an A/B test guide the ceiling.
4. Stopping on a weekend spike
The mirror image of the one-day test: the result crosses the line on Sunday, powered by weekend traffic that behaves nothing like your weekday core, and someone ships it Monday morning. Fix: end tests on whole-week boundaries so every day of the week is represented equally in the final data.
5. Slicing segments after the fact
The overall result is flat, so someone digs until they find a segment where the variant "won" — mobile users in one country, say. Slice enough segments and one will always look significant by chance. Fix: name the segments you care about before launch. Post-hoc segment discoveries are hypotheses for the next test, never conclusions from this one.
The design mistakes
6. Testing trivia
Button-color tests on a low-traffic site are astrology with extra steps: the plausible effect is far below your minimum detectable effect, so the test can only end flat or falsely positive. Fix: size the test before you build it. If the required sample is impossible, test something bolder or do not test.
7. Running without guardrails
The variant lifted email signups 12% — and quietly slowed the page enough to hurt purchases. A single success metric cannot see collateral damage. Fix: pair every test with two or three guardrail metrics — revenue, retention, performance — that the variant must not harm.
8. Randomizing at the wrong unit
You randomize by pageview, so the same person sees the control at breakfast and the variant at lunch. Their conversion lands in one bucket, their exposure in both, and the comparison dissolves. Fix: randomize at the level you measure — almost always the user — and keep assignments sticky across sessions and devices where you can.
The follow-through mistakes
9. Reusing the same users across overlapping tests
Six tests run at once on the same flow, and users carry effects from one into another. Mild overlap is usually survivable when assignments are independent, but stacking tests on the same page or the same decision point invites interactions that no single test's analysis can see. Fix: keep concurrent tests on separate parts of the funnel, or explicitly account for overlap when tests must share territory.
10. Shipping flat winners
The test was inconclusive, but the team spent three weeks on the variant, so it ships anyway. Now the codebase changed for no measured benefit, and the sunk cost got laundered into a "win". Fix: decide in advance what a flat result means. Usually it means keep the control — it is already built, already cached, already familiar.
11. Never cleaning up
The test concluded in March. In November, both variants still live in the code behind a flag nobody remembers the meaning of. Multiply by every test ever run and you get a codebase where nobody can tell what is actually live. Fix: treat cleanup as part of the experiment, not an afterthought — the test is not done until the losing branch is deleted. This is common enough that Trevo opens the cleanup PR automatically when an experiment concludes; if you run tests by hand, put the cleanup ticket in the sprint the day the test starts.
What a clean loop looks like
Size the test before building. One primary metric, guardrails beside it. Randomize by user. Check the split ratio early. Run whole weeks. Read pre-named segments only. Keep the control on a flat result. Delete the loser. None of this is glamorous, which is exactly why automating the loop pays: Trevo runs this checklist by construction — sequential stats safe to watch, SRM checks built in, guardrail auto-pause, and a cleanup PR when the test concludes — so the discipline does not depend on anyone's memory.
Frequently asked questions
What is the most common A/B testing mistake?
Peeking: checking results daily with fixed-horizon statistics and stopping the first time the dashboard shows significance. It feels diligent but badly inflates false positives, because noise gets many chances to cross the line. Fix it by committing to a sample size in advance or by using sequential statistics designed for continuous monitoring.
Why should I throw away a test with sample ratio mismatch?
Because SRM means assignment itself was broken — bots, redirects, or caching pushed a non-random mix of users into one group. Once randomization fails, the two groups differ in ways beyond your change, so the comparison no longer isolates the variant. No amount of downstream statistical adjustment reliably repairs a corrupted split.
Should I ship a variant if the test result is flat?
Usually not. A flat result means you could not detect a difference, so shipping the variant buys you no measured benefit while still paying the costs of change: new code paths, cache misses, retraining users. Keep the control unless the variant is clearly cheaper to maintain or strategically necessary for other reasons.
Can I run multiple A/B tests at the same time?
Yes, if they touch different parts of the product and assignments are independent — overlap then mostly adds noise, not bias. The danger is stacking tests on the same page or decision point, where variants interact. Keep concurrent tests on separate funnel stages, or run them sequentially when they compete for the same click.