Sample ratio mismatch: the silent test-killer

Of all the ways an A/B test can lie to you, sample ratio mismatch is the most insidious, because the lie comes with great production values. The dashboard renders, the p-value computes, the lift looks plausible. Nothing on the surface hints that the two groups being compared stopped being comparable weeks ago. The tell is one humble number most teams never check: whether the groups are the size you asked for.

You configured 50/50. You observe 50,900 users in control and 49,100 in the variant. That is 50.9/49.1 — close enough to shrug at, and far too skewed to be chance at this sample size. A fair coin flipped a hundred thousand times does not miss by nine hundred. Something is systematically removing users from one arm or stuffing them into the other, and whatever that something is, it is not random. Your randomized experiment has quietly stopped being randomized.

Why a small skew is a big deal

The damage is not the imbalance itself — statistics handles unequal group sizes fine. The damage is who went missing. Users do not vanish uniformly: the ones dropped by a slow redirect are on slow connections, the ones eaten by a bot filter came from particular sources, the ones lost to a caching quirk are returning visitors. Whatever selected them out correlates with behavior, so the surviving populations differ in engagement, device mix, or intent — not just in which page they saw. From that point on, your metric comparison measures your change plus the selection effect, in unknown proportions.

What causes SRM

• Redirects on one arm. Classic client-side testing sends the variant through an extra redirect; slow connections abandon during the hop. The variant arm loses precisely its most impatient, slowest-network users.
• Bots filtered asymmetrically. Crawlers hit the control URL cached in indexes, or a bot-detection rule fires differently across variants. Bot traffic lands unevenly, and so does its removal.
• Caching and CDNs. An edge cache serves one variant's assignment to users who should have been split, or caches an old config for a subset of regions. Assignment stops being per-user random.
• Assignment bugs. Hash functions with modulo bias, user IDs that are not uniformly distributed, assignment happening before consent so opt-outs drain one arm, retries that reassign on failure.
• Exposure logging differences. Both arms get assigned fairly, but the variant's exposure event fires from a different code path that drops events under load. The users exist; the records do not — same contamination, different plumbing.
• Mid-test changes. Ramping traffic percentages, fixing a bug in one arm, or excluding a segment after launch — each rewrites who is in the test, and the before/after populations mix incoherently.

How the check works

The standard SRM check is a chi-square goodness-of-fit test, and the intuition is simple: given the split you configured and the total users you observed, how far from the expected counts are the actual counts, and is that distance plausible as coin-flip noise? Random assignment produces small, unbiased wobble that shrinks (proportionally) as the sample grows. A systematic leak produces a skew that persists as the sample grows — which is exactly what the test detects. The check is cheap, runs on counts you already have, and belongs on every experiment automatically, not just when someone gets suspicious.

This is also why SRM monitoring should be continuous rather than a launch-day ritual: a CDN config change in week two can break a split that was clean in week one. Trevo runs the ratio check on every experiment for the experiment's whole life; the implementation details — thresholds, timing, and the false-alarm math — are in our engineering write-up on detecting sample ratio mismatch.

Why you throw away SRM-flagged results

The tempting move, especially when the flagged test shows a win, is to salvage: reweight the groups, exclude a suspicious segment, adjust for the imbalance. Resist it. The problem is not arithmetic — it is that you do not know which users are missing or how they differ, so there is nothing principled to adjust toward. Reweighting corrects the group sizes while leaving the selection bias fully intact. And salvage attempts have a conflict of interest built in: nobody reweights a flagged loser.

1. Stop treating the result as evidence. Wins and losses alike — the comparison is contaminated in an unknown direction.
2. Find the mechanism. Segment the ratio by browser, device, traffic source, geography, and time. The leak is usually concentrated: one browser, one campaign's bot traffic, one region behind one CDN pop.
3. Fix and verify with an [A/A test](/blog/what-is-an-aa-test) — the cheapest way to prove the split is clean again without risking another real test on broken plumbing.
4. Rerun the experiment fresh. New assignment, new data. Mixing pre-fix and post-fix data smuggles the contamination back in.

It stings to discard six weeks of data. But the alternative is shipping decisions built on a comparison of two different populations — and the habit of "adjusting" flagged tests corrodes the credibility of every result your program produces. SRM discipline is a cheap price for a testing program people can trust; it is also mistake number two on our list of A/B testing mistakes for a reason.