A/A tests: the sanity check most teams skip

Before you trust a scale, you weigh something whose weight you know. An A/A test is that move for your experimentation stack: split traffic 50/50 between two groups, show both groups the identical experience, and measure everything you would measure in a real test. The correct result is boring — no significant difference, groups the same size, metrics flowing evenly. Anything interesting is a bug.

Teams skip this because it feels like running an experiment that cannot win. That framing is backwards. Every A/B result you will ever act on inherits its credibility from the machinery the A/A test checks. Skipping it means calibrating nothing and trusting everything.

What an A/A test actually catches

Broken randomization

Assignment is supposed to be a fair coin flip per user, sticky across visits. In practice, hashing bugs, ID collisions, and assignment-before-consent flows all break the coin. If your A/A groups differ systematically — one group skews mobile, or returning, or from one region — your randomizer is sorting users, not splitting them, and every A/B test on top of it compares different populations rather than different designs.

Sample ratio mismatch

You asked for 50/50 and got 52/48. On real traffic that gap is very unlikely by chance, and it means users are leaking out of one arm — a redirect that drops slow connections, a bot filter that fires asymmetrically, a caching layer serving one variant stale. An A/A test surfaces sample ratio mismatch with no variant code to blame, which isolates the fault to your infrastructure.

Instrumentation bugs

The subtlest catch: both groups behave identically, but the measurements differ. Exposure events fire twice in one arm, a conversion event races page navigation and drops more often in one code path, timezone handling shifts events across day boundaries. An A/A test is the only test where measurement differences cannot be explained by user behavior — so whatever difference you see is your pipeline talking.

How to run one

1. Use the full production path. Real assignment code, real exposure logging, real metric pipeline. An A/A test of a simplified setup validates the simplified setup.
2. Run it like a real test. Same traffic allocation, same runtime you would give an A/B test — at least a full week, so weekday and weekend traffic both flow through.
3. Check the ratio early. Sample ratio problems show up within days; there is no need to wait weeks to catch a 55/45 split.
4. Read every metric you rely on. Not just the headline conversion — guardrails, engagement, performance. Each metric is a separate piece of pipeline that can be separately broken.
5. Expect boredom. One metric in twenty may flicker "significant" at the 95% level by pure chance. That is the false-positive budget working as designed, not a failure.

How often should you run A/A tests?

• When you adopt or build a testing tool. Before the first real experiment. This is non-negotiable — you are about to route product decisions through this machinery.
• After big changes to the stack. New CDN or caching layer, a rewrite of assignment logic, an analytics migration, a consent-flow change. Each of these has broken real teams' splits.
• Periodically as a heartbeat. A low-traffic-cost A/A running quietly in the background, or one scheduled each quarter, catches the slow rot — SDK updates, bot evolution, config drift.
• When results start smelling wrong. If several A/B tests in a row produce implausible swings, an A/A test is the fastest way to establish whether the instrument or the hypotheses are at fault.

Reading a failed A/A test

A failure is a gift with bad wrapping. Work from the outside in. First check the sample ratio: if the split is off, the fault is upstream of metrics entirely — look at redirects, bots, caching, and assignment timing. If the ratio is clean but a metric differs, the fault is in measurement: diff the event payloads from each arm side by side and look for double-fires, drops, and ordering races. If everything replicates on a re-run, stop trusting recent A/B results until the cause is found — any bias strong enough to fail an A/A test has been quietly shading your real experiments too.

And resist the temptation to shrug a failure away because the difference is small. A persistent 2% measurement bias is invisible inside any single A/B test and large enough to flip close calls — which is precisely the kind of call you run tests to make. This is why platforms increasingly automate the checks an A/A test performs: Trevo, for instance, monitors the assignment ratio on every experiment continuously, so the most common A/A failure mode — SRM — is caught during real tests, not just during scheduled sanity checks. A full list of the process failures an A/A test protects you from is in 11 A/B testing mistakes.

The cost-benefit that makes this easy

An A/A test costs you nothing but calendar time on a surface you were not testing anyway — no design work, no variant code, no user-facing risk. In exchange it answers the only question that ranks above every hypothesis you have: can this system tell the truth? Run it before your first real test, run it after big changes, and let automation watch the ratios in between.