A/B/n testing: how many variants is too many
The design review produced four decent hero layouts, and the tempting move is obvious: test them all at once and let the data pick. Why run three sequential tests when one A/B/C/D/E test answers everything in a single pass?
Because that single pass costs more than the three sequential tests — in traffic, in time-to-decision, and in statistical care. Sometimes it is still the right call. But "test everything at once" has two hidden invoices, and most teams only find out about them when the test refuses to conclude.
Invoice one: the traffic split
A two-arm test gives each arm half your traffic. Five arms give each 20%. Since the sample you need is per arm, not per test, the five-arm test takes roughly two and a half times as long to gather the same evidence per comparison — before any statistical correction. Traffic that arrives in two weeks for an A/B test arrives in five or six for the same page as A/B/C/D/E.
Invoice two: more arms, more chances to be fooled
Four variants against control is four comparisons. As with multiple success metrics, every extra comparison is another ticket in the noise lottery: the chance that some arm beats control by luck grows with each arm you add. Honest multi-arm testing corrects for this — Bonferroni-style adjustments or their gentler cousins — and every correction raises the evidence bar, which means still more traffic per arm. The two invoices compound: thinner slices, each needing a stricter standard.
When A/B/n beats iterating
None of this makes multi-arm testing wrong — it makes it a purchase. Here is when it is worth buying:
- The variants are genuinely different directions, not shades of one idea. Three distinct value propositions is an A/B/n question; three phrasings of the same proposition is a copy iteration question.
- You have traffic to burn. High-volume pages absorb thin splits. If a two-arm test on this surface concludes in days, a four-arm test concluding in two weeks is fine.
- Sequencing would cost you the season. When results must inform a launch and running three tests back-to-back misses the window, one wider test is the honest compromise.
- You need the comparison itself, not just a winner — for instance, showing stakeholders that the bold direction beats the safe one before investing in it.
And when traffic is modest, iterate instead: rank your ideas, test the strongest against control, fold what you learn into the next challenger. Sequential A/B keeps every test fully powered, gives you a learning step between rounds, and — because each test can conclude and be cleaned up before the next begins — keeps the codebase carrying one experiment at a time instead of five.
Killing losers early: how sequential stats change the math
The strongest argument for A/B/n in a modern stack is that the thin-slice penalty is front-loaded, and sequential statistics can claw it back. Under fixed-horizon testing you committed to feeding all five arms until the predetermined sample — even the arm that was clearly hopeless by day four. Always-valid methods like mSPRT let you monitor continuously and act the moment evidence is sufficient, so a clearly losing arm can be dropped early and its traffic redistributed to the remaining contenders.
- Launch wide. Split evenly across control and all variants, one primary metric, guardrails on.
- Watch with always-valid statistics. Sequential methods stay honest under continuous monitoring — no peeking penalty.
- Prune the hopeless. When an arm's evidence says it cannot win, kill it and let the survivors split its traffic.
- Finish as a narrow race. The test often ends as the two-arm comparison it should have been, having spent only days — not the full run — on the duds.
This launch-wide-prune-fast pattern is how Trevo runs multi-variant experiments: each arm ships as reviewable PR code, mSPRT monitors every arm continuously, guardrail damage auto-pauses, and concluded arms get cleanup PRs so the losers leave the codebase as decisively as they left the test.
A quick decision table
| Situation | Recommendation |
|---|---|
| High traffic, 3+ distinct directions | A/B/n with corrections, prune early |
| High traffic, variations on one idea | A/B now, iterate on the winner |
| Modest traffic, any number of ideas | A/B your best idea, iterate |
| Low traffic | Bigger swings, fewer arms — see below |
| Deadline forces one shot | A/B/n, accept the longer read |
If you are at the low end, the variant question is secondary to a harder constraint — testing with low traffic is its own discipline, and the short answer is: two arms, big difference between them.
Frequently asked questions
How many variants should an A/B test have?
As few as answer your actual question — two arms for most teams, most of the time. Add arms only when the options are genuinely distinct directions and your traffic can feed each slice to a conclusion in reasonable time. Every extra arm thins the split and adds a comparison to correct for.
What is A/B/n testing?
A/B/n testing compares a control against two or more variants simultaneously — A/B/C, A/B/C/D, and so on. Each arm receives a share of traffic and is compared to control. It differs from multivariate testing, which varies multiple elements independently to measure interactions between them rather than comparing whole alternatives.
Does adding variants to a test require more traffic?
Yes, roughly linearly and then some. Required sample size is per arm, so five arms need about two and a half times a two-arm test's total traffic for equal evidence per comparison — plus more on top, because correcting for multiple comparisons raises the evidence bar each arm must clear.
Can you remove a losing variant mid-test?
With sequential (always-valid) statistics, yes: they remain honest under continuous monitoring, so you can drop an arm once its evidence shows it cannot win and give its traffic to the survivors. Under fixed-horizon statistics, mid-test changes undermine the analysis — the sample size was committed in advance.