Minimum detectable effect: the number that decides your test before it starts

Every A/B test makes a promise before it collects a single visitor: "I can see effects this big or bigger; anything smaller is invisible to me." That promise is the minimum detectable effect, and it is set — deliberately or by accident — the moment you pick a runtime and a traffic level. Most failed testing programs did not fail at statistics. They failed here, by running tests whose promises their traffic could never keep.

The MDE is worth treating as the first-class design decision it is, because it is where statistics and business value actually meet. Everything else in test design is technique. This is the part that decides whether the test can tell you anything at all.

What the MDE actually is

Formally, the MDE is the smallest true effect your test would detect with your chosen power — conventionally 80% — at your chosen significance level. Informally: it is your test's resolution. A microscope that resolves down to one micron simply cannot show you anything smaller, no matter how carefully you look. A test with a 10% MDE cannot see a 4% lift. The 4% lift may exist. Your test will shrug at it.

The MDE trades off directly against sample size, and the exchange rate is steep: detecting an effect half as large costs roughly four times the users. That inverse-square relationship — covered mechanically in how to size a test — is why the MDE cannot be wished downward. It has to be paid for in traffic, and the bill grows quadratically as your ambitions shrink.

How to choose an MDE from business value

The wrong way is common: pick the effect you hope for, or worse, the effect that makes the calculator return a comfortable runtime. The right way runs the logic in the other direction — from the decision, backwards.

1. Price the ship. What does shipping this variant cost? Engineering time, added complexity, migration risk, design debt. Rough numbers are fine.
2. Find the break-even lift. How large a lift on this metric pays for that cost within a horizon you care about? A change to a high-revenue checkout might break even at a 2% lift; a marketing page tweak might need 15% to be worth its maintenance.
3. Set the MDE at break-even, or above. Any true effect below break-even is a lift you would not ship for. Spending traffic to detect it is spending traffic on an answer that cannot change your decision.
4. Check affordability. Run the sample-size math at that MDE against your real traffic. If the runtime is absurd, do not quietly lower your power or raise your MDE past break-even — change the test instead.

The fiction of chasing 2% lifts on low traffic

Here is the trap this framing exposes. A team with modest traffic reads that mature experimentation programs at giant companies grind out small single-digit wins, and decides to hunt the same game. But those programs can afford 2% MDEs because millions of users pay the bill. At a few thousand conversions a month, a 2% MDE costs years of runtime. The team runs the test for three weeks anyway.

What happens next is worse than nothing. The test ends flat — as designed, since it never had the resolution — and the team concludes the idea failed. Or noise produces a fluke crossing, and because underpowered tests exaggerate, the dashboard shows a lift far larger than reality; the team ships, celebrates, and the metric never moves in production. Both outcomes feel like learning. Neither is. The test was fiction from the day its MDE outran its traffic, and every week it ran was traffic taken from a test that could have worked.

The honest response to an unaffordable MDE is to change what you test, not how hard you squint: swing bigger so the plausible effect clears the resolution you can afford, move the metric upstream where events are denser, or decide without a test. That full menu is in A/B testing with low traffic.

MDE vs the observed effect

A confusion worth killing: the MDE is not a prediction of the result, and the observed effect is not a check against it. The MDE is a property of the design — the resolution you paid for. The observed effect is what the data then showed, and it comes with uncertainty of its own, which is why it is best read as a confidence interval rather than a point estimate (more on that in p-values for product teams).

• Observed effect well above MDE, clearly significant: the happy case. Ship with reasonable trust in the direction, mild skepticism about the exact size.
• Observed effect below MDE, not significant: the test worked as designed — it told you any effect is smaller than what you cared to detect. Keep the control; that was the deal.
• Observed effect barely significant near the resolution limit: treat the size with suspicion. Marginal detections systematically exaggerate, because the flukes that cross the line are the lucky ones.

Make the contract explicit

Write the MDE into the test plan next to the metric, and let it kill bad tests early — that is its job. This is also a place where automation keeps the contract honest: when Trevo proposes an experiment from your funnel, the proposal accounts for what your traffic can actually detect, so tests whose MDE outruns your users do not get pitched in the first place. However you run the loop, the rule is the same: no test launches without knowing what it can see and what it would ship for.