P-values for product teams: a working interpretation

The p-value might be the most acted-upon number in product development that almost nobody can define. It sits on every experiment dashboard, gets rounded to a verdict — significant or not — and quietly shapes what ships. Yet ask a room what p = 0.03 means and you will hear three confident, mutually incompatible answers.

This post is the working interpretation: precise enough to keep you out of trouble, short enough to actually retain, aimed at people whose job is shipping product rather than publishing papers.

What a p-value actually measures

Every p-value starts with a thought experiment. Imagine your change did nothing at all — the variant converts exactly like the control, and any difference between the groups is random noise from which users happened to land where. In that imaginary world, results still vary: run the same null experiment a thousand times and you get a thousand slightly different gaps. The p-value locates your actual result in that distribution. It is the fraction of no-effect worlds that would produce a result at least as extreme as yours.

So p = 0.03 says: in a world where this change does nothing, only 3% of experiments would show a gap this large. Your data is awkward for the "it did nothing" story. That is genuine evidence — surprise is informative. But notice how conditional the statement is: it describes the data assuming no effect. It says nothing directly about the world where your variant works, and it does not tell you which world you are in.

The misreadings that cause real damage

• "p = 0.03 means a 97% chance the variant is better." The classic inversion. The p-value is computed assuming no effect; the probability the variant is better also depends on how plausible the change was before you tested it. A surprising result for an implausible hypothesis is still often a fluke.
• "Smaller p-value means bigger effect." No — it means more confident effect. A huge sample can produce p = 0.001 on a lift too small to justify the code it ships with. The p-value confounds effect size and sample size into one number; you have to unconfound them by reading the effect estimate.
• "p = 0.2 means the change did nothing." It means you failed to detect an effect, which on a small sample says more about your sample than about the change. Absence of evidence is weak evidence of absence — how weak depends on your power.
• "p = 0.06 failed, p = 0.04 succeeded." Those two experiments observed nearly identical evidence. The threshold is a decision convenience, not a cliff in reality; treating it as sacred leads teams to relaunch near-misses until one clears the bar, which is p-hacking with extra steps.

Why peeking breaks p-values

The standard p-value carries an expiry condition few dashboards mention: it is valid for one look, at a sample size fixed in advance. The thought experiment underneath assumes each no-effect world runs to the same horizon and gets read once. Check daily and stop the first time p dips under 0.05, and you have changed the procedure to "did the noise ever cross the line?" — which it does far more than 5% of the time. The same data, read impatiently, supports a claim the math no longer backs.

The fix is not more willpower. Always-valid p-values — built on sequential methods like mSPRT — are constructed so the guarantee survives continuous monitoring: read the number every hour if you like, act whenever it crosses your threshold, and the false-positive rate holds. How that works is the subject of sequential testing, explained; it is also the statistics Trevo runs on every experiment, precisely because "nobody looks until the horizon" is not how any real team behaves.

The more useful lens: confidence intervals

Here is the quiet problem with p-values for product work: they answer "is there probably some effect?" when your actual question is "how big is the effect, and is it worth shipping?" The confidence interval answers the real question. Instead of one number about surprise, you get a range for the effect itself: "the lift is somewhere around +2% to +11%" carries the size, the direction, and the uncertainty in a single readable object.

Reading one is mechanical. If the whole interval sits above zero, you have the same information significance gives you — plus the magnitude. If the interval spans zero widely, the test is uninformative and needed more data, a problem you can head off by sizing the test up front. And the most valuable case: an interval entirely above zero but entirely below the minimum effect worth shipping tells you the change works and is still not worth the complexity. A p-value alone would have called that one a win.

The working interpretation, on one card

Small p-value: this data would be surprising if the change did nothing. Not a probability of victory, not an effect size, not a ship decision. Check the confidence interval for magnitude, check it against the smallest lift you would ship, and only trust p-values that were built for the way you actually monitor — continuously. That is the entire discipline, and it fits on an index card taped to the dashboard.