Chapter 06 · 5 min
Reading results and shipping a winner
The verdict, the three tiles, how to read “Too soon to tell”, and the cleanup PR.
The results page is reduced to three questions: is it winning, by how much, and should I ship. The verdict banner answers them in words, and three tiles underneath — lift, evidence, guardrail — carry the numbers. Everything lower on the page is derived from the same counts, so the banner cannot contradict the chart.
How to read the Evidence tile
The Evidence tile never shows a percentage, because “0% confidence” reads as a failed test to anyone who is not a statistician when it actually means “we cannot tell yet”. It shows one of four states: Too soon to tell (with a bar for how much of the evidence is in and how many more days it needs), No difference yet, Proven, or Proven — losing. Under the tiles, one sentence explains the state from counts you can check yourself: how many people saw the test, how many of them did the thing being measured, and how far apart the two arms are.
The demo has the two cases side by side. The guarantee strip has been seen by more than 60,000 people and still reads “Too soon to tell”, because it is measured on purchases and only about 3% of visitors buy: 60,000 visitors is roughly 2,000 orders, and the arms are 45 orders apart — easily luck. The related-products shelf has a similar audience but is measured on add-to-cart, which one visitor in six does; its arms are 420 events apart on 13,000, and it reads “Proven”. What matters is not how many people saw the test but how many did the thing you are counting. When a card says “about 83 more days”, that is the honest price of a small effect on a rare event, and it is far cheaper to know it on day eight than to guess.
For the people who want them, the p-value and the interval are one click away under “Advanced stats” on each variant. Trevo runs a mixture sequential probability ratio test, so those numbers are valid at every moment — peeking on day four does not spoil them, and “Proven” means the gap would happen by luck less than one time in twenty, most often far less.
Guardrails and data health
Every experiment names a downstream metric that must not degrade — purchase rate under an add-to-cart test, page performance under a layout test. If it degrades significantly past a magnitude floor, assignment flips to control automatically and a revert PR opens; nobody has to be watching. A sample-ratio check runs alongside: if traffic did not split the way it was configured, something upstream is broken and the result is flagged rather than reported.
Shipping
When the test calls it, “Ship it” opens the cleanup PR: the winning arm becomes the only code path, the flag and the losing arm are deleted. Merge it and the ledger starts counting — baseline revenue per day × measured revenue lift × days live. A flat result gets the opposite PR, and a proposal that ran to no difference is still a result: it is on the list so nobody proposes it again.