What is A/B testing? A practical guide for product teams
Every product team argues. Should the pricing page lead with the annual plan or the monthly one? Does the signup form need the company-size field? Is the new onboarding actually better, or does it just look better in Figma? A/B testing is the practice of settling those arguments with users instead of opinions: you ship both versions at once, split traffic between them at random, and let the metric decide.
That last word — random — is what separates a real experiment from a before/after comparison. If you ship a change on Tuesday and compare this week to last week, you are also measuring the weather, a competitor launch, a marketing email, and the day of the week. Random assignment gives both versions the same mix of everything, so the only systematic difference between the groups is the change you made.
How an A/B test actually works
- Pick one metric to move. Purchases, signups, activation — something that matters, not clicks on the button you changed.
- Form a hypothesis. "Moving social proof above the fold will lift checkout starts, because hesitant buyers need reassurance before they commit."
- Split traffic randomly. Each visitor is assigned to A (control) or B (variant), usually 50/50, and stays in that group on every return visit.
- Run until you have enough evidence. Not until the dashboard looks good — until the statistics say the difference is unlikely to be noise.
- Ship the winner, delete the loser. Including the dead code. Abandoned experiment branches are how codebases rot.
What A/B testing is good for — and what it is not
A/B testing shines when you have a decision with real stakes, measurable outcomes, and enough traffic for the answer to arrive in weeks rather than quarters. Checkout flows, pricing pages, onboarding steps, headlines, paywalls, and email capture are classic territory because small percentage changes compound into real revenue.
It is the wrong tool for questions users cannot answer with behavior ("should we rebrand?"), for changes so small no plausible sample could detect them, and for strategy. A test tells you which of two options performs better; it does not tell you whether you are building the right product. Teams that treat experimentation as a substitute for judgment end up optimizing their way into a local maximum.
The three mistakes that quietly ruin results
1. Peeking at fixed-horizon statistics
Classic t-test statistics assume you decide the sample size in advance and look at the result exactly once. Nobody does that. Teams check the dashboard daily and stop the moment the result looks significant — which inflates false positives dramatically, because you are giving noise many chances to cross the line. The honest fixes are either committing to a horizon and not looking, or using sequential statistics that stay valid at every peek.
2. Calling it early because the graph looks good
Early experiment data is dominated by your most engaged users, who react differently from the average visitor. Variants routinely look brilliant for three days and flat by day fourteen. If your test would need eight weeks to detect the effect honestly, a one-week "win" is usually a story you told yourself.
3. Measuring the wrong metric
Clicks on the new button will almost always go up — it is new. The question is whether purchases went up, and whether anything you care about went down. Good experiments pair a success metric with guardrail metrics (revenue, retention, page performance) so a variant cannot win on a technicality while quietly hurting the business.
What it costs to do this well
The statistics are the easy part — libraries exist. The real cost of an experimentation program is engineering: every test needs variant code written behind an assignment mechanism, exposure tracking wired in, a dashboard someone trusts, and cleanup when the test concludes. On most teams that is days of engineer time per experiment, which is why companies that "believe in testing" often ship three tests a quarter.
That cost structure is the problem Trevo exists to attack: it reads your repo, proposes experiments argued from your actual funnel, writes the variant code as a pull request you review, tracks results with sequential statistics, and opens the cleanup PR when the test decides. The judgment stays with your team; the days of plumbing per test go away.
Frequently asked questions
How much traffic do I need for A/B testing?
It depends on your baseline conversion rate and the smallest lift you care about. As a rough shape: detecting a 10% relative lift on a 5% conversion rate needs tens of thousands of visitors per variant, while a 30% lift on the same baseline needs a few thousand. Low-traffic teams should test bigger, bolder changes — large effects need far less data to detect.
What is the difference between A/B testing and split testing?
Nothing — they are two names for the same practice of randomly splitting traffic between versions and comparing a metric. "Multivariate testing" is the genuinely different technique: it varies several elements at once to measure interactions, and needs much more traffic to read cleanly.
How long should an A/B test run?
At minimum one full business cycle — usually one to two weeks — so every weekday and weekend is represented, and longer if your traffic is small relative to the effect you are hunting. Stopping the first day a dashboard shows significance is the most common way teams fool themselves.
Can A/B testing hurt SEO?
Not when done normally. Google explicitly allows A/B testing; use rel=canonical on variant URLs if you test with redirects, serve googlebot the same experience rules as users, and do not cloak. Most modern tools swap content in place or server-side, which avoids the issue entirely.