The A/B testing glossary: 40 terms in plain English

Experimentation has a jargon problem: half the terms are borrowed from statistics, half from engineering, and readout meetings casually mix both. This is the working glossary — the 40 terms you will actually hear, defined the way a practitioner uses them. For the full narrative version, start with what A/B testing is.

A–C

• A/A test — an experiment where both arms get the identical experience, run to verify your setup: if an A/A test finds a "winner", your assignment or tracking is broken. More in what an A/A test tells you.
• A/B test — randomly splitting users between two versions and comparing a metric, so the only systematic difference between groups is the change itself.
• A/B/n test — the same with more than one variant against control; every extra arm thins the traffic and adds a comparison. See how many variants is too many.
• Assignment — the decision of which arm a given user gets. Must be random and sticky: the same user gets the same arm on every visit.
• Bucketing — the mechanics of assignment, usually hashing a user ID into a numeric range so the split needs no database lookup.
• Confidence interval — the range of effect sizes consistent with your data; a lift of "4% ± 5%" says the true effect could plausibly be negative.
• Control — the current experience, the A in A/B. The baseline every variant must beat.
• Conversion rate — the share of users who did the thing you count as success, out of those who had the chance to.

D–G

• Decision metric — the single pre-declared metric that determines ship-or-kill; we argue for exactly one per test.
• Exposure — the moment a user actually experiences their assigned variant. Analysis should count exposed users, not merely assigned ones.
• False positive — the test says the variant wins when the true difference is nothing. The rate is set by your significance threshold and inflated by peeking and multiple comparisons.
• False negative — a real effect the test failed to detect, usually because the sample was too small for the effect size.
• Feature flag — a switch controlling which users get which code path; the release-control cousin of experimentation. See flags vs tests.
• Fixed-horizon test — the classical design where sample size is committed in advance and results are read once, at the end. Valid only if nobody peeks.
• Guardrail metric — a metric a variant must not damage (errors, latency, refunds, retention); it cannot make a test win, only stop it. See guardrail metrics.

H–M

• Holdback — a small group deliberately kept on the old experience after shipping, to measure long-term impact of the change.
• Hypothesis — the falsifiable claim a test evaluates: change X will move metric Y because of mechanism Z. If Z is missing, it is a guess with paperwork.
• Interaction effect — when the combined effect of two changes differs from the sum of the parts; the real concern when running multiple tests on the same surface.
• Lift — the relative improvement of variant over control; a move from 4.0% to 4.4% conversion is a 10% lift.
• Minimum detectable effect (MDE) — the smallest lift a test is designed to catch; the input that dominates sample size. See MDE explained.
• mSPRT — mixture sequential probability ratio test, a sequential method producing always-valid results you may check at any time. It is what Trevo runs on.
• Multiple comparisons problem — every extra metric, segment, or arm is another chance for noise to look significant; honest analysis corrects for the number of looks.
• Multivariate test (MVT) — varying several elements independently in one test to measure combinations; needs far more traffic than A/B and is usually overkill.

N–R

• Novelty effect — the temporary lift a change gets simply for being new; it decays, which is why early results flatter variants. See the novelty effect.
• Null hypothesis — the assumption the statistics try to reject: that there is no real difference between arms.
• Overall evaluation criterion (OEC) — a composite metric some teams use to encode the balance between growth and quality in one number.
• p-value — the probability of seeing data at least this extreme if the null were true. Not the probability the variant wins. See p-values for product teams.
• Peeking — checking a fixed-horizon test repeatedly and stopping when it looks significant; it silently multiplies false positives. Sequential methods exist to make looking safe.
• Power — the probability your test detects an effect of the size you care about, given your sample. Underpowered tests mostly return noise.
• Primary metric — same as decision metric; the one the test is about.
• Randomization unit — the thing you randomize: users, sessions, or pageviews. Metrics must be computed at the same unit or the statistics quietly break.

S–Z

• Sample ratio mismatch (SRM) — when arm sizes deviate from the configured split beyond chance; a red alert that assignment or logging is broken. See SRM explained.
• Sample size — how many users each arm needs before the test can decide; driven by baseline rate and MDE. See sample size math.
• Segment — a slice of users (device, channel, plan) analyzed separately; powerful when pre-registered, treacherous when fished. See segmenting results.
• Sequential testing — statistical methods that stay valid under continuous monitoring, so you can stop the moment evidence suffices. See sequential testing explained.
• Significance (statistical) — the observed difference is unlikely under the null hypothesis, at your chosen threshold. Not the same as practical significance — a real but tiny effect can be both significant and worthless.
• Split test — another name for an A/B test; the terms are interchangeable.
• Stickiness — the guarantee that a user keeps their assignment across visits and devices; without it, users experience both arms and the comparison dissolves.
• Stopping rule — the pre-declared condition under which the test ends; the alternative is stopping when the dashboard flatters you.
• Treatment — the changed experience; the B (and C, and D) in A/B/n. Also called the variant.
• Type I / Type II error — the formal names for false positive and false negative, in that order.
• Uplift — synonym for lift, more common in marketing-flavored writing.
• Variant — any experience being compared in a test; in strict usage the non-control arms, in loose usage all of them.

Vocabulary is the cheap part; the discipline it describes is the work. That gap is what Trevo automates — it proposes experiments from your funnel evidence, ships variants as pull requests, and runs the statistics (mSPRT, so peeking is safe by construction) with guardrails that auto-pause on damage. You bring the judgment; the glossary above becomes how you read its output.