✎ Blog
Notes from building an experiment engine.
What we have learned about assignment, statistics, growth teams, and the stubborn work between an experiment idea and a clean result. Product claims are tied back to the system we are building; comparisons link to the vendors' own documentation.
The A/B testing glossary: 40 terms in plain English
Every A/B testing term you will meet in a readout meeting — from A/A test to variant — defined in plain English, with working one-liners, not textbook prose.
When NOT to A/B test
A/B testing is the wrong tool below the traffic floor, for irreversible bets, for ethically sensitive changes, and for obvious bugs. When to decide and ship.
Is A/B testing bad for SEO? What Google actually says
A/B testing does not hurt SEO when done right: no cloaking, canonical tags on redirect tests, crawlers treated like users. The rules and a practical checklist.
Running multiple A/B tests at once: interactions and overlap
Overlapping A/B tests are usually fine: randomization spreads other tests evenly across arms. When interactions bite, and how layering keeps concurrency safe.
One decision metric per test (and as many guardrails as you like)
Every extra success metric in an A/B test is another chance to declare a false win. Why one primary decision metric, plus guardrails, keeps experiments honest.
A/B/n testing: how many variants is too many
Every variant you add to an A/B/n test splits your traffic thinner and adds another comparison. How to decide between two arms, four, or iterating.
A/B testing in mobile apps: what changes
Mobile A/B testing runs into release cycles, store review, and fragmented versions. How remote config and server-driven variants make app experiments workable.
Segmenting A/B test results without fooling yourself
Slicing A/B test results by segment finds real insights and fake ones at the same rate. How to pre-register segments and handle multiple comparisons.
The flicker effect: why client-side tests look broken
The flicker effect is the flash of original content before a client-side A/B test swaps in the variant. What causes it, how it biases results, and the fixes.
ICE, PIE, and reality: prioritizing your experiment backlog
ICE and PIE score experiment ideas by impact, confidence, and ease. Useful until the scores are guesses. How to prioritize your A/B test backlog from evidence.
Server-side vs client-side A/B testing: which one and when
Server-side A/B testing renders the variant before the page ships; client-side swaps it in the browser. Here is how each works, and when the trade-offs bite.
The novelty effect: why your variant looks great for a week
The novelty effect makes A/B test variants look great early because change itself draws attention. How to spot it with a new-user split and guard decisions.
Feature flags vs A/B tests: related, not the same
Feature flags control who gets a feature; A/B tests measure what a change causes. Where the two overlap, how they differ, and how a flag becomes a test.
Sample ratio mismatch: the silent test-killer
Sample ratio mismatch means your A/B test groups are not the size you asked for — randomization broke. Causes, the chi-square check, why you discard results.
Bayesian vs frequentist A/B testing: what the argument is really about
Bayesian and frequentist A/B testing answer different questions with different guarantees. A fair summary of both, and why the tribal argument misses the point.
A/B testing with low traffic: what actually works
A/B testing with low traffic works if you change the game: bigger swings, fewer variants, upstream metrics, sequential stats — and knowing when not to test.
A/A tests: the sanity check most teams skip
An A/A test splits traffic between two identical experiences to verify your testing setup. What it catches, how often to run one, and how to read a failure.
What is A/B testing? A practical guide for product teams
A/B testing splits traffic between two versions and measures which wins. How it works, when it pays off, and the mistakes that quietly ruin results.
Statistical significance in A/B testing: what 95% actually means
Statistical significance at 95% does not mean a 95% chance your variant is better. What the threshold really controls, and how to pick one per decision.
Sequential testing, explained without the math degree
Sequential testing lets you check A/B test results continuously without inflating false positives. How always-valid stats work and when they beat fixed-horizon.
P-values for product teams: a working interpretation
A p-value measures how surprising your A/B test result would be if the change did nothing. A working interpretation, common misreadings, and a better lens.
Minimum detectable effect: the number that decides your test before it starts
Minimum detectable effect is the smallest lift your A/B test can reliably see. How to choose an MDE from business value, and why 2% hunts on low traffic fail.
How long should an A/B test run?
Most A/B tests should run one to four weeks: full business cycles, and enough sample for the effect size. How to work out your number, and when to stop early.
Guardrail metrics: how to win without breaking something else
Guardrail metrics protect revenue, retention, and performance while your A/B test chases a win. How to pick them per funnel stage and act on a breach.
11 A/B testing mistakes that quietly ruin your results
The most common A/B testing mistakes — peeking, ignored sample ratio mismatch, testing trivia, skipped cleanup — and the concrete fix for each one.
A/B test sample size: how to size a test without a statistics degree
A/B test sample size comes down to three inputs: baseline rate, minimum detectable effect, and power. The intuition for each, and what to do when n is huge.