✎ Blog
Notes from building an experiment engine.
What we have learned about assignment, statistics, growth teams, and the stubborn work between an experiment idea and a clean result. Product claims are tied back to the system we are building; comparisons link to the vendors' own documentation.
Segmenting A/B test results without fooling yourself
Slicing A/B test results by segment finds real insights and fake ones at the same rate. How to pre-register segments and handle multiple comparisons.
The flicker effect: why client-side tests look broken
The flicker effect is the flash of original content before a client-side A/B test swaps in the variant. What causes it, how it biases results, and the fixes.
ICE, PIE, and reality: prioritizing your experiment backlog
ICE and PIE score experiment ideas by impact, confidence, and ease. Useful until the scores are guesses. How to prioritize your A/B test backlog from evidence.
Server-side vs client-side A/B testing: which one and when
Server-side A/B testing renders the variant before the page ships; client-side swaps it in the browser. Here is how each works, and when the trade-offs bite.
The novelty effect: why your variant looks great for a week
The novelty effect makes A/B test variants look great early because change itself draws attention. How to spot it with a new-user split and guard decisions.
Feature flags vs A/B tests: related, not the same
Feature flags control who gets a feature; A/B tests measure what a change causes. Where the two overlap, how they differ, and how a flag becomes a test.
Sample ratio mismatch: the silent test-killer
Sample ratio mismatch means your A/B test groups are not the size you asked for — randomization broke. Causes, the chi-square check, why you discard results.
Bayesian vs frequentist A/B testing: what the argument is really about
Bayesian and frequentist A/B testing answer different questions with different guarantees. A fair summary of both, and why the tribal argument misses the point.
A/B testing with low traffic: what actually works
A/B testing with low traffic works if you change the game: bigger swings, fewer variants, upstream metrics, sequential stats — and knowing when not to test.
A/A tests: the sanity check most teams skip
An A/A test splits traffic between two identical experiences to verify your testing setup. What it catches, how often to run one, and how to read a failure.
What is A/B testing? A practical guide for product teams
A/B testing splits traffic between two versions and measures which wins. How it works, when it pays off, and the mistakes that quietly ruin results.
Statistical significance in A/B testing: what 95% actually means
Statistical significance at 95% does not mean a 95% chance your variant is better. What the threshold really controls, and how to pick one per decision.
Sequential testing, explained without the math degree
Sequential testing lets you check A/B test results continuously without inflating false positives. How always-valid stats work and when they beat fixed-horizon.
P-values for product teams: a working interpretation
A p-value measures how surprising your A/B test result would be if the change did nothing. A working interpretation, common misreadings, and a better lens.
Minimum detectable effect: the number that decides your test before it starts
Minimum detectable effect is the smallest lift your A/B test can reliably see. How to choose an MDE from business value, and why 2% hunts on low traffic fail.
How long should an A/B test run?
Most A/B tests should run one to four weeks: full business cycles, and enough sample for the effect size. How to work out your number, and when to stop early.
Guardrail metrics: how to win without breaking something else
Guardrail metrics protect revenue, retention, and performance while your A/B test chases a win. How to pick them per funnel stage and act on a breach.
11 A/B testing mistakes that quietly ruin your results
The most common A/B testing mistakes — peeking, ignored sample ratio mismatch, testing trivia, skipped cleanup — and the concrete fix for each one.
A/B test sample size: how to size a test without a statistics degree
A/B test sample size comes down to three inputs: baseline rate, minimum detectable effect, and power. The intuition for each, and what to do when n is huge.