ICE, PIE, and reality: prioritizing your experiment backlog
Every team that takes experimentation seriously ends up with the same problem: more ideas than traffic. The backlog holds forty tests, you can run three this month, and the loudest voice in the room is not a prioritization framework. ICE and PIE exist to fix that meeting.
Both work the same way. Score each idea on three dimensions from 1 to 10, multiply or average, sort descending, run the top of the list. ICE scores Impact (how much the metric moves if it wins), Confidence (how likely it is to win), and Ease (how cheap it is to build). PIE swaps in Potential (how much room the page has to improve) and Importance (how much traffic and value flows through it). The differences are cosmetic; the mechanism is identical.
What scoring frameworks genuinely get right
- They force comparison. An idea cannot be "obviously worth doing" in isolation; it has to beat the other thirty-nine on the same scale.
- They expose pet projects. When the CEO's redesign idea scores a 4 on Confidence and someone has to say why, the conversation improves.
- They make Ease count. Two tests with similar upside are not equal if one ships in a day and the other takes a sprint. Frameworks stop teams from ignoring cost.
- They create a paper trail. Six months later you can see why you ran what you ran, which is more than most backlogs can say.
The failure mode: garbage in, confident garbage out
Here is the uncomfortable part. In most teams, all three scores are vibes. Impact is a guess about an effect size nobody has estimated. Confidence is how excited the scorer feels. Ease is an engineering estimate made by someone who does not write code. Multiply three guesses and you get a number with three digits of false precision — a ranking that feels rigorous and is actually the same opinions wearing a spreadsheet.
The scores also inherit a structural bias: easy-to-imagine tests (button colors, headlines) score high on Confidence and Ease, while the uncomfortable structural tests (pricing, onboarding flow, when not to test at all) score low on Ease and get deferred forever. The framework quietly optimizes for trivia.
How to ground the scores in evidence
Impact: start from the funnel, not the idea
Impact should be computed, not felt. Take your funnel and find the step where the most value leaks: sessions times drop-off rate times value per conversion gives every step a ceiling in currency. A test on a step that 4% of users ever reach cannot beat a mediocre test on a step that loses half your signups, no matter how clever it is. If you have not instrumented the funnel, that is the real first task — choosing an activation metric is a good place to start.
Confidence: demand a mechanism and a signal
Confidence deserves points only when something observable backs it: session recordings showing users stall at the exact field you want to remove, support tickets naming the confusion, a segment that converts wildly better when it happens to skip the step. "I believe users want this" is a 2. "Users who reach pricing via the comparison page convert at twice the rate" is an 8, and it also tells you what to build.
Ease: include detection, not just development
Ease usually means "how long to build". It should also mean "how long to read". A subtle test on a low-traffic page might need months to reach a decision — that is cost, and it belongs in the score. A quick sample size estimate per idea turns Ease from a build estimate into a time-to-decision estimate, which is the number you actually care about.
A weekly cadence that keeps the backlog honest
- Monday, 30 minutes: review concluded tests first — what won, what the experiment review decided, what follow-ups it spawned. Results reshuffle priorities more than new ideas do.
- Score only new ideas. Do not re-litigate the whole list; re-score an old idea only when new evidence arrived.
- Fill open slots from the top. However many tests your traffic supports — often one or two — take them strictly from the ranking. Exceptions need a reason written down.
- Prune monthly. Anything that has sat below the line for a quarter gets deleted. A backlog is a queue, not a museum.
The pattern behind all of this: prioritization quality is downstream of evidence quality. That is also where automation genuinely helps. Trevo builds its proposals the evidence-first way — it reads your PostHog funnel, finds the steps leaking the most value, and argues each proposed experiment from that data, with the variant code arriving as a pull request. You still rank and veto; the difference is that every candidate arrives with its funnel citation attached instead of a vibes score.
Frequently asked questions
What is the ICE framework for A/B testing?
ICE scores each experiment idea on Impact (how much the metric would move), Confidence (how likely the idea is to win), and Ease (how cheap it is to build), each 1–10, then multiplies or averages them to rank the backlog. It is fast and forces comparison, but the ranking is only as reliable as the evidence behind each score.
What is the difference between ICE and PIE prioritization?
Structurally nothing — both multiply three 1–10 scores. PIE uses Potential (room for improvement on the page), Importance (traffic and value through it), and Ease. ICE uses Impact, Confidence, and Ease. PIE anchors slightly more to page-level data; ICE is more general. Choose either and spend your energy on evidence quality instead.
How do you prioritize A/B tests with limited traffic?
Weight time-to-decision heavily. With limited traffic, subtle tests take months to read, so prioritize bigger, bolder changes on your highest-traffic steps, where effects are large enough to detect quickly. Estimate required sample size per idea before ranking, and drop tests your traffic cannot decide within about a month.
How many A/B test ideas should be in a backlog?
Enough to fill your next few testing slots with quality — usually ten to twenty live ideas. Beyond that, a backlog becomes storage for decisions nobody wants to make. Prune quarterly: if an idea has never ranked high enough to run, new evidence should promote it or it should be deleted.