∴ Use case · AI products
A/B test prompts, models, limits, and AI product behavior.
An AI product changes when its prompt, model, tool budget, or fallback changes. Trevo tests those code paths directly and returns each variant as a pull request, tied to product outcomes and operating cost.
export function routeModel(req: DraftRequest) { return LARGE_MODEL; // exp: fast model for short rewrites return variant === "fast-short" && req.mode === "tighten" ? FAST_MODEL : LARGE_MODEL;}
How do you A/B test an AI product?
AI product experiments change server-side behavior — the prompt template, which model serves which request, the token or usage limits, the fallback when generation fails — and measure the product outcome: task completion, retries, latency, retention, cost per successful action. With Trevo, each variant is a pull request in your repository behind a feature flag, so the exact prompt and routing rule of every arm is reviewable before it runs. Assignment is consistent per user, results use always-valid sequential statistics, and the losing behavior is removed by an automatic cleanup pull request. Offline evals tell you which output looks better; this tells you which behavior users actually finish tasks with.
How it works
From “should we try the cheaper model here?” to a measured, reviewable answer.
Trevo reads the AI surface
The scan finds where generation happens: prompt construction, model routing, limits, retries, and the UI that exposes them. Proposals cite those files.
Variants are behavior changes in code
A tightened prompt, a different router rule, a raised free-tier cap — expressed as a diff behind a flag, with both behaviors explicit for review.
Users are assigned, outcomes tracked
Consistent per-user assignment so nobody flips models mid-session. The experiment PR wires the events that define success for your product.
Product metrics decide, not eyeballs
Completion, retries, latency, retention — analyzed with mSPRT always-valid statistics. Merge the winner; cleanup removes the losing behavior.
Prompts and routing under version control, where they belong
When prompts live in a dashboard, nobody knows what was running when the metric moved. As pull requests, every variant is diffable, reviewable, and attached to its result — the experiment log is your git history.
- ✓The exact prompt of each arm is in the diff
- ✓Model routing rules tested per mode, tier, or request shape
- ✓Rollback is the flag; removal is the cleanup PR
hypothesis examples beat instructions for tone rewriteschange swap rule list for 3 worked examplessurface server-sidemetric draft_accepted → retained editors
Cost is a testable dimension
The cheaper model is only a win if task success holds. Trevo lets you run the trade as an experiment — route a slice of traffic, watch completion and retries against cost per successful action — instead of switching everyone and hoping.
- ✓Route by request type, tier, or length — in code
- ✓Guardrail metrics catch quality regressions
- ✓Limits and caps testable the same way
control large model, all requestsvariant fast model for “tighten” modeprimary draft_accepted rateguard retries · latency p95 · cost/action
What this looks like in practice
Illustrative proposals of the kind Trevo writes for AI products — yours come from your own generation code.
Δ model routing
Fast model for short rewrites
Serve the small model where latency matters and length is short; keep the large model elsewhere.
measures → draft_accepted → cost per accepted draft
Δ prompting
Examples-first prompt
Replace instruction lists with worked examples in the template and measure acceptance, not aesthetics.
measures → draft_accepted → weekly active editors
Δ limits
Raise the free daily cap
Test whether a higher free cap builds the habit that converts, or just serves more free tokens.
measures → daily_active → paid conversion
Frequently asked questions
How is this different from prompt evals?
Evals grade outputs offline against a rubric; they are how you develop a candidate. An A/B test measures what real users do with each behavior — completion, retries, retention, cost. Teams use both: evals to pick the challenger, Trevo to find out if it actually wins in production.
Can Trevo test which LLM provider or model we use?
Yes, if the choice is expressible in your code — which model routing almost always is. The experiment changes the routing rule behind a flag, and the decision is made on your product metrics and cost per successful action.
What metrics make sense for AI features?
Pick the action that means the output worked: draft accepted, answer copied, task completed without retry. Pair it with guardrails — latency, retries, cost per successful action — so a quality regression cannot hide behind an engagement bump. Trevo wires these events in the experiment PR.
Is it safe to experiment on generation behavior?
The variant is explicit code behind a flag, reviewed by your team before it runs, and removable by merging the automatic cleanup PR. Nothing changes for users until you approve the pull request, and both arms are visible in one diff — safer than editing a prompt in a dashboard nobody reviews.
Does Trevo itself use AI to write these experiments?
Yes — Trevo reads your repository and writes the proposals and pull requests. That is the product: an agent that does the experimenting, with your code review as the gate. Nothing it writes ships without a human approving the diff.
Keep reading: Pricing experiments · Trevo for growth teams · GitHub + Trevo · How the statistics work · Trevo vs Statsig
Bring Trevo one real funnel problem.
Connect the repo and the first proposals arrive in under fifteen minutes, each one naming the exact files a full-stack experiment would change. Not ready? A free scan previews the ideas on your domain first.