A/B testing in mobile apps: what changes

On the web, an experiment is a deploy away: push the variant, split traffic, read the results, push the cleanup. In a mobile app, the code users run left your hands weeks ago, passed through app review, and sits on devices you cannot touch — some of them two, five, ten versions behind. The statistics of experimentation do not change on mobile. The logistics change completely.

Teams that port their web testing habits to an app hit the same three walls in the same order: the release cycle, the review process, and version fragmentation. Here is what each does to your experiments, and the delivery pattern that gets around all three.

The three constraints that reshape mobile testing

• The release cycle owns your iteration speed. Web tests iterate in hours. An app change rides the next release train — typically every one to two weeks — plus review time. If launching a test requires shipping code, your experiment cadence is capped at your release cadence.
• Store review limits what can change remotely. Both major stores allow config-driven behavior changes but prohibit downloading new executable code outside the store. Variants must therefore be code paths already in the binary, toggled remotely — not new logic injected later.
• Version fragmentation splits your population. Users adopt updates on their own schedule. Whatever you shipped in 4.2 coexists with 4.1 and 3.9 for months, so any experiment must define which versions are eligible and accept that the eligible population ramps up slowly as adoption spreads.

The pattern that works: ship both, decide remotely

The industry-standard answer is to break the link between shipping code and launching experiments. Both variants ship inside the same binary, dark. A remote configuration service — the app-world equivalent of a feature flag — assigns each user and tells the app which path to run. Launching the test, adjusting the split, pausing on damage, and shipping the winner are all config changes, live in minutes, with no release and no review.

A stronger version of the same idea is server-driven UI: the app renders what the backend describes — an onboarding sequence, a paywall layout, an offer — so variants live server-side entirely. That moves mobile experimentation surprisingly close to server-side web testing: variants are ordinary backend code, reviewable in pull requests, changeable without touching the binary at all. The cost is upfront architecture; screens must be built to render from data. Most teams apply it only to their highest-experimentation surfaces — paywalls, onboarding, offers — which is usually where the money is anyway.

What changes about measurement

Mobile analytics have their own physics, and a few web habits translate badly:

• Events arrive late. Offline usage, batched uploads, and OS background restrictions mean conversions trickle in hours or days after they happened. Reading a mobile test too early undercounts recent cohorts; build the lag into your patience.
• Identity behaves differently. No cookies — device identifiers, app instance IDs, and your own login system have to agree on who a user is, especially across reinstalls. Sloppy identity here shows up as sample ratio mismatch later.
• Assignment must be sticky offline. A user who opens the app in airplane mode should see their assigned variant, not the default. Persist the assignment locally at first exposure.
• Guardrails include app health. Crash rate, cold-start time, and battery drain join the usual guardrail metrics — a variant that lifts conversion while raising crashes is a loser wearing a win.

Cleanup: the mobile version of experiment debt

On the web, deleting a losing variant is a small PR the day the test concludes. In an app, the losing code path stays in the binary until a future release removes it — and because removal never blocks anything, it tends not to happen. Binaries accumulate dead paywalls and orphaned onboarding branches, each one still reachable if a config value regresses.

The discipline is the same one we describe in cleanup PRs and experiment debt, with one mobile twist: schedule the deletion into the next release train as part of concluding the experiment, and keep the config pinned to the winner until the last experiment-bearing version falls below your support floor. Concluding a mobile test has three steps, not one: lock the config, delete the code next release, retire the flag when old versions age out.

Trevo's loop covers the web and backend side of this today — Next.js, React, and server-side code, where variants ship as pull requests and cleanup PRs arrive when the experiment decides. For native app teams the same shape applies even when the tooling differs: variants as reviewed code paths, one decision metric, always-valid statistics, and a cleanup step nobody has to remember. And anything your app drives from the backend — pricing logic, onboarding sequencing, server-rendered content — can run through the full loop now.