A/B Testing Models
The only measurement of business impact is to let two models decide for two comparable populations and compare what happens — with stable assignment, guardrails, enough sample, and honesty about interference and about the users in the experiment.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Why is a live experiment the only way to know whether a model improved the business, and what makes such an experiment invalid?
Two teams each have a candidate ranking model and both beat the champion offline. Leadership wants to know which one to ship, and "which is more accurate" is not the question they are asking — they want to know which one makes more money without hurting the marketplace.
Split traffic three ways — champion and the two candidates — run for a few days, and ship whichever arm has the highest revenue per search. A/B testing is the gold standard; the platform already supports it; the result is the truth.
After three days the best arm leads by an amount well inside the noise. Shipping it is a coin flip with a ceremony attached; a week later the "winner" is behind (Metric Uncertainty).
- After three days the best arm leads by an amount well inside the noise. Shipping it is a coin flip with a ceremony attached; a week later the "winner" is behind (Metric Uncertainty).
- Candidate A wins on revenue by concentrating purchases on a few high-converting sellers. Seller concentration in its arm rises; smaller sellers see no traffic. Nobody set a guardrail, so the experiment reports a win that the marketplace team would call a loss.
- The arms share inventory. Candidate A's arm buys up the best-priced listings in the first hours; candidate B's arm, ranking the same listings, finds them gone and shows the next best. B's revenue per search is depressed by A's existence, and the measured difference is partly interference, not quality.
- Assignment is by search, not buyer. The same buyer sees three rankings on three queries, learns none of them, and the return-rate guardrail is measured on a population that was in every arm.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Rank listings in a marketplace search to maximise completed purchases. The label offline is a click or purchase on the shown ranking; the goal online is revenue per search over a period, with marketplace health — seller concentration, buyer return rate — as constraints.
- A ranking is an intervention: what is shown decides what is bought. The offline label was produced under the champion's rankings.
- Offline: a click log under the champion, with the position each clicked item was shown at. Both candidates were evaluated on it, which rewards agreeing with the champion about what to show at the top (Offline vs Online Evaluation).
- Online: each search assigned to an arm by buyer id, with the arm, the model version, the shown ranking, the clicks and the purchase logged per search; the marketplace-health metrics computed per arm over the experiment.
- The arms share the same sellers and the same inventory. A listing bought in one arm is gone from the other.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- An A/B test is a randomised comparison: units are assigned at random to arms, each arm experiences one policy, and the difference in outcomes between arms estimates the causal effect of the policy — because randomisation makes the arms comparable in everything except the policy. That is the whole reason it can measure business impact where offline evaluation, which observes outcomes under one policy, cannot (Causality vs Prediction).
- The estimate has uncertainty that shrinks with sample size. The minimum detectable effect, the metric's variance and the desired confidence together fix how many units are needed, and dividing by the arrival rate fixes the duration. Stopping when the difference first looks significant, or checking daily and stopping on a good day, inflates false positives.
- The estimate is unbiased only if the arms do not affect each other. When arms share a resource — inventory, sellers' attention, a budget, a social graph — one arm's policy changes the other arm's outcomes, and the difference measures the policies plus their interference. The usual repairs are assignment by cluster (market, region, seller) or time-sliced switching, each with its own cost in variance.
Why the offline number cannot answer the business question
Both candidates beat the champion on the click log. The click log was produced by the champion's rankings: every click is on an item the champion chose to show, at the position it chose. A candidate that would show different items at the top has no clicks to be scored on for those items, and a candidate that agrees with the champion about the top is rewarded for it. The offline number measures agreement with the incumbent policy at least as much as it measures quality.
That is a structural property of any log produced under a policy, not a flaw in the metric. The only way to observe outcomes under the candidate's policy is to run the candidate's policy, on a population comparable to the one the champion runs on. That is the experiment.
Both candidates score higher than the champion on click-through over the champion's logged rankings, with candidate A ahead of B.
In a four-week buyer-randomised experiment, A shows higher revenue per search and a seller-concentration guardrail breach; B shows a small, interval-excluding-zero revenue gain with guardrails flat. The arms are later found to have interfered through shared inventory.
- 1The click log rewards agreeing with the champion; A agreed more, and its offline lead was partly that.
- 2A's revenue came from concentrating purchases on a few sellers — a policy effect the click log could not show because the champion never did it.
- 3A's arm consumed the best-priced shared inventory first, depressing B's measured revenue; the true A-versus-B gap is smaller than measured.
Guardrails, sample size and stopping
The primary metric answers "did it help"; the guardrails answer "did it hurt anything we care about more". Both are chosen before the experiment, because a metric chosen after the results are in can always be found to favour the result. Sample size is arithmetic from the smallest effect worth shipping and the metric's variance, and the duration follows from it — with a floor of one weekly cycle, because weekends and weekdays are different populations.
Stopping is where experiments quietly fail. Checking the result each day and stopping on the first significant one is a search over stopping times for a good number, and it finds one whether or not there is an effect. Commit to the duration or use a sequential method built for peeking.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Daily peeking, stop on first significance | A run of "wins" that do not persist after shipping | Multiple looks inflate the false-positive rate; the stopping time was selected on the result | Pre-commit the duration or use a sequential test; report the number of looks |
| Shared inventory or budget across arms | The winner's margin shrinks or reverses when shipped to everyone | One arm's decisions changed the other arm's available outcomes | Randomise on the unit that contains the interference — seller, region, time slice |
| Primary metric wins, no guardrails set | Marketplace-health complaints weeks after shipping | The policy improved the metric by harming something unmeasured | Guardrails agreed before the start, with bounds, as stop conditions |
| Assignment by request rather than buyer | Return-rate guardrail is flat in every arm; arms look identical | Every buyer was in every arm; per-buyer outcomes cannot be attributed | Assign on the unit the outcome is measured on, stably, with the experiment id in the hash |
1from math import ceil2from statistics import NormalDist3 4def searches_per_arm(baseline_rev_per_search, sd_rev_per_search, min_effect_pct,5 alpha=0.05, power=0.8):6 # smallest absolute difference worth shipping7 delta = baseline_rev_per_search * min_effect_pct8 z_a = NormalDist().inv_cdf(1 - alpha / 2)9 z_b = NormalDist().inv_cdf(power)10 n = 2 * ((z_a + z_b) * sd_rev_per_search / delta) ** 211 return ceil(n)12 13n = searches_per_arm(baseline_rev_per_search=2.40, sd_rev_per_search=18.0, min_effect_pct=0.02)14# revenue per search is heavy-tailed: sd is large relative to the mean, so n is large.15# duration = n / (searches per day per arm), rounded up to whole weeks.The sd is the honest input. Revenue per search is mostly zero with occasional large purchases, so its standard deviation dwarfs its mean and the sample the arithmetic returns is large. A team that skips this step is choosing a duration by patience.
The people in the experiment
An experiment deliberately gives some users a policy the team suspects is worse, to find out. For a ranking that is a mildly worse search; for a credit limit, a medical triage score or a content moderation model it is a materially different treatment of a person who did not choose to be in a study. The experiment is still the only way to measure impact; that does not make every experiment acceptable.
The practical questions are the ones a review board would ask: is the expected harm to the worse arm bounded and small; does a guardrail stop the experiment if it is not; can a user be excluded; is the experiment measuring something that could not be learned less intrusively. The assumption a team makes without asking these is that its users consented to being experimented on, and for high-stakes decisions that assumption should be written down and defended (Human Oversight, Fairness).
The harm to users in the arm that turns out worse is bounded, small relative to the benefit of knowing, and would be stopped by a guardrail before it grew.
holds when The decision is low-stakes and reversible (a ranking, a suggestion); the guardrails cover the harms that matter to the user and not only to the business; the experiment is as small and short as the sample-size arithmetic allows.
breaks when The decision is a credit limit, a price, a triage score or a moderation action; the guardrails are all business metrics; the experiment runs longer than needed because nobody computed the sample size; the arms are unequal in a way that tracks a protected group.
respond Stop the arm. For a high-stakes decision, prefer a design that measures less intrusively — a small canary with a human in the loop — and accept the wider interval.
How to build it
Most important first.
- Choose the primary metric and the guardrails before starting, in the business's terms: revenue per search as primary; seller concentration, buyer return rate, latency and refund rate as guardrails with pre-set bounds. A win on the primary with a guardrail breach is not a win.
- Compute the sample size from the minimum effect worth shipping and the metric's variance, and commit to the duration — at least one full weekly cycle. Do not peek and stop; if early stopping is needed, use a method designed for it.
- Assign on the unit the outcome is measured on — the buyer — stably and with the experiment id in the hash. Where interference is expected, assign on the unit that contains it: a region or a seller cluster.
- Log the arm and the model version on every request so the outcome can be joined back per arm (Prediction Logging), and record the experiment with the same care as a training run — it is the only measurement of impact you will get (Experiment Tracking).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The difference in the primary metric between arms with a confidence interval, at the pre-committed sample size. This is the number a shipping decision rests on.
- Each guardrail per arm against its bound. A breach is a stop regardless of the primary.
- Offline metrics of the candidates — click-through on the champion's log, NDCG on held-out queries — are the reason the candidates are in the experiment. They are not the result.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Assignment is random with respect to the outcome, stable per unit for the duration, and independent of assignments in previous experiments.
- Arms do not interfere — or the assignment unit was chosen so that interference is contained inside a unit — so that the difference between arms is the difference between policies.
- The metric measured during the experiment is the metric the decision is about, over a period long enough to include the weekly cycle and outlast novelty.
- Offline: an A/A test — the same model in both arms — to confirm the assignment and metric pipeline show no difference when there is none. If it does, the experiment platform is broken.
- Online: the pre-committed duration, the pre-committed metrics, the guardrails checked per arm; an interference check comparing outcomes for units near the boundary (shared sellers, shared inventory) against units far from it.
- Over time: after shipping, a holdback on the loser for a period, to confirm the measured effect persists past novelty and past the label delay.
What can go wrong
- The experiment is stopped the first day the difference is significant. Across many experiments, this ships noise about as often as real wins.
- The units are buyers but the interference is through sellers: a seller whose listings sell out in arm A raises prices, and arm B's buyers pay more. Buyer-level randomisation cannot see this.
- A novelty effect: the candidate's different ranking gets extra clicks for a week because it is different. The experiment ends inside that week and reports a win that fades.
- For the experiment's duration, some fraction of users receive the worse policy on purpose. That is a real cost to those users and the reason the ethics question is not rhetorical.
- Sample-size arithmetic often says weeks, and a team that wants to ship monthly cannot A/B test every candidate; a canary with proxies is the pragmatic fallback and measures less.
- Cluster-randomised designs contain interference and need far more clusters than buyer-level designs need buyers; the interval widens, and small effects become undetectable.
- "The candidate is more accurate offline, so we don't need an experiment." The offline log was produced under the champion. The candidate's accuracy on it says how well it predicts what the champion showed and users clicked; it says nothing about what happens when the candidate decides.
- "Arm A had higher revenue, so ship A." At what interval, over what period, with which guardrails, and how much of the difference is A eating B's inventory? Each of those can reverse the answer.
- "A/B testing is just for product features; a model change is a backend change." A model change is a policy change, and it is the one kind of change whose impact cannot be measured any other way.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a randomised comparison is the only unbiased measure of a decision policy's effect holds for any model whose predictions influence outcomes; the design details — unit, duration, guardrails — are specific to the product.
- DOMAIN-SPECIFICMarketplaces, ad auctions and social feeds share inventory, budget or attention across arms and need cluster or switchback designs; a churn-call model or a spam filter has arms that barely interact and buyer-level assignment is fine.
- CONTESTEDA serious position holds that A/B testing every model change is a luxury of large-traffic products: at modest volume the sample-size arithmetic yields months per experiment, the opportunity cost of not shipping dominates, and a canary with guardrails plus a post-hoc holdback captures most of the safety at a fraction of the delay. That is right about the delay; the counter-argument is that a canary measures proxies and the holdback is itself an A/B test with a bad split, so the choice is between a small experiment and no measurement, not between an experiment and a faster equivalent.
Where the depth lives
This domain teaches the model and hands the rest off by name.