Champion / Challenger
A candidate earns production by beating the incumbent on the same traffic under the same threshold policy, on more than one number. A better validation score is a nomination, not a promotion.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A retrained model scores better offline than the one in production. What comparison justifies replacing the incumbent, and why is the validation delta not it?
Every quarter someone trains a new fraud model and it has a better validation number than the one running. We ship it, and roughly one time in three the review queue changes size or the analysts complain. We have no way to know before shipping which time it will be.
Compare validation scores. The challenger was trained on more recent data with the same pipeline, its AUC is higher, so it is the better model. Promote it, and if anything looks wrong, roll back.
The challenger's scores are more spread out, so the champion's threshold flags twice as many transactions. The queue doubles, analysts are overwhelmed, and precision at the actual operating point is worse — even though the ranking, measured by AUC, is better.
- The challenger's scores are more spread out, so the champion's threshold flags twice as many transactions. The queue doubles, analysts are overwhelmed, and precision at the actual operating point is worse — even though the ranking, measured by AUC, is better.
- The challenger is better overall and worse on the one merchant segment that produces most of the chargeback losses. The aggregate delta hides a regression on the slice that matters (Evaluation Slices).
- The challenger's validation set is the most recent period, which the champion has been running on. The champion's production outcomes on that period are available and were never used, so the fair comparison was possible and was not made.
- Roll back after something looks wrong takes the length of the label delay to trigger, because "looks wrong" is a precision measurement and precision needs chargebacks. The queue size was the leading indicator, and nobody was watching it.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a transaction will be charged back within 90 days. The label is the chargeback; the decision is a review queue whose size is fixed by analyst headcount.
- The question being answered by promotion is not "is the new model more accurate" but "will replacing the incumbent improve the outcomes the queue exists for, at the queue size we can afford".
- The incumbent (the champion) has months of production predictions with outcomes now attached. The candidate (the challenger) has only the offline evaluation from its training run, on a validation set the champion never saw as validation.
- The champion's threshold was tuned to produce a queue of a certain size on its score distribution. The challenger's score distribution is different — same features, different weights — so the same threshold produces a different queue.
- The two models were evaluated on different data: the champion's validation set is a year old, the challenger's is recent. Their two numbers are not comparable.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Two models with different weights produce different score distributions for the same inputs, even when they rank similarly. A threshold is a point on one distribution; moved to the other, it selects a different fraction of traffic. Any comparison at a fixed threshold is a comparison of two different operating points, and any comparison by AUC ignores the operating point entirely (Threshold Selection).
- A fair comparison holds the traffic fixed and the policy fixed: both models score the same examples, each with the threshold that gives it the same queue size (or the same expected cost), and the outcomes are compared at that matched operating point, overall and per slice.
- Promotion is a decision about a system — quality at the operating point, per-slice regressions, latency, memory, cost, calibration, robustness to the inputs the champion has been receiving — and the offline score is one input to it (Promotion Is a Checklist, Not a Score).
The same threshold is not the same operating point
The champion's threshold was chosen to flag a fraction of traffic that fills the review queue. The challenger produces different scores for the same transactions — more confident on some, less on others — so the same threshold selects a different fraction. The comparison "at the same threshold" compares a full queue with an overflowing one and calls the difference model quality.
The fix is to choose thresholds in the business's units. Give each model the threshold that produces the queue size the analysts can handle, then compare what each queue catches. That is the only comparison the analysts would recognise as fair, and it is the one AUC cannot make for you.
Both models thresholded at the champion's tuned value. The challenger flags twice as much traffic; its precision at that point is lower; its recall higher. The comparison reports "recall improved, precision dropped" and someone decides by taste.
Each model thresholded so it flags exactly the fraction of traffic the queue can absorb. Precision and recall compared at that matched point, on the same transactions, with intervals, per merchant segment.
The queue size is the constraint the business actually has. A threshold is a way of hitting it on one model's score distribution and means nothing on another's. Matching on the constraint compares the two things that would actually be deployed.
Scoring the challenger on the champion's traffic
The champion has been serving real requests for months, and the outcomes for most of them have arrived. That log is the best evaluation set the challenger will ever get: the exact serving-time features, the exact traffic mix, the real outcome. Scoring the challenger on it removes the two biggest sources of unfairness — different validation windows and warehouse-versus-serving features — at no cost beyond having kept the log.
The remaining unfairness is the champion's own influence on the outcomes. Transactions it flagged were reviewed and some blocked, so their chargeback outcome is unknown; the comparison runs on the outcomes the champion let through. A challenger that disagrees with the champion on the blocked set cannot be scored there, and this is the limit no offline comparison escapes.
1-- one row per production request the champion served, with its outcome2with served as (3 select p.request_id, p.feature_ref, p.score as champion_score,4 p.decision as champion_decision, o.chargeback5 from prediction_log p6 left join outcomes o on o.request_id = p.request_id7 where p.model_version = 'fraud-v7'8 and p.ts between '2026-05-01' and '2026-05-31'9),10-- the challenger scored on exactly the served feature vectors, not on recomputed ones11rescored as (12 select s.*, c.score as challenger_score13 from served s join challenger_scores c using (request_id)14)15select merchant_segment,16 count(*) filter (where chargeback is null) as outcome_unknown, -- blocked by the champion17 count(*) as n18from rescored r join requests using (request_id)19group by merchant_segment;The outcome_unknown column is the honest part. Where the champion blocked, there is no label, and the challenger's opinion on those rows cannot be scored. If that count is large in a segment, the comparison in that segment is the champion grading its own homework.
Promotion is a decision about a system
A challenger that ranks better and is slower, or ranks better on average and worse on the high-loss segment, or ranks better and is badly calibrated so the queue's expected-loss ordering is wrong, has not earned promotion. Each of those is a separate criterion with a separate owner, and the registry should record which passed.
The trade-off matrix below scores a typical trio. None of the scores is the answer; the point is that a single-number comparison would have promoted the wrong one.
The threshold policy is expressed as queue size, expected cost or recall target and re-derived for each model, so that champion and challenger are compared at matched operating points.
holds when The policy lives in the promotion pipeline as a procedure, not as a number in a config; the queue size is monitored as its own signal after every promotion.
breaks when Someone copies the champion's threshold into the challenger's config; the queue capacity changes and the threshold is not re-derived; a slice's mix changes so the matched queue is not matched in analyst effort.
respond Re-derive the threshold for the challenger before any further rollout. Do not retrain; the scores were not the problem.
| Option | Quality | Latency | Cost | Interpretability | Operational | Note |
|---|---|---|---|---|---|---|
| Champion (incumbent GBM) | Known behaviour, tuned threshold, months of outcome history; ageing. | |||||
| Challenger A (retrained GBM) | Better at the matched operating point, same latency; a small regression on one merchant segment that its owner accepts. | |||||
| Challenger B (larger ensemble) | Best ranking; doubles p99 latency past the checkout budget; needs a new serving image. | |||||
| Challenger C (neural, richer features) | Comparable quality; depends on three new features whose serving path has not been skew-tested. |
caveat The scores cannot express the confidence intervals — A's quality edge may be inside noise — or the fact that C's risk is unknown rather than low. A matrix makes a trade-off visible; it does not make it.
How to build it
Most important first.
- Score the challenger on the traffic the champion actually served, with the champion's production outcomes as labels. This is the fairest offline comparison available and it costs nothing but a prediction log (Prediction Logging).
- Apply the same threshold policy to both: same queue size, same expected cost, or same recall target — chosen once, in the language of the decision. Never the same numeric threshold.
- Compare on the promotion criteria the registry defines: quality at the operating point, worst-slice quality, latency, memory, calibration and cost, each with a required non-regression. A challenger that wins on one and loses on another is a discussion, not a promotion.
- Then move to live traffic — shadow, then canary — because the offline comparison, however fair, still cannot see skew or the outcome of decisions the challenger would make differently (Shadow Deployment, Canary Rollout).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Precision and recall at the matched operating point, on the same examples, for both models, with a confidence interval on the difference (Metric Uncertainty). This is the number the promotion rests on.
- The worst per-slice delta, on the slices the business names — merchant category, region, new versus established customers. A challenger is promoted on its worst slice, not its average.
- AUC delta is a ranking property and does not map to the queue. It is a reason to run the comparison, not its result.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The traffic the champion served, with its outcomes, is representative of the traffic the challenger would serve — and the champion's own interventions have not removed outcomes in a way that biases the comparison.
- The threshold policy is expressed as a business quantity — queue size, cost, recall target — and is re-derived for each model rather than carried over as a number.
- The comparison covers the slices that carry the business risk, and a slice regression blocks promotion regardless of the aggregate.
- Offline: the challenger scores the champion's logged production traffic; both models are thresholded to the same queue size; the outcome comparison is reported overall, per slice and with intervals.
- Online: the promoted challenger runs in shadow, then on a canary slice, and the queue size and per-slice flag rates are watched as leading indicators before any chargeback arrives.
- Over time: keep every promotion decision with its comparison table, so that a regression discovered months later can be traced to which criterion was missing.
What can go wrong
- The prediction log lacks the features, so the challenger is scored on features recomputed from the warehouse — a different distribution — and the comparison inherits train/serve skew (Train / Serve Skew).
- The champion's decisions changed outcomes: transactions it flagged were reviewed and some were blocked, so their chargeback label is missing. The challenger is compared on the champion-filtered outcome set, which favours agreeing with the champion.
- The threshold policy is matched on queue size, but the challenger's queue is a different mix of merchants and the analysts' throughput on it differs. The matched operating point was not as matched as it looked.
- A fair comparison needs a prediction log with features, model version and outcomes joined — infrastructure that has to exist before the first challenger.
- Matching the operating point per model means the "same threshold" is not a config value but a procedure, and every threshold change becomes a small re-evaluation.
- Multi-criteria promotion produces challengers that are better on most axes and are not promoted. That is a correct outcome that teams find hard to accept.
- "If offline AUC improved, ship it." AUC is threshold-free and the queue is not. A better-ranking model can produce a worse queue at every threshold you can afford.
- "The challenger is better on average, so it is better." Average across slices hides the segment where the losses are. Promote on the worst slice you care about.
- "We can compare their validation scores; it is the same pipeline." Different validation windows, different periods, different base rates. The only common ground is the champion's production traffic, and that is where the comparison belongs.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat two models' score distributions differ, so a fixed numeric threshold means different operating points, holds for every scoring model; the procedure of matching on a business quantity applies whether the decision is a queue, a limit or a ranking cut-off.
- TASK-SPECIFICFor a ranking task without a threshold — a recommender's top-k — the matched operating point is k itself and the comparison is on outcomes at k; for a regression model there is no threshold and the comparison is on error at the decision's tolerance.
- CONTESTEDA defensible position holds that multi-criteria promotion gates are where model improvements go to die: a challenger that is clearly better on the headline metric gets blocked by a small regression on a slice or a few milliseconds of latency, the team stops training challengers, and the champion ages for years. The answer is not to drop the criteria but to make them explicit trade-offs with owners who can accept a regression on purpose — which is more organisational work than a single score, and that cost is real.
Where the depth lives
This domain teaches the model and hands the rest off by name.