Promotion Is a Checklist, Not a Score
A challenger is promoted on quality, latency, memory, cost, robustness and — where relevant — fairness, compared against the champion on the same slice with the same threshold policy. One improved offline number is not a reason to ship.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The retrained model has a better validation score than the one in production. What else has to be true before it replaces the champion?
A lending team's weekly retrain produced a challenger with a visibly higher validation AUC than the champion. It was promoted the same afternoon. Within days the decision service began timing out at peak, the approval rate for one applicant segment dropped sharply, and the on-call engineer rolled back to a champion that was, by the one number everyone had looked at, worse.
Compare the headline metric. If the challenger beats the champion on validation, promote it — that is what the metric is for, and waiting costs the business the improvement every day it is not live.
The challenger is larger and slower; p99 latency at peak crosses the decision service's timeout and the fallback path denies applicants the model would have approved (Latency Breakdown, Serving Fallbacks).
- The challenger is larger and slower; p99 latency at peak crosses the decision service's timeout and the fallback path denies applicants the model would have approved (Latency Breakdown, Serving Fallbacks).
- The comparison used each model's default threshold. The champion's threshold was tuned to a target approval rate; the challenger's was not, so the "improvement" in the ranking metric arrived with a very different approval rate (Threshold Selection).
- The headline metric improved on the whole set and worsened on one applicant segment; nothing in the pipeline reported per-segment numbers, so the regression surfaced as a complaint (Evaluation Slices, Fairness).
- The AUC difference was within the interval a bootstrap over the validation set would have given; the "better" model may not be better at all (Metric Uncertainty, Beating the Baseline).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a loan applicant will default within twelve months; the label arrives a year late, so evaluation uses a historical cohort whose outcomes are complete.
- The prediction becomes a decision through a threshold and a policy; the business cares about approval rate, default rate among approved, latency of the decision, and treatment across applicant segments.
- One example is one historical application with its features as of application time and its default outcome. The validation split is the most recent cohort with complete outcomes, which is over a year old.
- The challenger is a larger gradient-boosted ensemble with three new aggregation features; the champion is a smaller ensemble served for eight months, with a threshold chosen to hit a target approval rate.
- Both were evaluated by the training pipeline, which reports one headline metric on the whole validation set at the default threshold.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A model in production is a bundle of properties, and the offline quality metric is one of them. The decision service has a latency budget; the host has a memory ceiling; the cost per prediction has a budget; the policy has an approval-rate target and a fairness constraint; the input distribution has edges the model must survive. A challenger can improve the first and break any of the others.
- A champion comparison is only meaningful when both models are evaluated on the same examples, at thresholds chosen by the same policy, with the metric that maps to the decision. Different thresholds compare different decisions; different slices compare different populations; a metric without an interval cannot distinguish improvement from noise.
- The promotion gate is therefore a conjunction of checks, each with a stated pass condition, recorded on the registry entry (The Model Registry). A gate that can be passed by one number is not a gate.
- Some of the checks cannot be run offline. Latency under real concurrency, memory under real batch shapes and behaviour on live input edges need the serving image and live traffic — shadow or canary (Shadow Deployment, Canary Rollout) — before Production.
One number, six properties
The challenger improved the metric the pipeline reports. The pipeline reports one because one is easy, and the promotion decision was made on it because it was the only number in the room. Every property that then failed — latency, approval rate, segment treatment — was a property of the model that no one had measured.
The matrix makes the shape of the decision visible: quality is one column among several, and a challenger that wins it and loses two others is not an improvement, it is a different set of trade-offs. The scores are illustrative; what the matrix cannot say is which column is a hard constraint in this domain, and that is the first thing the gate has to state.
| Option | Quality | Latency | Cost | Interpretability | Operational | Note |
|---|---|---|---|---|---|---|
| Champion (smaller ensemble, tuned threshold) | Known behaviour for eight months, fits the latency budget, threshold matched to the approval-rate policy. | |||||
| Challenger (larger ensemble, new features) | Better ranking on the validation cohort; slower, heavier, three new features that need serving support, threshold never matched to policy. | |||||
| Challenger, re-thresholded and pruned | Same weights with the policy threshold applied and the two least useful new features dropped; a plausible candidate the gate would actually evaluate. |
caveat The scores cannot express that latency is a hard constraint here — a score of 2 that crosses the timeout is a failed gate regardless of every other column — nor that the quality difference is only meaningful with an interval and at the policy threshold, which the matrix silently assumes.
Same slice, same threshold, or it is not a comparison
The champion decides at a threshold chosen to hit an approval-rate target. The challenger was reported at its default. Comparing them is comparing two different policies, and the ranking metric hid that because it does not have a threshold at all. Applying the same policy to both — choose each model's threshold to hit the target approval rate on the same cohort, then compare default rate among approved — produces two confusion matrices that can be read side by side.
The matrices below are illustrative counts for the shape of the argument. What they show is the thing the headline metric could not: at matched approval rate, the challenger's gain in caught defaults is real but modest, and whether it clears the interval is the question the gate has to answer with a bootstrap, not by eye.
Numbers are illustrative. Read this beside the champion's matrix at the same approval rate; the difference in the top row is the comparison, and it needs an interval before it is a decision.
The gate as a checklist
A gate is a list of conditions, each with a pass rule stated before the challenger exists, each evaluated on champion and challenger together, each recorded as evidence on the registry entry. The list below is a starting point; the domain decides which items are hard constraints and which are informational.
The point of writing it down is that it can then be automated, reviewed and audited. A gate that lives in one engineer's judgement is passed by whoever wants to ship, and the rollback afterwards is how the missing item gets discovered.
Champion and challenger were evaluated side by side on the same cohort and slices, at thresholds chosen by the same policy, in the serving image under peak-shaped load, at the time of promotion.
holds when The gate re-runs the champion rather than reading its stored numbers, the policy threshold is applied to both, and the latency and memory items run in the serving image with production-shaped batches.
breaks when The champion's year-old registry metrics are reused; each model is compared at its own default threshold; the latency check runs on a quiet afternoon; a slice regression is dismissed without an interval.
respond Re-run the gate properly before deciding; a rollback triggered by a constraint the gate never checked means the gate is missing an item, not that the challenger is bad.
PROMOTION GATE — ticket: challenger v38 vs champion v37, cohort holdout-2026-08 [ ] decision metric at policy threshold, with 95% bootstrap CI on the difference (constraint) [ ] same, on each named slice: region, applicant age band, product (constraint: no slice regresses beyond CI) [ ] p99 latency in serving image at peak-shaped concurrency <= budget (constraint) [ ] peak RSS memory <= host ceiling with headroom (constraint) [ ] cost per 1k predictions vs champion (informational unless > +20%) [ ] robustness suite: perturbations, missing-field, out-of-vocabulary (constraint: no crash, bounded score shift) [ ] fairness constraint at policy threshold across protected segments (constraint, domain-mandated) [ ] feature-definition version matches serving (constraint) [ ] replay sample reproduces in serving image (constraint) [ ] shadow: prediction distribution vs champion, 7 days (evidence) [ ] sliced canary at 5% with auto-rollback conditions (evidence) Reviewer (not the author): ________ Evidence attached to registry entry: [ ]
How to build it
Most important first.
- Write the gate as a checklist with pass conditions before the challenger exists: the decision metric at the policy threshold with an interval, the same on every named slice, p99 latency in the serving image under peak-shaped load, peak memory, cost per thousand predictions, robustness on the perturbation suite (Robustness Testing), and the fairness constraint where the domain has one.
- Run the champion through the same checklist on the same data at the same time. The champion's stored numbers are a year old and a different validation set; only a fresh side-by-side comparison is a comparison.
- Compare at thresholds chosen by the same policy — the approval-rate target, or the cost-optimal point — not at each model's default. Report the interval on the difference, not just the point estimates.
- Attach the checklist result to the registry entry as the evidence for the Staging → Production transition, and make the transition impossible without it.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The difference in the decision metric at the policy threshold, with a confidence interval, on the validation cohort and on every named slice. This is the number that decides quality; a headline ranking metric on the whole set is not.
- p99 latency and peak memory in the serving image at peak concurrency, against the budget. These decide whether the challenger can be served at all.
- Not "did the validation AUC go up". It is a component of the checklist, it does not carry a threshold, and by itself it does not map to any decision the business makes.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The validation cohort and the named slices still resemble the population the model will decide on; if the applicant mix has moved since the cohort closed, the comparison is about a population that no longer applies.
- The threshold policy used in the gate is the policy serving applies, and the latency and memory measured in the gate were measured in the serving image under peak-shaped load.
- The champion's numbers in the comparison are fresh — computed on the same data at the same time as the challenger's, not read from the registry entry it was promoted with.
- Offline: the full checklist on champion and challenger side by side, on the same cohort and slices, at policy thresholds, with bootstrap intervals on every difference.
- Online: shadow the challenger on live traffic for latency, memory and prediction-distribution comparison against the champion; then a sliced canary on a fraction of decisions with an automatic rollback condition.
- Over time: when the twelve-month outcomes for the canary period arrive, compare realised default rates between the two arms; that is the only measurement of the thing the gate was estimating (Ground-Truth Delay).
What can go wrong
- The checklist exists and every item is marked "pass" by the same engineer who wants to ship; the gate has become a ritual. Automate the items that can be automated and require a second reviewer for the rest.
- The fairness check passes on the historical cohort and the challenger's new aggregation features shift a segment in live traffic that the cohort under-represents; the canary catches it only if the canary's metrics are sliced.
- The latency check is run on a quiet afternoon at low concurrency and passes; peak traffic has a different batch shape (Inference Batching).
- The champion is never re-evaluated, so a champion that has decayed (Performance Decay) is compared to its own year-old number and the challenger looks like no improvement when it is a large one.
- A checklist gate is slower than a number, and an improvement that is real waits weeks for shadow and canary evidence; the cost of that delay is a legitimate line in the decision.
- Re-evaluating the champion on every promotion doubles the evaluation cost and occasionally reveals the champion has decayed, which starts an incident nobody was looking for.
- Slicing every metric multiplies comparisons, and with enough slices one will regress by chance; the gate has to state in advance which slices are constraints and which are informational.
- "If offline AUC improved, ship it." AUC has no threshold and no latency, and it was measured on one cohort at one time. It is a reason to run the checklist, not a substitute for it.
- "The challenger lost on one slice, so it is unfair — reject it." One slice regressing within its interval is noise; the question is whether the regression is outside the interval and whether that slice is a stated constraint.
- "The rollback proves the champion was better." The rollback proves the challenger violated a constraint that was never in the gate. Add the constraint; the challenger may still be the better model once it fits the latency budget.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a production model has properties beyond the offline metric — latency, memory, cost, behaviour on slices and edges — holds for every model family and task; only which properties are binding constraints varies.
- DOMAIN-SPECIFICIn lending, hiring and healthcare the per-segment comparison is a legal constraint and belongs in the gate as a hard condition; in ad ranking it is usually informational, and the binding constraints are latency and cost per prediction instead.
- CONTESTEDA serious position holds that heavyweight gates slow iteration to the point that the compounding value of frequent small improvements is lost, and that a fast automated gate on two or three metrics plus a canary with automatic rollback gives most of the safety at a fraction of the delay. That is right where rollback is cheap and outcomes are fast; it fails where a wrong decision cannot be undone — a declined loan, a missed diagnosis — and where the outcome that would trigger rollback arrives months later.
Where the depth lives
This domain teaches the model and hands the rest off by name.