Retraining as a Decision
Retraining is a change to a running system with a cost, a risk and a precondition. Four questions decide whether it is due; "drift" is not one of them.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The drift monitor fired, the model is six weeks old and the team is asking whether to retrain. How do you decide, and what makes the answer "no"?
We run a churn model that a retention team calls from every week. The dashboard shows two features drifting and the model was trained in the spring. Half the team says retrain now, the other half says the model is fine. Nobody can say which half is right, so we retrain every Monday to stop the argument.
Retrain on a schedule — weekly, say — on the most recent data. Fresh data is better data, the pipeline is automated, and a regular cadence means nobody has to make a judgement call. If drift is detected, retrain sooner.
The weekly retrain picks up a broken feature the week the plan-tier table was renamed. The feature is null for the whole training window, the model learns that it is uninformative, validation looks normal because validation has the same nulls, and the model has permanently forgotten a strong signal.
- The weekly retrain picks up a broken feature the week the plan-tier table was renamed. The feature is null for the whole training window, the model learns that it is uninformative, validation looks normal because validation has the same nulls, and the model has permanently forgotten a strong signal.
- The drifting features were marketing-driven shifts in the population that the model handles fine — ranking quality on arriving labels has not moved. Retraining on them changes nothing except the weights, and every weekly artifact is a fresh chance for a regression.
- The Monday retrain trains on the retention team's own successes. Subscribers who were called and stayed are negatives, so the model gradually learns that the profile of people-worth-calling is low-risk, and stops ranking them at the top.
- Training costs a GPU-day and a review, so a weekly cadence is a fixed cost with no measured benefit. When someone finally asks what retraining bought, nobody can answer, because the schedule never depended on a measurement.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict which subscribers will cancel in the next 30 days. The label is the cancellation event, observed 30 days after the prediction; a prediction made today cannot be scored until next month.
- The decision downstream is a ranked call list for a retention team with a fixed weekly capacity, so what matters is the ordering of the top few thousand scores, not the score of everyone.
- One example is one subscriber on one Monday, with usage aggregates over the trailing 7 and 30 days, plan tier, tenure and support history, joined point-in-time to whether they cancelled in the following 30 days.
- The training window is the trailing twelve months. Labels for the most recent 30 days do not exist yet, so the newest examples any training run can use are already a month old.
- The retention team's calls change outcomes: a subscriber who was called and stayed is a negative label that the model partly caused (Feedback Loops).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A trained model encodes the relationship between features and label as it existed in the training window. It goes stale only when that relationship changes — concept drift — or when the population moves into regions the model never learned. Feature drift alone does not imply either (Drift Is Not Failure).
- Whether retraining helps depends on four rates: how fast the world changes, how fast labels arrive, how fast measured performance decays, and how much a training run costs in compute, review and rollout risk. When labels arrive slower than the world changes, the freshest possible model is still trained on a world that has already moved.
- Retraining is a multiplier on whatever is in the pipeline. A clean pipeline and fresh labels produce a better model; a broken feature, a corrupted label or a feedback-loop-biased dataset produces a worse model that scores better offline, because validation shares the defect.
Four rates, one decision
A retraining policy is a comparison between rates. The world changes at some speed — customer behaviour, fraud tactics, the product itself. Labels arrive at some speed, fixed by what the label means: a click in seconds, a chargeback in months. Measured performance decays at some speed, which is the only one of the four you have to observe rather than know. And a training run costs something in compute, review and rollout risk.
When labels arrive slower than the world changes, no cadence helps much: every model is trained on outcomes from a world that has already moved, and the honest response is a leading indicator or a simpler model that decays more slowly. When labels are fast and the world is slow, retraining is cheap insurance and a schedule is fine. Most systems sit between, and there the decay curve decides.
The monitor shows drift and the model is six weeks old. What justifies a retrain?
when Top-k precision on the newest labelled window has fallen past the interval the metric's uncertainty allows, and label lag is normal so the window reflects the current model.
cost You ran a decaying model for the length of the label delay before you could be sure. The retrain must still beat the incumbent on the same window.
when A new segment, market or product surface entered the traffic, the model's scores on it are demonstrably miscalibrated, and enough labels for the segment exist to train on.
cost Retraining before the segment's labels arrive teaches the model the new features with the old outcomes — the failure the drift explorer at /ml/drift shows directly.
when A feature was broken and repaired, a label definition was corrected, or a leak was removed. The old model was trained on the defect and only a retrain removes it.
cost The new offline score may be lower than the old one — honestly lower, because the defect was inflating it — and the promotion gate must be told that.
when Never on its own. Drift with flat quality is the world moving inside the region the model handles; look, do not act.
cost Retraining here spends a training run and a rollout to change weights that were not the problem, and hands the pipeline a fresh chance to launder a defect.
Retraining as a multiplier
A retrain does not fix the pipeline; it re-expresses it. If a feature broke, the retrained model learns to live without it and the validation score — computed on the same broken feature — barely moves. If the label was redefined, the retrained model learns the new definition and reports a good score against it. If the retention team's calls changed outcomes, the retrained model learns that called-and-stayed profiles are safe.
This is why the precondition on retraining is pipeline health, not drift. The offline evaluation cannot see any of these, because it shares the training set's defect. The only thing that can is a test that runs before training: null rates, label-definition version, intervention logging.
Validation ranking quality of the retrained model is within noise of last week's — the pipeline is green, the artifact is promoted automatically.
Over the following month the retention team reports that the call list has stopped surfacing the high-tier subscribers it used to; churn among that tier rises.
- 1The plan-tier feature has been null since a column rename; the retrain learned it is uninformative and the validation set, which has the same nulls, could not object.
- 2The previous model still had the feature and would have kept ranking on it, had it not been replaced on schedule.
- 3Some genuine drift in the high-tier population may also have occurred, but it would not have been a step change on the retrain date.
What must hold for a retrain to help
Retraining is worthwhile only under an assumption most teams never write down: that the training set now available is a truer picture of the current world than the one the incumbent learned from. Fresher is not the same as truer. A fresh training set with a corrupted label, a missing feature or an intervention-biased outcome is a worse picture, and the retrain will faithfully learn it.
The assumption is checkable, and the checks belong before the training job, not after. A retrain that is allowed to start only when the pipeline is demonstrably healthy is the only kind that can be automated safely.
The newest labelled window reflects the current feature-label relationship more faithfully than the incumbent's training window did.
holds when Feature definitions, label construction and the serving path are unchanged since the incumbent was trained; null rates and label base rates are within their historical range; interventions driven by the model are logged and can be accounted for.
breaks when A feature silently broke, the label definition moved, the label join is lagging so the newest window is missing its positives, or the model's own decisions have reshaped the outcomes it will be trained on.
respond Block the retrain, fix the pipeline, and only then decide whether the incumbent needs replacing. A retrain is never the diagnosis.
1def retrain_allowed(candidate_window, incumbent):2 # incumbent.lineage records the feature + label definition versions it trained on3 if candidate_window.feature_defs != incumbent.lineage.feature_defs:4 return False, "feature definitions changed: retrain would compare unlike with unlike"5 if candidate_window.label_def != incumbent.lineage.label_def:6 return False, "label definition changed: fix or re-derive labels first"7 if candidate_window.label_lag_days > incumbent.lineage.label_lag_days * 1.5:8 return False, "labels are lagging: newest window is missing its positives"9 for name, null_rate in candidate_window.null_rates.items():10 if null_rate > incumbent.lineage.null_rates[name] + 0.05:11 return False, f"{name} null rate jumped: a broken feature would be learned as noise"12 decayed = candidate_window.topk_precision(incumbent) < incumbent.lineage.topk_precision - candidate_window.ci_halfwidth13 return decayed, "measured decay" if decayed else "no measured decay: nothing to fix"The last line is the actual decision; everything above it is the precondition. Notice that a retrain is refused when there is no measured decay — the gate says "no" more often than "yes", which is what a decision looks like and a schedule does not.
How to build it
Most important first.
- Make retraining a decision with a stated trigger: measured decay on arriving labels past a threshold, a known population change with enough new labels to learn from, or a pipeline fix that the old model must be retrained to benefit from. Write the trigger down before the first retrain (Retraining Strategies).
- Precondition every retrain on pipeline health — data and feature tests green, label definition unchanged, null rates normal — so that a retrain can never launder a broken feature into a worse model (Data & Feature Tests).
- Never promote a retrained model on its own validation score. Compare it against the incumbent on the same slice with the same threshold policy (Champion / Challenger) and roll it out gradually (Canary Rollout).
- Measure the effect of retraining. If two consecutive retrains changed nothing measurable, the cadence is too fast; if performance decays visibly between them, it is too slow. The decision tree at
/ml/decidewalks the "should we retrain?" branch.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Ranking quality at the retention team's capacity — precision among the top-k scores — on labels as they arrive, as a function of model age. This is the decay curve, and it is the number a retraining decision rests on.
- The delta between the retrained challenger and the incumbent on the same evaluation window. If it is inside the confidence interval, the retrain bought nothing (Metric Uncertainty).
- Feature drift magnitude is not the number. It is a prompt to look, not a reason to act.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Labels arrive fast enough that the decay curve reflects the current model, not the one from two months ago.
- The training pipeline that produces a retrained model is the same one, with the same feature definitions and label construction, that produced the incumbent — so a difference between them is a difference in data, not in code.
- The retention team's interventions are logged, so the training set can distinguish "stayed" from "stayed because we called".
- Offline: before retraining, run the incumbent on the newest labelled window and confirm decay is real and larger than the metric's uncertainty. Then run the challenger on the same window.
- Online: the retrained model goes out as a challenger on a slice, and the promotion decision is made on the slice's outcomes, not the offline delta.
- Over time: keep the decay curve per model version. A retraining policy is right when the curve is cut off at about the point the decay becomes costly, and not much before.
What can go wrong
- The decay curve is flat because labels have stopped arriving, not because the model is fine. Label lag has to be plotted next to it (Ground-Truth Delay).
- A retrain is triggered by real decay, but the decay was caused by a feature that broke; the retrained model is trained without the feature and scores well on validation that lacks it too. The trigger fired correctly and the response made it worse.
- The team retrains, the challenger loses to the champion, and nobody investigates why — a retrain that loses is a signal that something in the data changed, and it is discarded as noise.
- A decision-based policy needs someone to make the decision, with the monitoring to inform it. A schedule needs neither, which is why it is so common.
- Waiting for enough labels to confirm decay means running a decaying model for the length of the label delay. For a 30-day label that is a month of degraded calls.
- Comparing every retrain against the incumbent doubles the evaluation work and slows the cadence. That is the point, and it is also a cost.
- "Drift means retrain." Feature drift means the inputs moved. Whether the model's performance moved is a separate measurement, and retraining on drifted inputs with stale labels can make the model worse.
- "Fresh data is always better." Fresh data with a broken feature, a redefined label or an intervention-biased outcome is worse data, and a retrain bakes it in while the validation score improves.
- "We retrain weekly, so staleness is not a problem." Weekly retraining on 30-day labels produces a model trained on a world at least a month old, every week. The cadence does not shorten the label delay.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a retrain is only as good as the data it retrains on holds for every model family and task; the four rates — world change, label arrival, decay, cost — are the decision inputs everywhere, though their values differ by orders of magnitude between domains.
- DOMAIN-SPECIFICAd click models decay in days and get labels in seconds, so near-continuous retraining is right; a credit model gets labels in a year and the world moves slowly, so a retrain every quarter is already aggressive. The same policy cannot fit both.
- CONTESTEDA serious position holds that a fixed, frequent schedule is the better default: it makes retraining boring, exercises the pipeline so it cannot rot, catches decay without anyone having to notice it, and a gated promotion step (challenger must beat champion) removes the risk of shipping a worse model. That is a strong argument when labels are fast and training is cheap; it is weakest when labels lag, because then the schedule mostly retrains on stale outcomes and exercises a pipeline whose defects it cannot see.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — the pre-training gate is a test suite that decides whether a change may proceed, and the discipline of keeping it strict when it blocks a retrain someone wanted is a reliability-culture question this domain assumes.