Retraining Strategies
Scheduled, drift-triggered, performance-triggered, manual and continuous: each is right for a particular ratio of label speed to world speed, and continuous training has risks the others do not.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Which retraining trigger fits this system, and what does continuous training cost that a scheduled retrain with a human gate does not?
We have five models in production and each team picked its own retraining approach. The ads team retrains hourly and never looks at it; the credit team retrains when someone remembers; the fraud team retrains when the drift alert fires, which is most days. The platform team wants one policy for everyone.
Pick the safest-sounding option for everyone: retrain when drift is detected. It responds to change, it does not waste compute when nothing moved, and it can be fully automated from the monitoring the platform already has.
The fraud drift alert fires most days, because a live fraud system's inputs genuinely move. Drift-triggered retraining becomes daily retraining on 90-day labels, so every model is trained on outcomes older than the drift that triggered it.
- The fraud drift alert fires most days, because a live fraud system's inputs genuinely move. Drift-triggered retraining becomes daily retraining on 90-day labels, so every model is trained on outcomes older than the drift that triggered it.
- The credit model never drifts enough to trigger and quietly decays for a year, because the relationship changed — concept drift — while the inputs stayed put (Concept Drift).
- The ads team's hourly continuous training has no human gate. A logging change at 03:00 zeroes a feature; by 04:00 the model has learned to ignore it and the pipeline is green.
- Nobody can compare strategies afterwards, because none of the triggers left a record of what fired, what the model looked like before and what changed after.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Five different labels with five different arrival times — a click within seconds, a chargeback within 90 days, a default within a year. The retraining strategy is a function of the label, not of the team.
- The surrounding decision differs too: an ad ranking regenerated every request, a credit limit set once and rarely revisited, a fraud queue with fixed reviewer capacity.
- The ads model has a training set that grows by millions of labelled rows an hour; the credit model gains a few thousand labelled rows a quarter and most of them are from a year ago.
- The fraud model's drift alert fires on feature distribution against a training set that is refreshed every retrain, so each retrain resets the baseline the alert is computed against.
- None of the five records why a given retrain happened, so there is no way to say which strategy has been working.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Every strategy is a rule for when to start a training job, and the rule is only as informative as its input. A schedule uses the clock. A drift trigger uses input distributions. A performance trigger uses arriving labels. A manual trigger uses a person's judgement. Continuous training uses every new batch of labels and skips the question.
- The input decides the failure. The clock cannot see decay, so a schedule is too fast or too slow. Drift cannot see whether the model is wrong, so a drift trigger fires on harmless shifts and misses concept drift. Labels see everything but arrive late, so a performance trigger is the most accurate and the slowest.
- Continuous training removes the human gate and shortens the loop between what the model did and what it learns next. Both are the appeal and the danger: a defect enters the model within one cycle, a feedback loop closes faster (Feedback Loops), and by the time labels reveal the problem several generations of models have compounded it.
Five triggers, five inputs
The strategies differ in what they read, and what they read is what they can respond to. Put next to each other, none dominates: the accurate one is slow, the fast one is blind, the automated one has no judgement and the judged one has no schedule.
The platform team's request for one policy is answered by a rule for choosing, not a choice: read the label delay, the cost of a bad model and the training cost, and pick from the table.
What should start a training job for this model?
when Labels are reasonably fast, training is cheap, and the main value is keeping the pipeline exercised and the model from silently ageing. Good as a floor under any other trigger.
cost Blind to decay: too frequent when the world is stable, too slow when it moves. Every run is a rollout risk with no measured reason.
when Feature drift has been shown, for this model, to precede decay — and the trigger is on the features where that holds, against a pinned baseline.
cost Fires on harmless shifts, misses concept drift, and in a live system fires constantly. Retrains on labels older than the drift that triggered it.
when Labels arrive in days or weeks and the cost of running a decayed model for that long is tolerable. The most accurate trigger, because it reads the thing that matters.
cost Waits a full label delay before it can know anything. Needs a metric with tight enough intervals that noise does not trigger it.
when A person with context — a product change, a new market, a known pipeline fix — knows something no monitor does. Always available as an override.
cost Depends on someone remembering and being right. Unreproducible; leaves no trigger record unless forced to.
when Labels arrive in seconds to hours at volume, mistakes are cheap and reversible, and automated gates can stand in for the human — ads ranking, short-horizon recommendation.
cost No human gate; a defect is learned within one cycle; feedback loops close faster; per-step gates cannot see compounding across generations.
What continuous training compounds
A continuous loop retrains generation N+1 on labels produced under generation N. When the model's decisions shape those labels — which ads were shown, which items were recommended, which users were called — each generation learns from a world the previous one filtered. The per-step change is small and passes every gate; the drift across twenty generations is large and has no gate at all.
The same loop carries defects. A logging change that zeroes a feature is learned as "this feature is uninformative" within one cycle; the next generation is trained on the previous generation's predictions having already adapted to the loss. By the time a label-based metric shows the damage, several generations deep, rolling back one step recovers nothing.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A feature is zeroed by an upstream change at 03:00 | Pipeline green; hourly challenger beats incumbent by noise; quality against labels falls days later | Each generation trained on the null feature; validation shares the null; the gate compares against a model that had already adapted | Pipeline-health gate before training; compare against a pinned reference model, not the previous generation |
| Recommendations narrow toward what the model already shows | Engagement flat or rising short-term; catalogue coverage shrinking; new items never surface | Labels are produced under the model's own exposure policy; each generation learns the previous one's preferences (Feedback Loops) | Log exposure, weight by propensity or hold out an exploration slice (Exploration vs Exploitation); monitor coverage as a guardrail |
| Slow one-way population shift | No gate ever fails; the model a month later scores far below the one from a month ago on a fixed reference window | Every step is inside tolerance; the drift baseline resets each retrain | Evaluate every generation against a fixed reference window and alert on the trend |
| Label pipeline stalls | Training set stops growing; each cycle retrains on the same rows; predictions jitter between generations | The trigger reads "new batch available" rather than "new labels available" | Gate on label-lag age; hold the incumbent until labels resume |
The assumption every trigger shares
Whatever fires the retrain, promotion assumes the comparison is fair: challenger against champion, same slice, same window, same threshold policy, against a reference that does not move. A trigger that compares against the previous generation compares against a moving target, and the cumulative direction of travel is invisible.
The assumption is easy to state and easy to lose in automation, where "beat the last model" is the obvious thing to code.
Every retrained model is evaluated against a pinned reference — the current champion on a fixed window — so that improvement and decay are measured in absolute terms.
holds when The promotion pipeline stores the reference window and the champion's scores on it, and re-scores the challenger on exactly that window; the reference is moved only by an explicit decision.
breaks when The pipeline compares against the previous generation, the reference window is refreshed automatically with each retrain, or the drift baseline is reset to the newest training set.
respond Pin the reference; re-score the last several generations against it; if the trend is down, roll back to the best generation rather than retraining forward.
1model: churn-v32trigger:3 floor: { schedule: "0 6 * * 1" } # weekly, so the pipeline is exercised4 performance:5 metric: precision_at_k6 k: 50007 window_days: 288 fire_when_below: reference.precision_at_k - reference.ci_halfwidth9 manual: allowed10 drift: prompt_only # opens a ticket; never starts training11preconditions:12 - feature_null_rates_within_lineage13 - label_definition_unchanged14 - label_lag_days_max: 3515promotion:16 compare_against: champion # never "previous"17 reference_window: pinned # moved by explicit decision only18 threshold_policy: same_as_champion19 rollout: canaryThe two lines that matter are drift: prompt_only and compare_against: champion. The first stops the most common trigger from starting a training job; the second stops the most common automation from measuring against a moving target.
How to build it
Most important first.
- Choose the trigger from the label delay and the cost of a bad model. Fast labels and cheap mistakes: continuous, with automated gates. Slow labels and expensive mistakes: performance-triggered with a human decision, and a schedule as a floor so the pipeline is exercised.
- Whatever the trigger, the promotion gate is the same: the new model is a challenger and must beat the incumbent on the same slice with the same threshold policy (Champion / Challenger, Promotion Is a Checklist, Not a Score).
- Record every retrain — trigger, data window, lineage, comparison result — so that strategies can be compared on evidence after a year (Model Lineage).
- For continuous training, add the gates that the missing human would have applied: pipeline health before training, invariant tests on the artifact (Model Invariant Tests), and a hard stop when the new model's predictions move more than a bound from the previous generation.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per strategy: the fraction of retrains that produced a challenger that beat the incumbent. A strategy whose retrains rarely win is firing on the wrong signal.
- Time from a real decay beginning to the model that fixes it being in production. This is the number a strategy is supposed to minimise, and it includes the label delay whatever the trigger.
- The number of retrains per month is a cost, not a quality measure. More is not better.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The signal the trigger reads is the signal the model's usefulness actually depends on — labels for a performance trigger, and for a drift trigger only where drift has been shown to precede decay.
- The label delay is short enough, relative to how fast the world changes, that a retrain can ever catch up; when it is not, the strategy is a leading indicator or a more stable model, not a faster trigger.
- Each retrain is compared against a fixed reference — the champion, or a pinned baseline — rather than against whatever the previous retrain produced.
- Offline: replay the last year of labels and simulate each strategy — when would it have fired, and what would the model trained at that point have scored on the following period? The drift explorer at
/ml/driftdoes this on six scenarios. - Online: run the chosen trigger for a quarter with a schedule as a floor and record every firing. Compare the win rate of triggered retrains against scheduled ones.
- Over time: plot performance per model generation against a fixed reference window, not against the previous generation, so slow compounding shows as a trend.
What can go wrong
- A drift trigger whose baseline is the previous training set resets itself on every retrain, so a slow one-way shift never fires: each retrain moves the baseline to where the drift already is.
- A performance trigger on a metric with wide confidence intervals fires on noise and retrains into a new random draw (Metric Uncertainty).
- Continuous training with a promotion gate that compares the challenger against the previous hour's model: a slow degradation passes every gate because each step is small, and the model a month later is far worse than the one from a month ago with no gate having failed.
- Performance-triggered retraining is the most accurate and waits the longest — a full label delay before it can act.
- Continuous training gives the fastest response to real change and the fastest propagation of a defect; every gate added to make it safe makes it less continuous.
- A manual trigger has the best judgement and the worst availability — it retrains when someone remembers, and it is the one strategy that cannot be reproduced.
- "Continuous training is the most advanced option, so it is the goal." It is the right option when labels are fast and mistakes are cheap. For a credit model it is a way to learn from a year-old world every hour.
- "Drift-triggered retraining only retrains when necessary." It retrains when the inputs move, which in a live system is often, and does not retrain when the relationship moves with the inputs still — the case that most needs it.
- "One policy for all five models." The label delays differ by four orders of magnitude. One policy is either far too aggressive for credit or far too slow for ads.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- DOMAIN-SPECIFICThe ratio of label delay to world change is the deciding input, and it ranges from seconds-over-days in ads to years-over-years in credit; a strategy that is right for one is wrong for the other by construction.
- SCALE-SPECIFICContinuous training is only viable when new labelled data arrives in volume — millions of rows an hour; with a few thousand labels a quarter it retrains on the same data repeatedly and only the random seed changes.
- CONTESTEDPractitioners who run continuous training at scale argue that its risks are the same risks a scheduled retrain has, merely faster, and that a faster loop with automated gates catches a defect in hours where a monthly retrain lets it sit unnoticed for weeks; the human gate, they note, mostly rubber-stamps. That is fair where labels are fast; the counter-argument is that the human gate is the only stage that can notice a defect the automated gates were not written for, and compounding across generations is a failure the per-step gate cannot see.
Where the depth lives
This domain teaches the model and hands the rest off by name.