Retraining & Rollout
Retraining as a decision rather than a schedule, champion/challenger, shadow, canary, A/B — and rollback and fallback, which every serving system needs before it needs a second model.
Retraining is a change to a running system with a cost, a risk and a precondition. Four questions decide whether it is due; "drift" is not one of them.
Scheduled, drift-triggered, performance-triggered, manual and continuous: each is right for a particular ratio of label speed to world speed, and continuous training has risks the others do not.
A candidate earns production by beating the incumbent on the same traffic under the same threshold policy, on more than one number. A better validation score is a nomination, not a promotion.
The candidate scores production inputs but controls nothing. It catches skew, latency and crashes before a user sees them — and it cannot measure business impact, because it never makes a decision.
Give the candidate 1% of decisions, then 5%, 25%, 100%, watching quality, latency, cost and errors at each step — with the honesty that a 30-day label makes a 30-day canary.
The only measurement of business impact is to let two models decide for two comparable populations and compare what happens — with stable assignment, guardrails, enough sample, and honesty about interference and about the users in the experiment.
When the model is wrong, go back; when the model is gone, degrade. Rollback must restore the feature definition with the artifact or it reintroduces skew, and the previous model has to be warm.