Rollback & Fallback
When the model is wrong, go back; when the model is gone, degrade. Rollback must restore the feature definition with the artifact or it reintroduces skew, and the previous model has to be warm.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A model deployment is hurting the product. How do you go back in one step, and what does the product do while there is no model at all?
Last quarter a bad model went out and rolling back took four hours: the previous artifact had been evicted from the registry cache, the feature service had been upgraded in the same release, and when the old model came back it scored on the new features and was wrong in a different way. Then the model server crashed and every page that used it errored.
Rollback is a deploy: point the serving config at the previous artifact version and redeploy. Fallback is an error handler: if the model call fails, show an error or an empty page. Both are standard deployment concerns and the platform handles them.
The previous artifact is in the registry but not on any serving node; it has to be pulled and loaded, which for a large model is minutes to tens of minutes, during which the bad model keeps serving.
- The previous artifact is in the registry but not on any serving node; it has to be pulled and loaded, which for a large model is minutes to tens of minutes, during which the bad model keeps serving.
- The previous model comes back and scores on the upgraded feature service. It was trained on the old normaliser, so the feature it receives is on a different scale. The rollback replaced one wrong model with another, and the "known good" version is now also bad (Train / Serve Skew).
- The model server crashes. The error handler shows an empty page. The nightly default ranking exists in a cache twenty milliseconds away and nothing reads it.
- The rule-based fallback the team wrote a year ago has never run in production. When it finally does, it references a feature that was renamed, and it errors too.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- A ranking model for a product listing page; the decision is the order of items. With no model, the page still needs an order.
- The system-level target is availability of the page and the quality of what it shows, in that priority. A default ordering that loads beats a great ordering that does not.
- The registry holds model artifacts by version, each with a lineage record naming the feature-definition version, preprocessing version and training data snapshot it was built on (Model Lineage).
- The feature service is versioned independently and was upgraded — a new normaliser on one feature — in the same release as the model. The previous model was trained against the old normaliser.
- A nightly batch precomputes a default ranking per category, stored in a cache the page can read without calling the model.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A model version is a triple — artifact, feature definitions, preprocessing — and the model's predictions are correct only for the triple it was trained as. Rolling back the artifact alone leaves the other two at their current versions and produces a mismatch the previous model never saw (Feature and Model Versioning, Preprocessing Lives in the Artifact).
- Rollback speed is decided by what is already loaded. A warm previous model — resident in memory on the serving fleet, or on a standby fleet — makes rollback a routing change measured in seconds. A cold one makes it a deploy, and a deploy of a large model is minutes at best.
- Fallback is a different question from rollback: not "which model" but "what if none". The options form a ladder of decreasing quality and decreasing dependency — previous model, rule-based scorer, cached prediction, default ranking, no prediction — and which rung is right depends on what the decision is and what a wrong or absent decision costs (Serving Fallbacks).
Rolling back the triple
The four-hour rollback failed for a reason that had nothing to do with deployment speed. The previous model was trained against a feature normaliser that no longer existed in the feature service. Restored onto the new normaliser, it received a feature on a different scale and was wrong — not the same wrong as the bad model, but wrong, and now the team had two bad models and no known-good one.
A rollback target is therefore a lineage record: artifact, feature definitions, preprocessing. If the feature service was upgraded with the model and cannot be reverted, then the previous model is not a valid target, and the rollback plan for that release has to be a fallback rung instead. That is a fact to know before the release, not during the incident.
The fallback ladder
When there is no model — a crash, a timeout, a feature store outage — the product still has to decide. The options are a ladder from most to least model-like, and each rung trades quality for independence from the thing that just failed. The previous model needs the serving fleet; a rule scorer needs the features; a cached prediction needs the cache; a default ranking needs nothing but the catalogue; no prediction needs nothing and gives nothing.
Which rung a decision can tolerate is a product question the model team has to ask. A listing page can show a default order with almost no user-visible cost. A fraud decision cannot approve everything, and "review everything" overwhelms the queue in an hour. The failure simulator at /ml/failures has a "kill the model server" control whose diagnosis ends where this lesson begins: the business metric fell because the product had no fallback.
The model is unavailable. What does the product do for this decision?
when The current model is wrong but the serving path is healthy; the previous lineage record is valid against the current feature service; it is warm.
cost A second model held resident; a coupling between model and feature-service releases so the previous triple stays valid.
when The model server is down but features are available; a handful of features carry most of the signal; a wrong-but-reasonable decision is tolerable.
cost A second scorer to maintain that drifts from the model's behaviour; must be exercised in production or it rots.
when The entity was scored recently and the decision tolerates staleness — a user's risk tier from an hour ago, a listing's quality score from last night.
cost A cache with a freshness policy; wrong for entities the cache has never seen; wrong after the world changes faster than the cache refreshes.
when The decision has a sensible model-free answer — popularity order, category default, "approve" for low-value transactions — and the cost of that answer is small.
cost Users get a generic experience; for a decision with asymmetric costs the default may be the expensive side.
when A wrong decision costs more than no decision — a medical flag, a large credit line — and the caller can wait or route to a human.
cost The feature is unavailable for the outage's duration; the human queue absorbs the load or the user is told to try later.
Warm, exercised, and logged
A rollback plan that has not been run is a guess. The previous model has to be resident, its features have to be populated, the routing change has to be tested — and the cheapest way to keep all three true is to route a small fraction of production traffic through the rollback path continuously. The same holds for every fallback rung: a rule scorer that has never run in production will fail the first time it does.
And every decision has to say which rung produced it. A prediction log that records "score 0.42" without recording that the score came from the cache after a timeout hides a degraded system inside a healthy-looking one — the failure simulator's slow-feature-store control is exactly this, and the fix it names is a fallback counter you can alert on.
The previous lineage record is valid against the current feature service, its artifact is warm, and switching to it is a routing change that takes seconds.
holds when The feature service and preprocessing versions the previous model needs are still deployed or can be reverted with it; the artifact is resident on the fleet or on a standby; a continuous sliver of traffic flows through the path and its outputs are checked.
breaks when The feature service was upgraded in the same release and cannot go back; the previous artifact was evicted from the serving cache to make room; the standby fleet was scaled to zero to save cost; the threshold config was not versioned with the model.
respond If the previous triple is not servable, promote a fallback rung to the rollback plan for this release and say so in the release notes; do not discover it during the incident.
1release: ranking-2026-08-272serve:3 model: ranking-v84 features: v55 preprocessing: v56 threshold_policy: top_k_207rollback:8 target: { model: ranking-v7, features: v4, preprocessing: v4, threshold_policy: top_k_20 }9 features_v4_still_servable: true # checked at release time; if false, rollback is invalid10 keep_warm_days: 35 # at least the label delay11 continuous_traffic_pct: 0.512fallback:13 on_timeout: rule_scorer # exercised at 0.2% of traffic continuously14 on_error: cached_default_ranking15 log_rung: true # every decision records which path produced itTwo lines carry the lesson: features_v4_still_servable, which turns "we can roll back" from a belief into a checked fact, and log_rung, without which a fleet serving from cache after a timeout looks like a fleet serving predictions.
How to build it
Most important first.
- Version the triple. A rollback target is a lineage record, not an artifact id, and the rollback restores the feature-definition and preprocessing versions with the weights. If the feature service cannot be rolled back with the model, the previous model is not a valid rollback target and the plan needs a fallback rung instead.
- Keep the previous model warm for at least the length of the label delay after a promotion, so a rollback triggered by delayed outcomes is still a routing change. Test the rollback path on every release by actually routing a sliver of traffic through it.
- Choose fallback rungs per decision, and make each explicit in the prediction log so degraded decisions can be counted: a served-from-cache ranking is a different event from a model ranking and both must be visible (Prediction Logging).
- Exercise every fallback rung in production on purpose — a small fraction of traffic, continuously — so the rung that has never run cannot be the one you discover is broken during an outage.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Time from rollback decision to the previous model serving the majority of traffic. Seconds means warm; minutes means cold; hours means the plan was a document.
- After a rollback: the restored model's prediction distribution against its own last-known-good distribution. If it differs, the rollback restored the weights and not the features.
- Fallback rate per rung, continuously. A fallback rate of zero means the rung has never been exercised, not that the system is healthy.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The lineage record for every deployed model names the feature-definition and preprocessing versions it requires, and the rollback mechanism restores all three together.
- The previous model and its feature path stay warm and routable for the length of the label delay after a promotion.
- Each fallback rung is exercised in production continuously, and the prediction log distinguishes which rung served each decision.
- Offline: for every release, a rehearsal that rolls back to the previous lineage record in a staging environment and confirms the restored model's predictions match its last-known-good predictions on a fixed sample.
- Online: a small, continuous fraction of traffic routed through the previous model and through each fallback rung, with their outputs logged and their error rates alerted on.
- Over time: a drill per quarter that rolls back for real and measures the time to majority traffic; the number is recorded and its trend watched.
What can go wrong
- The previous model is warm but its features are not: the feature cache was repopulated for the new model's feature set and the old model's features miss, so the rolled-back model runs on defaults.
- The fallback cache holds yesterday's default ranking and the catalogue changed at midnight; the fallback serves items that are out of stock. Degraded is fine; wrong is not.
- The rollback restores the model and the feature definitions, and the threshold or post-processing config is left at the new value, so the restored model's scores are turned into decisions by the wrong policy (Threshold Selection).
- Keeping the previous model warm is a second model's worth of memory or a standby fleet, for weeks, for an event that may not happen.
- Versioning the triple and rolling it back together couples the model release to the feature-service release, which is exactly the coupling teams split services to avoid.
- Exercising fallbacks continuously means a small fraction of users always get a degraded decision on purpose, so that the rest can get one when it matters.
- "We can always roll back to the previous version." To the previous artifact, in minutes, onto the current features — which may be the wrong features for it. The previous version is the triple, and it is only a rollback target if all three can go back.
- "The fallback is to show an error." An error is the bottom rung. A cached ranking, a rule, a default order all sit above it and cost almost nothing to have ready; the question is which rung the decision can tolerate, not whether to have one.
- "Rollback is a DevOps concern; the model team just supplies artifacts." The DevOps rollback restores a deployable. The model team has to say what the deployable is — and it is not the artifact file.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a model is only correct for the feature and preprocessing versions it was trained with, so rolling back weights alone reintroduces skew, holds for every model family; the fallback ladder is general in shape and specific in which rung a given decision can tolerate.
- TASK-SPECIFICA ranking has a natural fallback — a default order — and a recommendation has popularity; a fraud decision has "approve" or "review everything", each with a real cost; a regression used for pricing may have no acceptable fallback except the previous model, which makes keeping it warm mandatory rather than prudent.
- SCALE-SPECIFICFor a small model, warm means a second process on the same fleet and is nearly free; for a large model the previous version is a second set of GPUs held idle for weeks, and the cost of warmth has to be weighed against the cost of a slow rollback.
Where the depth lives
This domain teaches the model and hands the rest off by name.