Serving Fallbacks
When the model or its features are unavailable, the system must return something defined: the previous model, a rule, a cached score, a default ranking, or an explicit "no prediction". Which one is a product decision.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model server is down, or the feature store is slow. What does the endpoint return, who decided that, and does the caller know it happened?
A recommendation carousel on a retailer's home page is driven by a ranking model. During a feature-store incident the carousel went blank for forty minutes and the on-call was paged for a revenue drop. Nobody had decided what the page should show when the model cannot answer.
Make the service reliable. Add replicas, retries and a health check; if the model cannot answer, return an error and let the frontend handle it. Failures should be rare enough not to need a design.
The frontend "handled" the error by rendering nothing. Rare failures with a blank fallback cost more per minute than frequent ones with a sensible one, and the forty-minute incident was rare enough that nobody had exercised the path.
- The frontend "handled" the error by rendering nothing. Rare failures with a blank fallback cost more per minute than frequent ones with a sensible one, and the forty-minute incident was rare enough that nobody had exercised the path.
- Retries against a slow feature store multiplied load on the thing that was already slow, extending the incident (Retry Storms: The Load You Generated Yourself).
- The bad-deploy mode was not caught by any health check: the model answered fast and confidently with rankings that put clearance items first. A fallback needs a trigger for "answering wrongly", not only for "not answering".
- The timeout was set on the whole request; when it fired, the service had already spent the budget and had nothing prepared to return.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding model ranks products for a user; the serving system's target is that the carousel always shows *something* defensible inside the page's latency budget, and that a degraded answer is distinguishable from a normal one.
- The decision downstream is what to render. Blank is one option, and the worst one; the fallback is a product choice about which cheaper answer is acceptable.
- One request carries a user id and page context. The service fetches user and product features from an online store, calls the ranking model over a candidate set and returns an ordered list.
- A popularity ranking is recomputed nightly and stored; the previous model version is still in the registry; the last ranking served to each user is not cached anywhere.
- Incident history shows three modes: model server unreachable, feature store slow past the timeout, and a bad model deploy returning nonsense with perfect latency.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A serving path has several things that can be missing: the model process, a feature value, a whole feature source, or a plausible output. Each has its own detection — connection error, per-fetch timeout, source-level circuit breaker, output sanity check — and each can map to a different fallback.
- Fallbacks form a ladder, cheapest and most degraded at the bottom: previous model version (same features, older weights), rule-based ranking (no model, live features), cached prediction (this user's last answer), default ranking (popularity, no user information), and explicit no-prediction. A system picks which rungs exist and in what order.
- A timeout must return a *defined* thing. The per-fetch timeout returns a training-time default for the missing feature and lets the model run; the request-level timeout returns the next rung down. Both must be chosen before the incident, because during it there is no time to choose.
- Degraded answers must be marked. The response carries a flag; the log records which rung served it; downstream metrics split by rung. Otherwise the model's quality metrics silently absorb the fallback's, and the incident is invisible in the business numbers except as a dip nobody can attribute (Prediction Logging).
The ladder
The ladder is a product artefact as much as an engineering one. Each rung is a statement about what the business will accept when the better answer is unavailable, and the order encodes which degradations it prefers. For the carousel the decision was: previous model, then live rules, then popularity, then a curated static list — never blank.
For the fraud model in Train / Serve Skew the ladder is entirely different: previous model, then a conservative rule that sends high-value transactions to review, then a manual queue. There is no rung that approves without a score.
- 1Model with live features
The normal path: fetch features, rank candidates, return the ordered list.
fails by Model server unreachable; feature fetch past its timeout; output fails the sanity check.
- 2Previous model version
Same features, last-known-good weights, loaded alongside the current model.
fails by Fails with the current model when the feature store is the problem.
- 3Rule-based ranking
Recency and category affinity from whatever features arrived; no model call.
fails by Needs at least some live features; rots if the feature schema changes and the rule is never run.
- 4Popularity default
Nightly precomputed top items by segment; no user features.
fails by Stale if the nightly job failed; identical for every user.
- 5Curated static list
A hand-picked list shipped with the service; always renders.
fails by Only by being the answer for too long without anyone noticing — hence the flag.
Every step returns inside the request budget because each rung is checked against a per-dependency timeout, not a request-level one. The request-level timeout is the last resort and returns the bottom rung.
Timeouts return something defined
The serving code for fallbacks is mostly ordinary backend engineering — timeouts, circuit breakers, bulkheads — applied at two boundaries the ML system adds: the feature fetch and the model call. What is specific to ML is that a missing feature is not an error but an input the model must have been trained to interpret, and a returned score is not necessarily valid.
So the per-feature timeout returns the training-time default and records that it did; the model-call timeout returns the next rung; and the output check can demote a perfectly fast answer.
1type Served = { items: string[]; rung: 'model' | 'prev-model' | 'rules' | 'popular' | 'static' }2 3async function rank(userId: string, candidates: string[]): Promise<Served> {4 // 1. features: each fetch has its own timeout and a training-time default5 const feats = await withTimeout(features.get(userId), 40, () => TRAINING_DEFAULTS)6 const degradedFeatures = feats === TRAINING_DEFAULTS7 8 // 2. model: its own timeout and a breaker; on failure, step down9 if (modelBreaker.closed()) {10 const scored = await withTimeout(model.score(feats, candidates), 60, () => null)11 if (scored && outputLooksSane(scored)) {12 return { items: order(scored), rung: degradedFeatures ? 'rules' : 'model' }13 }14 modelBreaker.recordFailure()15 }16 const prev = await withTimeout(prevModel.score(feats, candidates), 60, () => null)17 if (prev && outputLooksSane(prev)) return { items: order(prev), rung: 'prev-model' }18 19 // 3. no model answered: cheaper rungs, never blank20 if (!degradedFeatures) return { items: rulesRank(feats, candidates), rung: 'rules' }21 const popular = await popularity.get() // nightly table, in memory22 return popular ? { items: popular, rung: 'popular' } : { items: STATIC_LIST, rung: 'static' }23}The rung field is the point. It travels to the log and to the frontend, so the business metric can be split by rung and the carousel can be visibly plainer when it is degraded. A fallback nobody can see in the data is a fallback whose cost nobody can measure.
Drill the rungs before the incident
The ladder's assumption is that every rung works today. That is not true by default: the rule-based rung was last exercised at launch, the popularity table depends on a nightly job, the previous model needs its own feature contract. The failure simulator's "kill model server" and "slow feature store" controls are the offline version of this drill; the production version is a game day that turns each dependency off in turn and checks what the page shows.
The second assumption is that the degradation is visible. A rung that serves silently for a day because a breaker never closed is the quiet version of the blank carousel — cheaper per minute, far longer.
Each fallback rung returns a valid response inside the budget today, and each response records which rung served it.
holds when Rungs are exercised on a schedule against current features and tables; the flag survives every hop to the log and the page; breakers have half-open probes so the model rung returns after recovery.
breaks when A feature rename breaks the rule rung; the nightly popularity job fails silently; an intermediate service drops the flag; a breaker opens and stays open.
respond Treat a rung that fails its drill as a broken deploy, not as a note for later; treat a silently open breaker as an incident.
How to build it
Most important first.
- Decide the fallback ladder with the product owner, rung by rung, including what "no prediction" looks like on the page. Write the decision down next to the model, not in the on-call runbook.
- Implement per-dependency timeouts and circuit breakers so a slow feature store degrades the feature, not the request, and stops being called while it is down (in code this is mostly Timeouts and Circuit Breaker applied at the feature and model boundaries).
- Add an output sanity check — score distribution, top-k composition — that triggers the previous-model rung on a bad deploy, since latency and health checks will not (Model Invariant Tests give the invariants to check).
- Exercise every rung on purpose: the failure simulator's "kill model server" and "slow feature store" controls at
/ml/failuresare exactly this drill, and the production equivalent is a game day that turns each dependency off.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Fraction of requests served by each rung, over time. The number that says both how often the model is actually answering and what the incident cost — because each rung has a known quality relative to the model.
- Business metric split by rung: click-through for model-served versus popularity-served carousels. Without the split, the model's metric is a blend and the fallback's cost is unknown.
- Uptime of the model server measures the server, not the product; the product was "up" and blank.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every rung of the ladder runs correctly today — the previous model loads, the rule evaluates, the popularity table is fresh — proven by a regular drill rather than by the last incident.
- A degraded response is flagged end to end, so the rung that served each request is recoverable from the log and the business metric can be split by it.
- Per-dependency timeouts and breakers are set inside the request budget so that a fallback is returned with time to spare rather than after the caller has given up.
- Offline: run the ladder as a test suite — kill the model, slow the store, corrupt the output — and assert the response is the expected rung within the budget.
- Online: dashboard the rung share; alert when the model rung drops below its normal share, which catches an open breaker that never closed.
- Over time: compare business metrics by rung monthly; if the popularity rung is nearly as good as the model, that is information about the model, not only about the fallback.
What can go wrong
- The cached-prediction rung serves last week's ranking to a user whose interests moved, and does so with the confidence of a fresh answer because the flag was dropped by an intermediate service.
- The rule-based rung was written two years ago against a feature that has since been renamed; it has never run, and runs for the first time during the incident, and throws.
- The circuit breaker opens on the feature store, the default rung takes over, and it stays open long after the store recovers because nobody set a half-open probe; the model is silently unused for a day (Circuit Breaker).
- Every rung is code that must be kept working against features and tables that change; a five-rung ladder is five things that can rot.
- The previous-model rung keeps an old artifact loaded — memory and a second model to monitor. The cached-prediction rung is storage and a staleness question.
- Marking degraded responses adds a field to every response and a dimension to every metric; the alternative is not knowing what the incident cost.
- "The model server has three replicas, so we do not need a fallback." The feature store has one incident mode, the model has another, and a bad deploy has a third. Replicas address only the first.
- "Fallback means retry." Retrying a slow dependency adds load to it. The fallback is the thing you do *instead* of waiting, and the retry budget is part of the timeout, not a substitute for the ladder.
- "Return an error and let the caller decide." The caller is a page template. It will decide blank. The decision belongs with the people who know what a degraded carousel should show.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a timeout must return a defined result and that degraded answers must be marked holds for every online model, whatever it predicts; the specific rungs differ.
- DOMAIN-SPECIFICA recommendation carousel can fall back to popularity with a small revenue cost; a fraud decision cannot fall back to "approve everything", and its bottom rung is a conservative rule or a manual queue. Medical and lending systems often have "no prediction, route to a human" as the only acceptable rung.
- CONTESTEDOne position holds that fallback ladders are over-engineering for most teams: a single well-chosen default plus an honest error is easier to keep correct than five rungs that rot, and complex ladders have caused incidents of their own when a stale rung ran for the first time under load. That is a fair warning; the counter is that the single default must still be chosen deliberately, marked, and drilled, which is most of the work anyway.
Where the depth lives
This domain teaches the model and hands the rest off by name.