ObservabilityGENERALSIMULATEDSCALE-SPECIFIC

ML Incident Debugging

Conversion fell after a deploy. Investigate from the business metric down — prediction distribution, model version, feature values, feature pipeline, raw data — in that order, and do not retrain until the cause has a name.

Target & data What to measure What must stay true ▶ ML Failure Simulator

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

The business metric dropped after a model deployment. In what order do you look, what does each layer rule out, and why is retraining before diagnosis the wrong move?

The problem

We shipped a new version of the checkout recommendation model on Tuesday. On Thursday the product team reported that add-to-cart from recommendations is down by a fifth. The model team wants to retrain on the latest data; the platform team wants to roll back; nobody knows what happened.

The obvious approach

A model deploy was followed by a metric drop, so the model is worse. Retrain it on the freshest data and redeploy; if that does not help, roll back. Investigating six layers takes days, and every day costs a fifth of add-to-cart revenue.

Why it breaks

The retrain uses the same feature pipeline, which is where the defect is: the new feature is null for most rows because its source table is populated hourly and the serving cache refreshes daily. The retrained model learns that the feature is uninformative, validates well against the same nulls, ships, and the metric stays down.

How it breaks — usually after the offline metric looked fine

The retrain uses the same feature pipeline, which is where the defect is: the new feature is null for most rows because its source table is populated hourly and the serving cache refreshes daily. The retrained model learns that the feature is uninformative, validates well against the same nulls, ships, and the metric stays down.
The rollback restores the old model onto the new feature service. The old model does not use the new feature, so it happens to work — and the team concludes "the new model was bad" when the new model was fine and its feature was missing.
Nobody checked whether the drop was in the model's slot at all. On the same Tuesday a checkout redesign moved the recommendation slot below the fold on mobile, and half of the drop is layout.
The prediction distribution had shifted on Tuesday morning. It was on a panel nobody opened, because the panel that was opened was the error rate, which was flat.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

Rank items to show at checkout; the decision is the four items shown; the outcome is add-to-cart within the session — a fast label, which is what makes the drop visible within two days.
The investigation's target is the layer where the change entered. The model is one of six candidates and not the most likely one.

Data

Product analytics: add-to-cart rate from the recommendation slot, by day, by surface. Prediction logs: score distribution, flag rate, fallback rate by hour and model version. Feature monitoring: per-feature null rate and distribution distance against training. Pipeline runs: the feature job's schedule, status and row counts. Raw events: the upstream tables the features are built from.
The deploy on Tuesday included the new model artifact and — in the same release — a feature-service change that added a new feature the new model needs.
The failure simulator at /ml/failures provides the same nine signals for ten injectable causes, and its candidates() function shows which causes remain consistent with any subset of them.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

A model decision is the last step of a chain — raw data → feature pipeline → feature values → model version → prediction distribution → decision → business metric — and a defect entering at any link propagates to the metric. Because the metric is downstream of everything, it cannot say where the defect entered; it can only say that something did.
Investigating from the metric upward is a sequence of rule-outs, each cheap. The prediction distribution says whether the model's outputs moved at all (if not, the problem is downstream — a threshold, a layout, a logging change). The model version says whether the weights are the ones expected. The feature values say whether the inputs moved. The feature pipeline says whether the inputs moved because the code or schedule did. The raw data says whether the world moved. Each layer's reading narrows the candidates, and the order runs from cheapest and most likely to most expensive.
Some readings cannot distinguish causes. A feature whose serving distribution shifted, with the prediction mean and quality following, looks the same whether the population moved or the preprocessing changed. The simulator's candidates() makes this concrete: given only those three signals, both causes remain. What separates them is the raw distribution at the source and the null rate — signals that feel redundant until they are the only ones that decide.

From the metric down

The metric is the last link, so it is where the investigation starts and the only place it cannot end. Each step upward reads one layer's signal and eliminates the causes inconsistent with it; the order is from the layer that is cheapest to read and most often the answer to the one that is dearest and rarest. The model itself sits in the middle — not first, because a deploy that changed the model also changed other things, and not last, because it is easy to check.

The pipeline below is the order for the checkout incident. Each step names what the reading rules out, which is the useful thing a reading does.

Conversion dropped after a model deploy

1
Business metric
Confirm the drop is in the model's slot: slice by surface, device, slot rendered; list everything else that deployed. Rules out layout, tracking and unrelated releases.
fails by Taken at face value; half the drop is a mobile layout change and the investigation chases the model for it.
2
Prediction distribution
Score mean, flag rate and fallback rate by model version and hour. A shift on deploy day says the model or its inputs moved; no shift sends you to the decision policy and rendering.
fails by Compared against the old model's distribution instead of the new model's shadow or canary baseline, so "different" means nothing.
3
Model version
Is the served artifact the promoted one, with the expected feature-definition and preprocessing versions? Rules out a stale pod, a partial rollout, a wrong registry tag.
fails by Assumed; a third of the fleet is still on the old image and the distribution is a mixture.
4
Feature values
Per-feature null rate and distribution distance against training, especially for the new feature. A null spike names the layer.
fails by Only the old features are monitored; the new one was added in this release and has no panel.
5
Feature pipeline
The feature job's schedule, status, row counts, schema; the serving cache's refresh. Distinguishes "the code or schedule changed" from "the world changed".
fails by Belongs to another team and is skipped; the hourly source with a daily cache is never found.
6
Raw data
The upstream tables: row counts, category mix, new entities. If everything above is clean and the raw data moved, the world moved — and only now is retraining on the table.
fails by Reached first out of habit; a catalogue promotion is blamed for a null feature.

Retraining appears once, at the bottom, as the response to one specific finding. Every other layer's finding has a different fix, and none of them is a training job.

Readings that cannot tell two causes apart

The simulator's ten controls each produce a distinct pattern across all nine signals — but not across any three. Shift a feature (the world moved) and change preprocessing (the code moved) both raise the feature-distribution distance, raise the prediction mean and lower online quality. An engineer who reads those three, which are the three most teams have, sees drift and reaches for a retrain. The preprocessing case retrains on the old preprocessor and the skew survives.

What separates them is the null rate — the new preprocessor maps unseen categories to null — and the raw distribution at the source, which did not move. Those are the two panels that look redundant next to the feature-distribution panel until they are the only ones that decide. The simulator's candidates() returns both causes for the three-signal reading and one for the five-signal reading; that difference is the argument for monitoring all nine.

Four of the simulator's ten causes, and the trap each sets

Trigger	Symptom	Cause	Response
Serving preprocessor upgraded; training still on the old one	Feature PSI up, prediction mean up, quality down, null rate up; raw source unchanged	Train/serve skew introduced by a version mismatch	Pin the preprocessor to the training version; add a parity check per deploy. Trap: retrain — the mismatch survives.
Upstream column renamed; feature store serves null	Null-rate spike on one feature, its distribution panel empty, prediction mean down	A broken reader, imputed silently as the training mode	Fix the reader, backfill, add a null-rate contract test. Trap: retrain — the model learns to drop the feature.
Decision threshold moved in config	Every model-side signal normal; quality and business metric down	The policy between score and action changed	Restore the threshold; monitor flag rate as its own signal. Trap: retrain to "get recall back" — the scores were fine.
Outcome join stuck behind a migration	Online quality unknown, label lag up, everything else normal	Quality is unmeasured, not bad	Unblock the join; lean on label-free proxies and say quality is unknown. Trap: read the flat chart as fine and retrain on schedule without the newest outcomes.

What the simulator says about a partial reading

1import { candidates, distinguishing } from '@/ml/sim/failure'
2 
3// the three signals most teams have, read during the checkout incident
4candidates({ 'feature-distribution': 'up', 'prediction-mean': 'up', 'online-quality': 'down' })
5// -> ['shift-feature', 'change-preprocessing']   two causes, opposite fixes
6 
7// which signals would separate them
8distinguishing('shift-feature', 'change-preprocessing')
9// -> ['feature-null-rate']   plus, off-dashboard: the raw distribution at the source
10 
11// add the null-rate panel and the reading is unique
12candidates({ 'feature-distribution': 'up', 'prediction-mean': 'up', 'online-quality': 'down', 'feature-null-rate': 'up' })
13// -> ['change-preprocessing']

The middle call is the lesson: the signal that decides is the one that looked redundant. The simulator also has a reading with every model-side signal normal and only quality and the business metric down — the bad-threshold case — which no amount of looking at the model can explain, because nothing about the model changed.

Why retraining before diagnosis makes it worse

A retrain re-expresses the pipeline as it is now. If the pipeline has a null feature, the retrained model learns to ignore it. If the label join is stuck, the retrained model is trained without the newest outcomes. If the preprocessor is mismatched, the retrained model is trained on one representation and served on another. In each case the validation number is fine, because validation shares the defect, and the metric stays down with one more clue erased — the old model's behaviour, which was the evidence.

The assumption that makes a retrain safe is the one the incident has just called into question: that the pipeline is healthy. During an incident that assumption is exactly what is being investigated, and acting as if it held is the trap every simulator control names.

The Thursday retrain

offline evaluation said

The retrained checkout model, trained Thursday on the latest data, validates as well as the version shipped Tuesday.

production did

Add-to-cart in the slot remains down by the same fifth after the Friday deploy; the new feature is now absent from the model's top importances.

What explains the gap — most likely first

1The new feature was null for most served rows because its source refreshes hourly and the serving cache daily; the retrain learned it was uninformative and validation, on the same nulls, agreed.
2Half the drop was a mobile layout change shipped the same day, which no model can recover.
3A small part may be genuine decay in the new catalogue mix — indistinguishable until the null feature is fixed.

what it costs to close or detect Two days, two training runs and a deploy, and the erasure of the evidence: the Tuesday model's behaviour on the null feature was the clue, and the Thursday model no longer has it. Finding the cache schedule required one query against the feature-null-rate panel that existed all along.

must stay trueThe pipeline is healthy enough to retrain on

The feature pipeline, label join, preprocessing versions and decision policy are all in their known-good state, so that a retrain would learn from the current world rather than from the current defect.

holds when The investigation has reached the raw-data layer with every layer above it clean; the null rates, label lag and preprocessing versions match the incumbent's lineage; the deploy record shows nothing else changed.

breaks when Any layer above raw data shows a change on deploy day; the label lag is elevated; the served feature-definition or preprocessing version differs from the training one; a threshold or layout changed in the same release.

how you would know The metric-down walk itself, recorded layer by layer; the pre-training gate from the retraining lesson, which refuses a run whose null rates, label lag or versions differ from the incumbent's lineage.

respond Fix the named layer. Mitigate with a rollback only if the previous triple is servable. Retrain when — and only when — the finding is that the world moved and the labels for the moved world have arrived.

How to build it

Most important first.

Start at the business metric and confirm it is the model's: slice by surface, device, and whether the recommendation was shown; check what else deployed that day. A drop that is not in the model's slot is not the model's incident.
Then the prediction distribution and flag rate by model version and hour. A shift on deploy day localises the change to the model or its inputs; no shift sends the investigation downstream to decision policy, rendering or logging.
Then model version and feature values: is the served version the promoted one, and do the served features match training per feature — null rate, distribution distance. A null spike on the new feature names the layer. Then the pipeline (schedule, row counts, schema) and finally the raw data (did the world move).
Do not retrain until the layer is named. Rollback is acceptable as mitigation if the previous triple is servable (Rollback & Fallback); a retrain on a broken pipeline launders the defect (Retraining as a Decision). Run the simulator's "change preprocessing" and "shift a feature" controls back to back to see two causes with the same first three signals.

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

Add-to-cart rate in the recommendation slot, sliced by surface and by whether the slot was rendered, per day. This is the metric that was reported and the first thing to confirm.
Prediction mean and flag rate by model version and hour around the deploy; per-feature null rate and distribution distance for every feature the new model uses, especially the new one.
Offline validation of the new model is not a signal here. It was computed before the deploy on features from the training path, and the incident is in the serving path.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

Each layer has a signal that can be read for the deploy window: metric by slice, prediction distribution by version, per-feature null rate and drift, pipeline run history, raw-table row counts — and they share a time axis.
The deploy is recorded with everything it changed — model, feature service, decision config, frontend — so "what else shipped on Tuesday" is a lookup, not a Slack search.
No corrective action — retrain, threshold change, feature fix — is taken until a layer has been named, and the mitigation (rollback) is known to be safe against the current feature service.

How to verify — offline, online, and over time

Offline: rehearse with the simulator — inject a cause, read signals in the metric-down order, and check with candidates() after each reading how many causes remain. Stop when one does.
Online: during the real incident, record each layer's reading and the candidates it eliminated, so the postmortem can say which signal decided and which were missing (Model Postmortems).
Over time: for every incident, the layer at which the cause was found and the time to reach it; if the cause is repeatedly found at a layer without a monitor, that monitor is the action item.

What can go wrong

Failure modes in production

The investigation starts at the model because that is what the model team owns, and spends a day on validation curves for a defect that is in the feature cache schedule.
The prediction distribution is checked against the previous model's and looks different — because it is a different model with different scores. Without the new model's shadow or canary distribution as the baseline, "different" carries no information.
The raw-data layer is skipped because it belongs to another team; the world did move — a promotion changed the catalogue mix — and the model is retrained on Tuesday's data that lacks Wednesday's items.

What the recommended approach costs

The top-down order is disciplined and slower than a hunch; when the hunch is right it looks wasteful, and when it is wrong nobody remembers the hunch.
Confirming the metric is the model's requires sliced analytics that the product team has to maintain; without them the first step is a guess and the whole order inherits it.
Refusing to retrain during an incident costs days of a degraded metric when the cause really was decay — which is rare after a deploy, and the simulator shows why.

Misreads

"The metric dropped after the deploy, so the new model is worse." The deploy shipped a model, a feature service change and a layout change. The model is one of three suspects and the offline evaluation said it was better; the serving path is where to look.
"Retraining is cheap, so try it first." Retraining on a pipeline with a null feature produces a model that ignores the feature and validates fine; the metric stays down and the team has lost a day and a clue.
"Feature drift fired, so it is drift." A preprocessing change fires the same three signals as drift. The raw distribution at the source and the null rate separate them, and that is the reason to have both panels.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

GENERALThe metric-down order is a sequence of rule-outs that applies to any deployed model with a chain of layers; what differs is which layers exist and how fast the metric responds — a 30-day label makes the first step a proxy.
SIMULATEDThe failure simulator at /ml/failures is a declarative model of a serving system: ten failures, nine signals, one reading per pair, with baseline values chosen to make the diagnostic argument. Its fingerprints and the ambiguity candidates() exposes are constructed, not measured on a real service, and any numbers it displays are for the shape of the argument.
SCALE-SPECIFICOn a small team the six layers are three people and the order is a conversation; at scale each layer is a team with its own dashboards, and the order matters because it says whose dashboard to open first and whose pager not to pull.

Where the depth lives

This domain teaches the model and hands the rest off by name.

DevOpsincident-response deployment-centric-debugging deploy-annotations

Observability & Performanceincident-debugging metric-labels