Model Postmortems
A model incident postmortem records the assumption that broke, the signal that should have fired, the label delay that hid it and the test now added — and never concludes that "the model" was at fault.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What does a postmortem for a model incident have to record that a service postmortem does not, and why is "the model was wrong" never a root cause?
We ran a postmortem on the nine-days-of-approvals fraud incident. The document says root cause: "model degradation"; action item: "retrain more often". Three months later the same column rename happened to a different feature and the same thing occurred. The postmortem was filed and changed nothing.
Use the service postmortem template: timeline, impact, root cause, action items. The root cause is that the model made wrong predictions; the action items are to retrain more often and to add a quality alert. Blameless, filed, done.
The root cause names the symptom. The model made wrong predictions because a feature it depended on was silently null; "model degradation" is what a null feature looks like from the outside, and a postmortem that stops there fixes nothing.
- The root cause names the symptom. The model made wrong predictions because a feature it depended on was silently null; "model degradation" is what a null feature looks like from the outside, and a postmortem that stops there fixes nothing.
- The action item "retrain more often" would have made the incident worse: a retrain during the nine days would have learned that the feature was uninformative and validated fine, and the rollback target would have been gone.
- The action item "add a quality alert" adds an alert on a signal that cannot fire for 90 days. The signal that could have fired on day one — per-feature null rate — is not mentioned, because the template has no place for "what should have fired".
- Three months later a different column is renamed. There is no contract test on feature null rates, because no action item asked for one, and the rename is caught by the chargeback report again.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The postmortem's target is the next incident of the same shape. It succeeds if the assumption that broke is now monitored, the signal that should have fired now exists, and the test that would have blocked the change is now in the pipeline.
- The model under discussion is the same fraud model; the decision, label and delay are as in the rest of the module.
- The incident timeline: the column rename on day one, the null feature, the collapsed prediction mean, the halved flag rate, the chargeback report on day nine, the rollback on day ten.
- The signals that existed: latency, errors, saturation — all green. The signals that did not: per-feature null rate, prediction mean, flag rate. The quality panel existed, was flat, and was flat because no labels had arrived (Ground-Truth Delay).
- The reasoning loop the model was built on — problem, target, data, features, model, evaluation, deployment, monitoring — and the assumption at each stage that the incident tested (The ML Reasoning Loop).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A model is a set of assumptions with weights attached: that the features mean what they meant in training, that the label means what it meant, that the population is the one it learned, that the decision policy is the one it was evaluated under, that outcomes arrive in time to check any of this. A model incident is one of those assumptions breaking, and the root cause is the assumption, not the weights (What Can Go Wrong).
- Because the assumption broke silently, the incident's duration is set by how long it took a signal to fire. A service postmortem asks why the alert was late; a model postmortem has to ask which signal would have fired on day one had it existed, and why the one that did exist — quality — was structurally unable to, which is nearly always the label delay.
- The corrective action follows from the assumption and the signal: a monitor for the assumption at the boundary where it can be checked early, and a test that blocks the change that broke it. "Retrain" is a corrective action only when the assumption that broke is "the world is the one we trained on", and the incident has to have established that (ML Incident Debugging).
Four fields a service postmortem does not have
The service template asks what broke, when, how bad, and what will be done. A model incident answers those and leaves the useful part unsaid: which assumption the model depended on turned out to be false, which signal would have shown it, why the signal that existed could not, and what now stands between the same change and the same outcome.
Below is the fraud incident written both ways. The second version is longer, names two teams outside the model team, and would have prevented the recurrence three months later.
Root cause: model degradation led to under-flagging of fraudulent transactions for nine days. Impact: chargebacks up. Action items: retrain more frequently; add an alert on model quality.
Assumption that broke (features stage): `merchant_category` would keep the meaning and null rate it had in training; a rename upstream made it null and preprocessing imputed the mode. Signal that should have fired: per-feature null rate at the serving boundary, on day one, within an hour. What hid it: quality is computed on chargebacks with a 90-day delay, so the quality panel was structurally silent; the flag rate halved on day one and had no panel. Tests added: null-rate contract test failing the serving health check above a threshold; flag-rate floor alert on the decision service; schema-change notification from the owning team to every consumer. Not an action item: retraining — a retrain during the incident would have learned the null and validated.
The second version locates the failure at a stage, names a monitor that fires within the label delay, explains why the existing monitor could not, and produces artifacts with owners. The first names a symptom and prescribes the trap.
The assumption, not the artifact
The instinct to write "the model" as the cause comes from where the incident was experienced: the model produced the scores, the scores were wrong. But the weights did exactly what they were trained to do with the input they received. The failure is upstream of the model — the feature stopped meaning what it meant — and downstream of it — the decision policy turned a collapsed score into an approval with no floor. Neither is a property of the artifact.
Writing the assumption against the reasoning loop makes recurrence recognisable. The next incident, three months later, was the same assumption at the same stage with a different column. A postmortem indexed by stage would have matched it in a minute; one indexed by "model degradation" matched nothing.
| Reasoning-loop stage | Assumption that can break | Signal that fires early | Test that blocks the change |
|---|---|---|---|
| Data / label | The label means what it meant in training | Label base rate over time; label-definition version in lineage | Label-definition diff gate before training |
| Features | Each feature keeps its meaning, unit, null policy and distribution | Per-feature null rate and distribution distance at serving | Schema and null-rate contract test in the serving health check |
| Model / evaluation | The promoted model was compared fairly, at a matched operating point, per slice | Prediction distribution and flag rate by version on rollout day | Champion/challenger gate with slice non-regression |
| Deployment / serving | Serving reproduces the training preprocessing and feature versions | Train/serve parity score on a sample per deploy; fallback rate | Parity check blocking promotion; lineage triple asserted at deploy |
| Decision | The threshold policy is the one the model was evaluated under | Flag rate per decision against a floor and ceiling | Threshold under the same review as a model change |
| Monitoring / drift | Outcomes arrive in time to measure quality | Label lag on the quality panel | Alert when label lag exceeds its norm; quality panel shows "unmeasured" |
The signal that should have fired
The most useful sentence in a model postmortem is the one that names a signal that did not exist and says when it would have fired. It converts a story into a monitor. For the fraud incident: per-feature null rate at the serving boundary, day one, within the hour — and, independently, the flag rate, which halved before any chargeback could have arrived.
The second most useful sentence explains why the signal that did exist stayed quiet. Quality is joined against chargebacks, and chargebacks take up to 90 days; the panel was not wrong, it was empty, and it read as fine. Every model with a delayed label has this blind spot, and the postmortem is where it gets written down so the next quality panel shows label lag beside it.
For every model incident, the assumption that broke is now watched by a signal at the boundary where it can fire within the label delay, and the change that broke it is now blocked by a test.
holds when The postmortem's four fields are filled; each action item is a named artifact with an owner outside the meeting if necessary; a follow-up confirms the artifact exists; the assumption is indexed by reasoning-loop stage.
breaks when The root cause is written as "the model"; the action item is "retrain more often" or "improve monitoring"; the signal proposed is quality, which cannot fire inside the label delay; the owning team of the upstream change is not in the room.
respond Reopen the postmortem, fill the four fields, and reassign the artifacts to owners who can build them. Do not schedule a retrain as the outcome of a document.
1incident: fraud-approvals-2026-052assumption_broken:3 stage: features4 claim: "merchant_category keeps its training-time meaning and null rate"5 broke_because: "upstream column rename; reader served null; preprocessing imputed mode"6signal_that_should_have_fired:7 name: feature_null_rate[merchant_category]8 boundary: serving9 would_have_fired: "day 1, within 1h"10hidden_by:11 label_delay_days: 9012 quiet_signal: online_quality # empty, not flat; now shows label lag13tests_added:14 - { kind: monitor, name: feature_null_rate_alert, owner: ml-platform, done: true }15 - { kind: monitor, name: flag_rate_floor, owner: decisions, done: true }16 - { kind: contract_test, name: feature_schema_null_rate, pipeline: serving-health, owner: data-eng, done: false }17not_an_action_item:18 - retrain_more_often # would have learned the null and validatedThe done: false line is the reason the record is data rather than prose: a month later a script can list every postmortem with an unbuilt artifact, and three months later the column rename that recurs can be matched against stage: features in seconds.
How to build it
Most important first.
- Extend the template with four model-specific fields: the assumption that broke (stated as the reasoning-loop stage and the claim), the signal that should have fired and when, the label delay that hid the failure and for how long, and the test or monitor now added that would have caught it. The DevOps postmortem practice supplies the rest — timeline, blamelessness, ownership.
- Locate the failure in the pipeline or the decision policy, not in "the model". If the finding is that the model itself was the problem — it was trained on the wrong window, promoted on the wrong criterion — say which promotion criterion or evaluation assumption failed; that is still a process, not a mysterious artifact.
- Make every action item a checkable artifact: a named monitor with a threshold, a named test in a named pipeline, a named field added to the deploy record. "Improve monitoring" and "retrain more often" are not action items.
- Review the postmortem against the next incident: when one recurs, the first question is which action item from the last one would have caught it and whether it was done (Model Regression Tests).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- For each postmortem: did the assumption, the signal and the test get named, and does the test exist a month later? A postmortem without those is a timeline.
- Time from the assumption breaking to the first signal firing, per incident, over time. It should shrink as monitors are added at the boundaries that broke.
- Number of postmortems is not a quality measure; recurrence of the same assumption breaking is.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every model incident is written up against the reasoning loop, naming the stage and the assumption, so that recurrences are recognisable as the same assumption rather than as a new incident.
- Each action item is an artifact — monitor, test, deploy-record field — with an owner who can build it and a check that it was built.
- The postmortem examines the pipeline and the decision policy with the same rigour as the model, and the people who own them are in the room.
- Offline: for each postmortem, replay the incident with the added monitor or test and confirm it fires on day one — the simulator at
/ml/failurescan stand in for the replay. - Online: a month after each postmortem, check the action items exist as artifacts; a quarter after, check that no incident has recurred with the same assumption.
- Over time: a list of assumptions that have broken, by reasoning-loop stage; the stages with the most entries are where the next monitor belongs.
What can go wrong
- The postmortem correctly names the null feature and the missing null-rate monitor, and the action item is assigned to the team that owns the renamed column, who do not own the monitor. It is never built.
- The blameless framing is read as "no cause": the assumption that broke is an organisational one — the feature schema was not a contract between teams — and the postmortem declines to say so.
- The postmortem is written by the model team alone; the decision policy's role — a threshold that turned "low score" into "approve" with no floor on the flag rate — is not examined because nobody in the room owned it.
- A postmortem that names the assumption and the signal is longer and harder to write than one that says "model degradation", and it implicates teams outside the model team.
- Action items that are artifacts have owners and deadlines and get argued about; "improve monitoring" is agreeable and free, which is why it recurs.
- The reasoning-loop framing is a discipline the organisation has to learn; the first few postmortems written this way will be resisted as over-engineering for a "model bug".
- "Root cause: the model degraded." The model did what a model does with a null input. The root cause is the assumption that the feature would keep meaning what it meant, and the missing monitor on it.
- "Action item: retrain more often." Retraining during the incident would have learned the defect and validated fine. Retraining is the fix for one assumption — the world moved — and this incident was a different one.
- "Blameless means we don't say the pipeline was at fault." Blameless means no person is at fault. The pipeline, the schema contract and the decision policy are exactly what the postmortem exists to name.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a model incident is an assumption breaking, and that its postmortem must name the assumption, the signal and the test, holds for every deployed model; the specific assumptions and signals depend on the pipeline.
- CONTESTEDA defensible position holds that a separate ML postmortem template fragments the incident process: a service and a model postmortem for the same outage, two documents, two owners, and the model-specific questions could simply be prompts in the standard template. That is right about process overhead; the counter-argument is that without the four fields written down as required, the standard template's "root cause" slot is filled with "the model" every time, because that is the level at which the on-call engineer experienced it.
Where the depth lives
This domain teaches the model and hands the rest off by name.