Postmortems That Change Things
A postmortem is worth what it changes. One change that would have prevented this incident, owned by a named person, with a date — instead of ten action items that quietly expire.
The ask, the obvious response, and how it goes wrong
Every lesson starts where the work starts: someone asked for something, and the first response that comes to mind has a problem.
Why do the same incidents keep happening after thorough postmortems, and what would a postmortem have to produce to stop them?
The engineering manager, reviewing the quarter: "We did a postmortem for every incident, and we have 47 open action items. Two of this quarter's incidents were repeats. What is going on?"
Be thorough. Find every contributing factor, list an action item for each, assign them in the meeting and track them in the backlog. More action items means more learning.
Ten action items per incident, each "medium priority", compete with the roadmap. The easy ones — add a log line, update a doc — get done. The one that would have prevented the incident, "make order creation atomic with payment capture", is large and stays open.
- Ten action items per incident, each "medium priority", compete with the roadmap. The easy ones — add a log line, update a doc — get done. The one that would have prevented the incident, "make order creation atomic with payment capture", is large and stays open.
- Items are assigned to "the team" or to whoever was in the room, then that person changes projects. Nobody closes the item; nobody reopens it; it just ages.
- Three months later the same failure — payment captured, no order — happens through a different code path. The postmortem finds "the previous action item was not completed" and adds it again.
- The write-up is long and nobody outside engineering reads it. The PM learns the incident repeated from support, not from the postmortem, and stops believing postmortems do anything.
What is really going on
- A postmortem has one output that matters: a change to the system, the process or the product that makes this class of incident less likely or less harmful. Everything else — the timeline, the contributing factors — is the evidence for choosing that change.
- Action items compete with the roadmap, and the roadmap has sponsors. An item with no named owner, no date and no one asking about it will lose that competition every week until it is forgotten (Sequencing Work So Value Lands Early).
- Ten items dilute the one that matters. When every item is "medium", the important structural fix looks like one of ten, and the team feels productive closing the other nine. One item, argued for, is harder to ignore.
- The best single change is usually structural, not procedural. "Be more careful with promo code" asks people to be different. "Order creation and payment capture happen in one transaction, or payment is voided" makes the failure impossible.
- Postmortems change things only if someone outside the room can see them. A short summary the PM and support can read — what happened to customers, the one change, who, when — is what turns the postmortem into a commitment the team is held to (Writing for Stakeholders).
Ten items, none of them the one
The March postmortem for the payment-without-order incident was careful and long. It listed nine actions: add a log line, alert on order creation rate, update the promo runbook, add tests for null promo fields, review the deploy checklist, and four more. Seven were done within a month. The two that were not were "make order creation and payment atomic" and "reconcile payments and orders hourly".
In September the same harm happened through a different path. The postmortem found both items still open. Nothing about the first postmortem was wrong except its output: it made the important change look like one of nine.
1. Add log line for null promo (team) 2. Alert on order rate (team) 3. Update runbook 4. Tests for null fields 5. Review deploy checklist 6. Make order + payment atomic (backlog) 7. Hourly reconciliation (backlog) 8. Promo code review guideline 9. Training on pricing module
Change: order creation and payment capture succeed or fail together; a payment with no order within 60s is voided. Owner: Priya. Date: 31 Oct. Cost: ~8 days, saved-search moves a week (PM agreed). Also noticed (no owner): log line, alert, runbook wording, null-field tests.
In the first list the structural fix is item 6 of 9, owned by "backlog". In the second it is the only commitment, with a person, a date and an agreed cost — and the easy items are still visible, but nobody mistakes closing them for progress.
Choosing the one change
The question is not "what went wrong" — lots of things did — but "what single change would have made this impossible or harmless?" That question favours changes to the system over changes to people, and changes that cover the class of incident over changes that cover this path through it.
Which change gets the owner and the date?
when The incident was a one-off in a stable area and the class of failure cannot recur another way.
cost Cheap, and covers only this path. The next path to the same harm is untouched.
when The harm was mainly the time to detect, and faster detection would have kept it small.
cost Makes the next one shorter, not rarer. Worth doing; rarely the one change.
when The same customer harm can arise through several paths and has happened before.
cost Days of work and roadmap time; needs the PM to agree in the room. Removes the class of incident.
A skeleton that forces the choice
The format does much of the work. A template with a numbered action list invites ten items. A template with one "change" block, an owner field and a date field, and a separate "also noticed" list with no owner field, makes the choice explicit every time.
1# Postmortem: <what happened to customers, in one line>2 3## Customers4Who was affected, how many, what they lost, what they were told and given.5Customer clock: first harm -> last customer made whole.6 7## What happened8Timeline, short. Detection, mitigation, resolution. No names attached to mistakes.9 10## What made it possible11Contributing factors. As many as are true.12 13## The change14**What:** the single change that would have prevented this or made it harmless15**Why this one:** what class of incident it removes, not just this path16**Owner:** <one person>17**Date:** <yyyy-mm-dd>, agreed with <who controls their priorities>18**Cost:** <days, and what moves because of it>19 20## Also noticed21Things worth knowing. No owners, no dates. If one causes the next incident,22it becomes that postmortem's change.23 24## Status25<open | shipped yyyy-mm-dd, link> — reviewed weekly until shipped"Customers" comes first because the postmortem is read by people outside engineering, and that is what they need. "Also noticed" has no owner field on purpose.
Getting the date agreed in the room
The owner and date are only real if the person who controls the owner's time agrees to them in the meeting. That usually means the PM, because the change will displace something. The conversation is short if it is framed as a product trade, and long if it is framed as an engineering need (Tech Debt Is a Product Decision).
End of the postmortem meeting. The one change is about eight days. The PM is in the room and the saved-search feature is next on the roadmap.
"We have a few action items. The big one is making orders and payments atomic — we will try to get to it when we have capacity."
"The one change is making order and payment succeed or fail together. It is the second time customers have been charged without an order this year, and it will happen again through another path otherwise. It is about eight days; saved-search would move by a week. Can we agree Priya does it by 31 October? If not, I would like us to write down that we are accepting the risk."
How to do it
Most important first.
- Start from the customer impact and the timeline, briefly. Establish what happened to people and how long it lasted, then what made it possible.
- Ask one question: what single change would have prevented this, or made it harmless? Prefer a change to the system over a change to behaviour.
- Give that change a named owner — a person, not a team — and a date, agreed with whoever controls their priorities in the same meeting. If it needs roadmap space, the PM says yes or no in the room.
- Put everything else in a short "also noticed" list with no owner or date. If any of it is important, it will come back, and it does not dilute the main change.
- Review open postmortem changes in the weekly meeting until they are done, and say publicly when one slips and why (Done Means Someone Used It).
How to explain the decision
The sentences, the order, and what to lead with — for someone who did not make the call.
- Lead with the repeat: "This is the second time this year a customer has been charged without getting an order. The first postmortem listed nine actions; the one that would have prevented this was not done."
- Then the one change: "The change is to make order creation and payment capture succeed or fail together — if the order is not created, the payment is voided."
- Then the owner, the date and the cost: "Priya owns it, done by 31 October. It is about eight days of work and pushes the saved-search feature by a week. The PM has agreed."
- Then what you are not doing: "We noticed six other things. They are in the document. None of them would have prevented this, so none of them has a date."
- "But the other items are also real problems." They may be. They go in the "also noticed" list, and if one of them causes the next incident, it becomes that postmortem's one change. What we are not doing is pretending we will do ten things.
- "We cannot commit to a date without knowing the full scope." Then the first date is for the scope: "Priya will have a plan and an estimate by Friday." A date for finding out is still a date.
- "This is too big to fit in the roadmap." Then we should say so, in writing, to the people it affects: we are choosing to accept this risk, and here is what it could cost. That is a decision; letting the item age is not.
What can go wrong
- Choosing the change that fits in a sprint rather than the one that would have prevented the incident — the postmortem produces a monitoring tweak because the structural fix is too big to promise.
- Blame by other means: the "one change" is "engineer X will add a checklist step", which puts the fix on a person rather than the system.
- Treating the single item as a limit rather than a focus, so a second genuinely critical change — a security gap found during the incident — is left in the "also noticed" list.
- The date is agreed in the meeting and slips silently, because nobody asks about it after the postmortem is published.
- Writing the postmortem for engineers only, so the people whose trust the incident cost never see what was done.
- "More action items means a more thorough postmortem." More items means more to track and less chance the important one is done. Thoroughness belongs in the analysis, not in the list.
- "Blameless means nobody owns anything." Blameless means the analysis does not blame people for the failure. The fix still needs a person who owns getting it done.
- "The root cause is the one change." Incidents have several contributing factors; the one change is the one with the most leverage, which is not always the one closest to the trigger.
Knowing whether it worked
- The number of open postmortem items is small, and each has a person and a date that has not passed.
- Repeat incidents — the same customer harm through a different path — stop appearing.
- The PM and support lead read the postmortem summary and can say what changed without asking engineering.
- When the change ships, the postmortem is updated to say so, with a link, and the incident class it targeted is watched for a quarter.
- On a small team, the postmortem is a conversation and the one change is often done the same week. The discipline is still to name it and to write it down, so it is not rediscovered next year.
- At 10x team size, the change often lives in another team's system. The owner is then someone who can get it onto that team's roadmap, and the date is one they agree to, not one imposed in your meeting.
- At 10x incidents, patterns across postmortems matter more than any single one. Tagging each postmortem with the class of harm makes it possible to say "payment-without-order has happened four times this year" and to fund a structural fix across all of them.
- Picking one change means deliberately not doing others that seemed reasonable. Some of them will turn out to have mattered, and the team will say so.
- The structural change is usually expensive. Getting it a date means taking roadmap time, and someone has to accept a feature being later for it.
- Naming one owner puts visible pressure on one person. It needs to come with the time to do it, or it becomes a way of assigning blame for the next repeat.
Where this applies
Product advice is context-sensitive. These labels say what each claim is specific to, and where a different stage, team or product would differ.
- GENERALOne owned, dated change beats a long list anywhere action items compete with a roadmap. It differs where a separate reliability team has its own budget for postmortem work and the competition is weaker.
- PRODUCT-SPECIFICIn regulated products, a postmortem may have to list every identified gap with a remediation for auditors. Keep that list for compliance and still name the single change the team will prioritise first.
- SIMPLIFIED"One change" is a focusing rule, not a hard cap. An incident that exposes two independent critical gaps — say, a data-loss path and a security hole — can have two owned changes; the rule is against the long tail, not against a second genuine priority.
Where the depth lives
This domain teaches the product-side judgement and hands the mechanism off.