Learning Across Incidents
The learning that only exists in aggregate — patterns across incidents, near misses, and making the record something people actually read.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How does an organisation learn something no single incident could have taught it?
Each incident is analysed well and filed. Nobody reads across them, so a condition that contributed to eleven incidents looks like eleven separate unlucky events, and the near misses — where the same conditions held and someone got lucky — are not recorded at all.
We write a postmortem for each incident and store them in the wiki. The learning is in the documents.
Documents are written once and read never. The people who would benefit most do not know which one to read, because they do not yet know what they are about to hit.
- Documents are written once and read never. The people who would benefit most do not know which one to read, because they do not yet know what they are about to hit.
- Patterns are only visible across incidents. "Config change with no validation" contributing to eight incidents in a year is a platform investment; seen one at a time it is eight ordinary tickets.
- Near misses are the cheapest lessons available and are almost never captured. The canary that caught a bad release, the alert that fired at 2% before customers noticed — these are the same information as an outage at a fraction of the price.
- Knowledge concentrates in whoever has been on-call longest. That is a single point of failure with a notice period.
- Without aggregate reporting, the postmortem process cannot demonstrate value, so it loses the argument for time against roadmap work and decays into a form.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Individual analysis and aggregate analysis answer different questions. A postmortem asks "what happened here?". Aggregation asks "what keeps happening?" — and only the second one can justify structural investment.
- For aggregation to be possible, incidents have to be recorded in a comparable shape. This is the real reason for a template: not consistency for its own sake, but that contributing factors, detection times and affected services can be counted.
- Detection time is the most under-used aggregate signal in most organisations. Its distribution across incidents tells you whether you have a monitoring problem, and it is nearly free to collect (Alert on Symptoms, Not on Causes).
- Near misses have the same causal structure as incidents with none of the impact, so they are strictly cheaper to learn from. Capturing them requires that raising one costs almost nothing and produces no suspicion.
- Distribution beats storage. An incident review where the responders narrate the timeline to people who were not there transfers far more than the document does, because the audience asks the questions the document did not anticipate.
- Repeat incidents are the honest measure of whether learning is happening. If the same contributing factor appears twice with the follow-up closed in between, the follow-up was not a change (Action Items That Change the System).
From one incident to a system change
This is the path a lesson takes from a single event to something structural. Most organisations complete the first three steps well and stop. The value is concentrated in the last three, which is also where there is no deadline forcing the work.
- 1Respond
Impact is stopped and the sequence of events is captured while it is happening.
fails by No contemporaneous record, so the timeline is later reconstructed from memory.
evidence An incident channel or log with timestamps, kept without effort.
- 2Analyse
Contributing factors identified from the timeline.
fails by Single root cause; analysis stops at the trigger (Root Cause vs Contributing Factors).
evidence Several factors, each with a counterfactual.
- 3Act
Specific system changes with owners.
fails by Intentions instead of changes (Action Items That Change the System).
evidence Merged diffs, tracked to closure.
- 4Share
The account reaches people who were not there.
fails by Filed in a wiki nobody searches.
evidence Someone outside the responding team can explain the failure mode.
- 5Aggregate
Factors, detection times and repeats counted across incidents.
fails by Records not comparable, so nothing can be counted.
evidence A frequency table someone actually looked at this quarter.
- 6Invest
A repeated factor becomes a platform change or a readiness requirement.
fails by The pattern is identified and owned by nobody.
evidence The class of incident measurably declines (Production Readiness Review).
The stall is almost always between Act and Aggregate. Both ends have natural forcing functions — an outage forces the first, a quarterly plan forces the last — and the middle has neither.
Signals worth aggregating, and how each one lies
Every one of these can be gamed, and most of them will be if they become targets. That is not an argument against collecting them; it is an argument for reading them together, and for keeping them as signals rather than goals — the same caution the DORA metrics need (Learn and Improve).
| Signal | What it tells you | How it lies |
|---|---|---|
| Contributing factor frequency | Where structural investment would pay | Depends entirely on tagging discipline; a factor with no tag is invisible |
| Time to detect | Whether your monitoring covers real failure modes | Only counts incidents you detected at all; the ones you never noticed are absent |
| Time to mitigate | Whether recovery paths are fast and rehearsed | A slow mitigation may be the correct careful one; read with the timeline |
| Repeat rate for a factor | Whether action items actually changed the system | Sensitive to tag granularity — coarse tags hide repeats, fine tags invent them |
| Action item closure rate | Whether follow-up is funded | Closing tickets is easy; closing them with merged changes is the claim (Action Items That Change the System) |
| Near-miss volume | Reporting health, more than system health | Falls when people feel judged, which looks exactly like improvement |
| Incident count | Very little on its own | Falls when declaring gets discouraged; the worst single metric here (Severity: What It Should Reflect) |
Near misses, and why nobody reports them
A near miss is an incident that did not land: the canary caught the bad release, the alert fired before customers noticed, the migration was applied to staging by luck rather than by design. Causally it is identical to an outage. It costs nothing.
The reason they go unreported is straightforward — reporting one means volunteering that you nearly caused an outage, in an organisation where the consequences of that are usually unstated. The volume of near-miss reports is therefore a measurement of psychological safety wearing the costume of a reliability metric.
Making them cheap to raise is most of the work: a few sentences, no meeting, no template, no analysis unless a pattern emerges. What matters is that the conditions get recorded somewhere they can be counted.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Canary caught a bad release at 5% traffic | Rollout halted automatically; no user impact | The same defect that would have been a full outage without the canary | Record it: this is evidence the canary is worth its cost, and the defect class is still worth a test (Canary Analysis: Compared Against What?) |
| Destructive migration run against staging | Staging data lost; production untouched by luck | Identical conditions to a production incident; the target happened to be right | Treat as a production near miss and apply the production fix (Destructive Migrations) |
| Certificate renewed manually with two days left | No outage; automation had silently failed weeks earlier | Renewal automation with no alert on its own failure | Alert on renewal automation health, not only on certificate expiry (Renewal: Automating the Thing That Expires) |
| Alert fired at 2%, engineer rolled back immediately | Six minutes of low-level errors, no customer contact | A defect that reached production; detection and response both worked | Record what worked — this is the "what went well" evidence that otherwise never gets written down |
| Capacity headroom nearly exhausted during a failover test | Test passed; margin was much thinner than assumed | Capacity model not revalidated since traffic grew | Revalidate the failover capacity assumption before it is needed for real (Capacity During Failover) |
How to do it properly
Most important first.
- Tag contributing factors from a small, stable vocabulary — config, migration, dependency, capacity, rollout, permission, expiry, monitoring gap — so they can be counted. A free-text field cannot be aggregated.
- Review incidents in aggregate on a regular cadence. Look at factor frequency, detection time distribution, repeat rate and action-item closure, not at a count of incidents.
- Capture near misses with a lightweight path — a few sentences, no meeting — and treat the volume as a health signal rather than as a problem.
- Run incident reviews as narrated walkthroughs open to anyone, with the responders talking through the timeline and the decisions as they looked at the time.
- Make incident records searchable by symptom, so the engineer seeing an unfamiliar error can find the three previous times it happened.
- Feed the aggregate back into the readiness process: the factors that keep appearing become checklist items (Production Readiness Review).
- Report closure rate and repeat rate to whoever funds the time. The process needs evidence for its own value like anything else.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained only by luck. The failure mode is organisation-wide and slow: repeated incident classes, a knowledge monoculture, and a process that decays into paperwork.
What can go wrong
- Metrics becoming targets. Counting incidents makes people avoid declaring them, and the reporting improves while the system does not (Severity: What It Should Reflect).
- A taxonomy that grows to sixty tags, which is a free-text field with extra clicks.
- Aggregate review as a status meeting: numbers presented, no decisions, no owner for the pattern that was just identified.
- Near-miss reporting that quietly becomes evidence in performance conversations. It stops within a month of the first time, and permanently.
- Learning that stays inside one team while a neighbouring team hits the identical failure a quarter later, because nothing crossed the boundary.
- A wiki of two hundred postmortems with no search, no tags and no index — technically a record, functionally a landfill.
- "Fewer incidents means we are learning." It may mean people stopped declaring them. Repeat rate and detection time are harder to game than a count (Severity: What It Should Reflect).
- "Near misses are not worth writing up because nothing happened." Nothing happened *this time*, and the conditions are identical. They are the cheapest data you will ever get.
- "We have a wiki, so we have institutional memory." Memory requires retrieval. An unsearchable archive is storage.
- "Aggregate analysis is for large organisations." Three services and one team still benefit from noticing that the last four incidents were all config changes with no validation (Validate at Startup, Fail Clearly).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- You can answer "what caused most of our incidents last quarter?" from records rather than from opinion.
- A structural investment was made because a factor appeared repeatedly, and the incidents in that class fell afterwards.
- Near misses are reported at a healthy rate by people who were not obliged to report them.
- Repeat rate for the same contributing factor is tracked and is falling.
- An engineer who was not involved cites a past incident during a design review.
- Process weight is the reversible part. If tagging, reviews and near-miss reporting have grown into a burden that suppresses reporting, cut them back — under-reporting is much more damaging than a coarse taxonomy.
- A metric that has become a target should be removed from reporting rather than reinterpreted. Once people are optimising it, it no longer measures what it did.
- Automate aggregation: factor frequency, detection time distribution, repeat rate and action-item closure, generated from the records rather than assembled by hand for each review.
- Automate discoverability — full-text search over incident records, surfaced from the alert or the error message where possible, so the record reaches the person at the moment it is relevant.
- Do not automate the pattern-finding conclusion. A cluster of tags is a prompt for a conversation, not a finding; the interesting patterns are usually the ones the taxonomy does not have a tag for.
- Do not automate near-miss detection into a scoreboard. The moment volume is judged, reporting stops.
- Structured records are more useful in aggregate and more effort per incident. Push the structure too far and you suppress the writing that feeds it.
- Open reviews spread knowledge and increase the exposure of the people who responded. In a low-trust organisation, that trade goes the wrong way.
- Aggregate review costs recurring senior time — the people whose time it costs are the ones who can act on the findings, which is exactly why it works and why it is hard to protect.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- ORG-SPECIFICCadence, audience and formality scale with size. A five-person team learns across incidents by having been in all of them; at fifty engineers that stops working and the aggregate has to be built deliberately. Regulated industries may additionally owe external trend reporting on a fixed schedule.
- SCALE-SPECIFICAggregate patterns need enough incidents to be visible — roughly a dozen a year before frequency counts mean anything. Below that, read the records qualitatively and do not compute rates from four data points.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — turning a recurring contributing factor into a check that runs before production rather than a lesson learned after it.