Reconstructing What Actually Happened
An evidence-based sequence of changes, signals and actions — built from records, because memory reorders events with total confidence.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
After the incident, how do you establish what happened and in what order, well enough to learn from it?
Everything people believe about an incident afterwards is reconstructed under stress, and reconstruction reliably compresses, reorders and rationalises — producing a story that is coherent and wrong.
After it is over, get everyone in a room and write down what happened. People were there; they will remember.
Under stress, minutes and tens of minutes feel the same. Durations are compressed, and the order of events that happened close together is frequently reversed.
- Under stress, minutes and tens of minutes feel the same. Durations are compressed, and the order of events that happened close together is frequently reversed.
- People narrate causally rather than chronologically. "The deploy broke checkout" gets written down as a sequence, when the evidence may show checkout broke before the deploy landed.
- Everyone saw a different part. Without records, the loudest or most senior account becomes the timeline.
- Hindsight makes the cause look obvious, so the timeline records that people should have seen it — which is not what happened and is not learnable from (Postmortems).
- The most important gaps disappear. Nobody writes down "we spent eleven minutes unable to find the dashboard", and that is exactly the finding worth acting on.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A timeline is a claim about ordering, supported by evidence. Every row should be traceable to a record: an alert, a deploy event, a log line, a chat message, a graph. Where it is not, the row says so.
- Four kinds of row do different work, and separating them prevents the commonest error. A change is something that altered production. A signal is something the system reported. An action is something a human did. A recovery is impact ending or being confirmed ended.
- The distinction between change and signal is where causal claims either become defensible or fall apart. If a change appears after the first signal, it cannot have caused it — the strongest and cheapest inference a timeline gives you (Deploys on the Same Timeline as the Symptom).
- Detection lag is measured by the gap between the first signal and the first human action, and it is usually the most actionable finding in the whole document. It is also the one nobody remembers, because from the inside it felt immediate.
- Timestamps must share a clock and a zone. Records from a deployment system, a metrics store, a chat platform and a log pipeline routinely disagree by seconds to minutes, and the disagreement is itself a finding (Clock Synchronisation).
- Absence of a record is information. "No alert fired" and "no annotation exists for this change" are rows worth writing.
A worked timeline
This is the same afternoon outage used elsewhere in this module, written as a review would record it. Note the belief rows, the gap rows and the explicit exoneration — none of those would survive a reconstruction from memory a week later.
Read the two numbers it yields: detection lag of two minutes, and eleven minutes between the first signal and the mitigation. Those are the findings; everything else is the evidence for them.
- 13:41 UTCchangecheckout-api v482 rollout begins (source: deploy pipeline event)
- 13:49 UTCchangeRollout completes; both versions served for 8 minutes (source: pipeline)
- 13:52 UTCchangeFlag
new-tax-calcenabled at 10% (source: flag platform audit log) - 14:03 UTCchangeTimeout config applied to shared payment gateway (source: infrastructure audit log; no deploy annotation existed for this system)
- 14:04 UTCsignalCheckout 5xx rises from baseline to ~4% (source: metrics)
- 14:06 UTCsignalSymptom alert fires and pages the on-call (source: alerting platform)
- 14:06 UTCactionAcknowledged (source: paging platform)
- 14:07 UTCactionImpact statement posted: ~4% of checkouts failing, all regions (source: incident channel)
- 14:09 UTCactionv482 and the flag exonerated: both precede onset by more than 10 minutes with no error change in between (source: dashboard with deploy overlay)
- 14:11 UTCactionGateway timeout config reverted (source: infrastructure audit log)
- 14:13 UTCrecoveryError rate returns to baseline (source: metrics)
- 14:16 UTCactionTeam believed impact had ended — belief, not evidence; recorded because it explains the next nine minutes
- 14:20 UTCrecoverySynthetic probe passing; support queue quiet (source: probe history, support tool)
- 14:25 UTCrecoveryRetry backlog of failed order confirmations drained; last affected user served (source: queue metrics)
- 14:31 UTCactionIncident closed; review scheduled (source: incident channel)
Two gaps are the findings. The gateway config system emitted no deploy annotation, so the only change that mattered was the only one not on the dashboard — that is an action item with an owner. And user impact ended at 14:25, not 14:13: the backlog drain took twelve minutes longer than the error rate suggested, which is the difference between "we recovered in nine minutes" and "users were affected for twenty-one" (Operating Queues and Scheduled Work).
Four kinds of row
Classifying each row is what makes the timeline analysable rather than narrative. It also makes the commonest analytical mistake visible: a causal claim where the change follows the signal.
The last column is what the row licenses you to conclude — which is usually less than people assume.
| Kind | What it records | Best source | What it supports concluding |
|---|---|---|---|
| change | Something altered production | Deploy, flag, config and infrastructure audit logs | Candidate cause if before onset; exoneration if after |
| signal | Something the system reported | Metrics, alerts, logs, probes | When it started and how bad — the impact arithmetic |
| action | Something a human did or decided | Chat log, command audit, paging platform | Response quality: detection lag, decision points, dead ends |
| recovery | Impact ending, and confirmation of it | Probes, support queue, backlog metrics | Actual user impact duration, which usually exceeds the error-rate window |
How timelines go wrong
Each of these produces a document that looks finished and teaches the wrong lesson. They are worth naming because they are hard to see from inside the team that wrote it.
The pattern across all of them is the same: a plausible narrative is easier to produce than an evidenced sequence, and nothing in the review process forces the difference.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Written a week later from memory | Durations compressed; close events reordered | Stress and hindsight reliably distort recall of ordering | Capture live with a scribe; pull machine records first (Roles During an Incident) |
| Sources with different clocks | A change appears after the symptom it caused | Skew between deploy, metrics and log systems | Normalise to one clock; treat skew as a defect worth fixing (Clock Synchronisation) |
| Change made outside the pipeline | A gap where the actual cause should be | Console edits and emergency sessions emit nothing | Route and record all production change; annotate break-glass sessions (Break-Glass Access) |
| Analysis written into the timeline | Rows assert causation | Narrative and evidence collapsed into one document | Timeline states what; analysis states why, in a separate section (Postmortems) |
| Hindsight applied to the responder | "Failed to notice the obvious signal" | The cause is obvious only after it is known | Record what was visible on the screens actually open at the time |
| Timeline ends at mitigation | Impact duration understated | Recovery work is not treated as part of the incident | Continue to the last affected user, including backlog and reconciliation |
How to do it properly
Most important first.
- Build it while the incident is happening. A scribe posting actions and observations with timestamps costs almost nothing and is dramatically more accurate than reconstruction (Roles During an Incident).
- Pull from systems first — alerts, deploy events, audit logs, flag changes, autoscaling events — and use human recollection to fill gaps and add reasoning, not to establish order (The Audit Trail).
- Record everything in UTC and render locally, so responders in different zones read the same sequence (Timezone and DST Failures).
- Include what people believed at each point, marked as belief rather than fact. "14:22 — team believed the cause was connection reuse" is how a reader understands why the next twenty minutes went the way they did.
- Record the gaps explicitly: time spent finding a dashboard, waiting for access, waiting for a build. These are the rows that produce useful action items (Action Items That Change the System).
- Separate the timeline from the analysis. The timeline says what happened; the analysis says what it means. Mixing them is how a contributing factor becomes "the root cause" without anyone deciding to say that (Root Cause vs Contributing Factors).
- Mark uncertainty in place. "Approximately 14:05, from a chat message" is more useful than a precise-looking time nobody can support.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A wrong timeline contains nothing and misdirects the action items, so the same incident is prevented on paper and recurs in production.
What can go wrong
- A timeline written to justify the response rather than describe it.
- Precise timestamps with no source, which look authoritative and cannot be checked.
- Clock skew between systems placing a change on the wrong side of a symptom, which inverts the causal reading.
- Missing rows for changes that bypassed the pipeline — the manual fix, the console change, the emergency access session (Manual Production Changes).
- Hindsight framing: "the responder failed to notice" instead of "the signal was on a dashboard nobody had open" (Postmortems).
- The timeline stops at mitigation, omitting the recovery work — backlog drain, reconciliation — which is often where the real user impact ended.
- "The timeline shows the root cause." It shows ordering. Cause is an inference from ordering plus a mechanism you can state (Root Cause vs Contributing Factors).
- "Events close together are related." At second-level resolution, coincidence is common, especially on a platform where many teams deploy.
- "We remember it clearly, so we do not need records." Confidence in incident recall is high and accuracy is not; this is the specific reason the practice exists.
- "The timeline is the postmortem." It is the evidence base. The learning happens in the analysis built on top of it (Learning Across Incidents).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Every row cites a source, or is explicitly marked as recollection.
- The timeline changed someone's mind. If it only confirmed what everyone already believed, it was probably written from belief.
- Detection lag and mitigation lag can be read directly off it as numbers.
- Gaps and dead ends appear, not just the path that turned out to be right.
- A timeline is a living document during the review. Correcting a row when better evidence appears is normal, and the correction should be visible rather than silent — an amended timeline that quietly changed is worse than one that never did.
- If two sources genuinely conflict and cannot be reconciled, record both. A single confident wrong ordering is more damaging than an acknowledged ambiguity.
- Automate collection: alerts, deploys, flag flips, config applies, scaling events and access sessions into one timestamped feed, in one zone.
- Automate the chat log capture so the responders' own messages are already timestamped rows.
- Automate the arithmetic — detection lag, time to mitigate, total impact duration — so nobody computes it by hand or by feel.
- Do not automate the interpretation. What the ordering means, and which factors contributed, is analysis that requires knowing the system (Five Whys, and Its Limits is one technique among several, not the method).
- A rigorous timeline takes hours to assemble, on a team that has just lost a night. That cost is real and is why timelines get skipped.
- Evidence-only timelines miss reasoning; recollection-heavy ones invent order. The workable answer is both, clearly labelled, which is more work than either.
- Building it live requires a scribe, which is a person not investigating.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALMemory's unreliability about ordering under stress is not a property of any stack. The evidence sources differ; the need to prefer them over recollection does not.
- TOOL-SPECIFICHow much assembles itself depends on whether your platforms emit machine-readable events with consistent clocks. Where deploys, flags and infrastructure changes live in separate systems with no shared feed, the timeline is manual work and correspondingly less complete — which is an argument for building the feed, not for skipping the timeline.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — how evidence quality determines whether a postmortem changes the system or just describes it.