Debugging an Incident in Progress
Mitigation and diagnosis are different jobs, and doing them in the wrong order costs users minutes they never get back. Stabilize first, then form a hypothesis you can disprove in two minutes instead of browsing dashboards hoping something looks odd.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Stabilize first, diagnose second
These are two different jobs and they compete for the same people. Mitigation restores service: roll back the deploy, shift traffic away from a region, shed non-critical load, raise a pool size, disable the expensive feature flag. Diagnosis explains why it happened. The instinct to understand before acting is a good engineering instinct and a bad incident instinct — every minute spent understanding is a minute of user-visible damage that a rollback would have stopped.
The exception that proves the rule: mitigate blindly only when the mitigation is cheap and reversible. Rolling back a deploy is cheap. Restarting the process that is leaking memory is cheap and buys hours (Memory Leaks: Growth That Does Not Come Back). Truncating a table because you suspect it is large is not — irreversible mitigations need the diagnosis first. Sort your options by "what does this cost if I am wrong about the cause", and start with the ones that cost nothing.
Say out loud which job you are doing. Incidents go badly when half the responders think they are mitigating and half think they are diagnosing, because the mitigators keep changing the system the diagnosers are measuring. One person mitigates, one person diagnoses, one person communicates — and the diagnoser notes every mitigation as an event on the timeline, because a rollback at 14:12 explains a metric change at 14:13 that would otherwise look like evidence.
Write the hypothesis before you look
The failure mode of incident diagnosis is not ignorance, it is confirmation bias at speed. You open a dashboard, something looks slightly elevated, and thirty minutes later four people are optimizing it. Writing the hypothesis down first — *and the evidence that would disprove it* — turns dashboard browsing into an experiment with an outcome.
A usable hypothesis names a mechanism and predicts a specific reading. "The database is slow" predicts nothing. "The 14:00 deploy added an unindexed filter to the orders list query, so database time per request rose while queries-per-request stayed flat" predicts that DB time per request roughly doubled, that rows_scanned jumped, and that the p99 rise is confined to the endpoints that use that query. Three checks, two minutes, and you are either right or moving on.
Prefer hypotheses you can kill fast over hypotheses that feel likely. The cheapest check first is not a heuristic about probability, it is a heuristic about time: you can afford four wrong two-minute hypotheses; you cannot afford one wrong forty-minute one. This is the same discipline Measure Before You Optimize applies outside an incident, compressed into a much less forgiving clock.
114:05 "latency is up"214:07 opens the service overview314:09 "CPU looks a bit high on node-3"414:12 three people investigating node-3514:26 node-3 is fine; it is always like that614:28 "maybe it is the database?"714:31 opens database dashboard8...915:10 someone finally checks the deploy log1H1: the 14:00 deploy is the cause.2 predicts → p99 step-changes at 14:00-14:03, not before3 check → deploy marker vs p99 chart [40s]4 RESULT: p99 was already climbing from 13:40. KILLED.5 6H2: a dependency degraded before our deploy.7 predicts → upstream span duration up from ~13:408 check → trace p99 split by downstream service [90s]9 RESULT: payment-api span 40ms → 1.9s from 13:38. HOLDS.10 11H3: our retries are amplifying it.12 predicts → payment-api RPS up while our RPS flat13 check → outbound RPS vs inbound RPS [60s]14 RESULT: outbound 3.1x inbound. HOLDS — and explains the shape.Both engineers looked at the same dashboards. The second one wrote down what each chart would have to show, so a chart that did not show it ended the theory instead of decorating it. Three minutes of checks replaced an hour of plausible-sounding investigation, and H1 — the theory everyone believes by default — died first.
The triage order that fits on a sticky note
Four questions, in this order, resolve most incidents to a layer before anyone opens a profiler. What changed? — deploys, config, flags, dependency releases, data migrations, traffic ("What Changed?" — Deploy Markers and the Invisible Deploys). Who is affected? — all users or one region, one tenant, one endpoint, one client version; a narrow blast radius names the variable. Which layer? — split the request duration across gateway, app, database, cache and dependencies from traces (The Critical Path Is the Only Path That Pays). Errors or saturation? — a system returning errors fast and a system serving slowly from a full queue need opposite responses.
That last split matters more than it looks. Errors with idle resources point at a dependency, a bad deploy or a config change. Rising latency with saturation points at capacity or a bottleneck resource, and the fix is shedding load or adding capacity, not rolling back (Saturation: The Reading Utilization Cannot Give You, Queueing: Why Systems Get Slow Before They Get Broken). Responding to a saturation incident with a rollback wastes the mitigation window; responding to a bad-deploy incident by scaling up adds broken capacity.
Record the answers in the incident channel as you get them, with timestamps. Not for process compliance — because the timeline you are unconsciously building in your head is the artifact the review needs (Reading a Timeline: Observation Order Is Not Causal Order), and human memory of a stressful forty minutes is reliably wrong about ordering.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| API p99 (all routes) | 380ms → 4.2s | Confirms the symptom. Tells you nothing about the cause. | suspect |
| p99 split by route | Only `/checkout` affected | Blast radius is one code path — the variable is in that path, not in the platform. | smoking gun |
| Error rate | 0.2% (baseline 0.15%) | Requests are completing, just slowly. This is a latency incident, not a failure incident. | normal |
| App CPU utilization | 22% | The service is waiting, not computing. Rules out our own CPU (Computing or Waiting?). | normal |
| Deploy markers | None in 6 hours | Kills the default hypothesis immediately. Look outward, not at our code. | suspect |
| Downstream span p99 | payment-api 40ms → 1.9s | Names the layer. Everything upstream is a victim, not a cause. | smoking gun |
Key points
- Mitigation and diagnosis are separate jobs — stabilize with cheap reversible actions first, and let the diagnosis continue with the clock stopped.
- Write the hypothesis and its falsifier before opening the dashboard; a check that cannot disprove anything is browsing, not debugging.
- Prefer hypotheses you can kill in two minutes over hypotheses that feel likely — four cheap wrong guesses cost less than one expensive one.
- Four triage questions resolve most incidents to a layer: what changed, who is affected, which layer holds the time, errors or saturation.
- Every mitigation is an event on the timeline; a system being changed while it is being measured produces evidence that means nothing.
Progressive depth
Overview
An incident is a symptom plus a clock. Stop the user-visible damage with something cheap and reversible, then work out why, in that order.
Practical
Run the four triage questions — what changed, who is affected, which layer holds the time, errors or saturation — and write each hypothesis with the reading that would kill it before you open the chart.
Advanced
Separate victims from causes. In a distributed system a retry storm, a full connection pool and a saturated queue all produce "the database is slow" dashboards while the database is behaving normally (Retry Storms: The Load You Generated Yourself, Connection Pool Saturation: Waiting in Front of an Idle Database). Attribute time from traces rather than reading per-service dashboards, because per-service dashboards cannot tell waiting from working.
Internals
Detection lag differs per signal, which systematically distorts the timeline: a 1-minute-window alert fires later than a 10-second scrape, metric aggregation windows smear step changes into ramps, and trace sampling can miss the first slow requests entirely (Sampling Without Throwing Away the Evidence). The timestamps on your dashboard are when a measurement crossed a threshold, not when the system changed — and the gap between those is where most incorrect causal orderings come from.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Page → responder: a threshold crossed; the alert names a symptom, never a cause.
- 2Responder → golden signals: latency up, errors flat, traffic flat — a latency incident with no failure component, so the work is completing but taking longer somewhere.
- 3Responder → dimensional split: p99 elevated on one route only, so the cause lives in that code path or one of its dependencies, not in shared infrastructure.
- 4Responder → trace attribution: the added time sits in one downstream span; every upstream metric is measuring the wait, not the work.
- 5Responder → dependency: the downstream service degraded first, so our service is a victim, and our own CPU, memory and deploys are all irrelevant evidence that looked relevant.
- • "Latency rose right after our deploy, so the deploy caused it" — check whether the rise actually started before the marker; deploy-shaped coincidences are extremely common because deploys are frequent (Correlation Is Not the Root Cause).
- • "CPU is high on one node, that is the problem" — one hot node in a fleet is usually normal variance; compare against the fleet distribution before investing anyone in it.
- • "The database dashboard is red" — the database being slow and the database being *the cause* are different claims; a database overwhelmed by retried traffic is also a victim (Retry Storms: The Load You Generated Yourself).
- • "Nothing changed, so it cannot be a change" — config, feature flags, dependency deploys, certificate rotations, cron jobs and data growth are all changes that leave no deploy marker.
- • "It resolved itself, so it was transient" — self-resolving incidents usually mean something drained: a queue, a cache filled, a load balancer ejected a bad node. Something *acted*; find out what.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Golden signals for the affected service first: rate, errors, duration, saturation — thirty seconds, before any theory ([[golden-signals]]).
- • p99 split by route, region, tenant and client version — the dimension where the split appears is the blast radius, and the blast radius names the variable.
- • Request duration attributed across layers from traces: gateway, app, cache, database, each downstream dependency ([[critical-path]]).
- • Deploy, config and feature-flag change events on the same time axis as the latency chart ([[deployment-markers]]).
- • Resource saturation for the affected tier: CPU run queue, pool wait time, queue depth — to separate "slow because waiting" from "slow because computing".
- • Apply the cheapest reversible mitigation that plausibly restores service — rollback, traffic shift, flag disable, load shed — before the diagnosis is complete.
- • Assign roles explicitly: one mitigator, one diagnoser, one communicator, so nobody is changing the system another person is measuring.
- • Work the triage order (changed / affected / layer / errors-vs-saturation) instead of the loudest chart.
- • Log every action taken, with timestamps, into the incident channel — this is the timeline the review will need and memory will not supply.
- • Only after the mechanism is confirmed, ship the permanent fix, and pair it with the guard that would have caught it ([[performance-regression-detection]]).
- • The user-facing SLI recovers to its pre-incident baseline, not merely "looks better" — compare against the same weekday and hour, since traffic shape confounds a same-day comparison ([[sli]]).
- • The mechanism you claimed predicts the recovery shape: if you rolled back a query change, DB time per request should return to baseline while queries per request stays flat.
- • Undo the mitigation deliberately once the fix is live, and confirm the system stays healthy — a mitigation left in place forever hides whether the fix worked at all.
- • The originating signal (the downstream span, the pool wait, the queue depth) is back in range, not just the aggregate that paged you.
- • Mitigating before diagnosing sometimes destroys evidence — a rollback can erase the state that would have explained the bug, so capture a heap dump, a profile or a trace sample before you restart anything you cannot reproduce.
- • Role separation costs coordination overhead, which is a bad trade for a two-minute incident and an excellent one for a two-hour incident.
- • Fast-falsification discipline slows the first five minutes and saves the next fifty; under extreme pressure people abandon it precisely when it pays most.
- • Alerting on the revealing signal adds another alert to a rotation that is probably already noisy — pay for it by deleting one that has never led to an action ([[alert-fatigue]]).
- • Add the alert on the signal that actually revealed the cause, not the one that paged you — those are usually different signals, and the revealing one detects it earlier (Alerts Worth Waking Someone For).
- • Write the regression test or load-test scenario that reproduces the condition, so the next occurrence fails in CI rather than in production (Load Testing: What Question Is This Test Answering?).
- • Record the incident review including "why this was hard to detect" — the detection gap is the reusable finding, and the fix for it outlives this particular bug.
- • If a mitigation worked, automate it: a runbook step that a human executed under stress at 3am is a candidate for an automatic response.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe triage signal panel and the hypothesis log are teaching examples with invented numbers. Real incidents rarely present this cleanly — the shape of the reasoning transfers, the values do not.
- ENVIRONMENT-SPECIFICWhich mitigations count as "cheap and reversible" depends entirely on your deployment setup. A rollback is cheap with immutable deploys and blue-green traffic; it is a two-hour operation with in-place migrations.