Runbooks
A document written for someone tired, under pressure, who did not build this: symptom, possible causes, checks, mitigation, escalation, and how to verify recovery.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What does an operator actually need in front of them at 3am, and how is that different from documentation?
The person responding to an alert is frequently not the person who built the service, is frequently not fully awake, and has minutes rather than hours. Architecture documentation is written for a reader with time and context. They have neither.
The service is documented — architecture, API, deployment. Anyone can read that and work out what to do.
Documentation is organised by structure and a runbook is needed by symptom. The operator has an error rate graph, not a component name; nothing in the architecture doc is indexed by what they can see.
- Documentation is organised by structure and a runbook is needed by symptom. The operator has an error rate graph, not a component name; nothing in the architecture doc is indexed by what they can see.
- Reading comprehension under sleep deprivation and time pressure is genuinely worse. Prose that is clear at 2pm is not clear at 3am, and the failure is not a character flaw.
- Without a written mitigation, every incident is improvised, which makes response time a function of who happened to be on call.
- Knowledge stays with whoever built it, so on-call is only really staffed by two people regardless of how many are on the rotation (Rotations People Can Sustain).
- Nobody records how to tell it actually worked, so incidents get closed on the basis that the action was taken rather than that impact ended.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A runbook is indexed by symptom, because a symptom is what the operator has. Everything else in the entry hangs off that: the symptom is the lookup key, and if the operator cannot find their symptom the rest of the document is inert.
- Six parts do the work. Symptom — what you are seeing, phrased as it appears on a dashboard or in an alert. Possible causes — the small set of things that produce this, ordered by likelihood. Checks — the specific commands or panels that distinguish them. Mitigation — what stops user impact, with its own risks stated. Escalation — who to call, when, and what to tell them. Verification — the signal that confirms recovery.
- Verification is the part that is skipped and the part that prevents the second incident. An action taken is not a recovery observed, and the signal that confirms it should be the same one that detected the problem (Alert on Symptoms, Not on Causes).
- A runbook encodes understanding, not ritual. Each step should say what it is for, so an operator facing a variant nobody anticipated can reason from it rather than following it off a cliff (Runbook Anti-Patterns).
- Runbooks decay in a specific way: the system changes and the document does not. The countermeasure is that they are exercised — read during real incidents and during rehearsals — because an exercised document reveals its own errors.
Six parts, in the order an operator needs them
The ordering is not stylistic. An operator arrives with a symptom and a clock. They need to know what stops the bleeding before they need to know why it is bleeding, and they need to know how to tell it stopped before they can close the incident.
- 1Symptom
Names what the operator is seeing, in the words the alert or dashboard uses.
fails by Titled by component instead of symptom, so it cannot be found from what you can see.
evidence The alert links straight to this entry.
- 2Possible causes
Lists the small set of things that produce this symptom, most likely first.
fails by Exhaustive and unranked, so it reads as a list of everything that could ever go wrong.
evidence The causes match what has actually happened, drawn from incident records.
- 3Checks
Gives the specific commands or panels that distinguish the causes, and says what each result means.
fails by Commands with no interpretation, leaving the operator with output they cannot read.
evidence Each check has a stated meaning for its likely outputs.
- 4Mitigation
States what stops user impact, and what each option risks.
fails by A single unconditional action with no stated risk (Runbook Anti-Patterns).
evidence Mitigations that have actually been used, with their side effects written down.
- 5Escalation
Says when to escalate, to whom, and what to bring.
fails by "Contact the team" — no threshold, no rotation, no information to hand over.
evidence A named rotation and a time or severity trigger (Roles During an Incident).
- 6Verification
Names the signal that confirms impact has ended and the value that counts as recovered.
fails by Omitted, so incidents close on action taken rather than on impact ended.
evidence The same signal that detected the problem, back within its normal band.
One entry, written out
This is a complete entry for a common failure. Notice how much of it is reasoning rather than instruction — what each check means, what each mitigation costs, when to stop and escalate. That is the difference between a runbook and a list of commands.
Notice also that the last section is not optional. Without it, "we restarted the consumers" gets recorded as a resolution, and the incident reopens forty minutes later.
1SYMPTOM2 Alert: checkout_queue_age_seconds > 300 for 5m3 Users see: orders confirmed in the UI but no confirmation email,4 and no fulfilment record. Impact grows with the backlog.5 6POSSIBLE CAUSES (most likely first)7 1. Consumers crash-looping - most common, usually after a deploy8 2. Downstream dependency slow - fulfilment API latency9 3. Genuine traffic spike - backlog with healthy consumers10 4. Poison message - one message failing repeatedly11 12CHECKS13 Consumer health:14 kubectl get pods -l app=order-consumer15 -> CrashLoopBackOff or high RESTARTS means cause 1. Check the deploy16 timeline before the logs: a restart loop that began within minutes17 of a rollout is a bad release, not a runtime problem.18 Consumer logs:19 kubectl logs -l app=order-consumer --tail=100 --previous20 -> The SAME message id failing repeatedly means cause 4, not cause 1.21 A crash loop and a poison message look identical in pod status.22 Dependency latency:23 Dashboard: checkout-overview, panel "fulfilment API p99"24 -> Above ~2s with healthy consumers means cause 2. The consumers are25 working; they are blocked, so restarting them will not help.26 Arrival rate:27 Dashboard: checkout-overview, panel "queue arrival rate"28 -> Well above the recent baseline with healthy consumers and normal29 dependency latency means cause 3.30 31MITIGATION32 Cause 1 (bad release):33 Roll back the consumer deployment. Restarting without rolling back34 reproduces the crash, and each restart drops in-flight messages.35 kubectl rollout undo deployment/order-consumer36 Risk: in-flight messages are redelivered. Handlers are idempotent by37 message id, so redelivery is safe; verify no duplicate emails after.38 Cause 2 (slow dependency):39 Do NOT restart consumers - they are healthy and blocked. Reduce40 concurrency to stop adding load, and page the fulfilment team.41 Risk: lag grows while the dependency recovers. That is the correct42 trade; the alternative amplifies their outage into ours.43 Cause 3 (traffic spike):44 Scale consumers up. Check the connection budget first - each consumer45 holds 4 database connections and the pool ceiling is 200.46 Risk: scaling past the pool exhausts connections for the API too.47 Cause 4 (poison message):48 Move the offending message to the dead-letter queue by id. Do not49 purge the queue - that discards valid orders alongside the bad one.50 Risk: the message is a real order and needs manual reconciliation.51 Record the id in the incident channel.52 53ESCALATE54 - Backlog still growing 15 minutes after mitigation -> payments-oncall55 - Cause 2 confirmed -> fulfilment-oncall now56 - Any suspicion of data loss or duplicate charges -> incident commander,57 regardless of backlog size58 59VERIFY RECOVERY60 1. checkout_queue_age_seconds back below 60 and falling.61 2. Arrival rate and processing rate roughly equal - a falling age with62 processing below arrival means you are still behind, just less so.63 3. Spot-check three recent orders for a fulfilment record.64 4. Only then close. Consumers running is not recovery; the backlog65 being drained is.The two most valuable lines are the ones telling you *not* to act: do not restart healthy-but-blocked consumers, and do not purge the queue. A runbook that only lists actions will get those wrong under pressure, because acting feels better than waiting.
Keeping it true
Runbooks fail by decay, and decay is silent: nothing breaks when a runbook becomes wrong, which is exactly the property that makes it dangerous. The defences are all forms of exercising it, because an exercised document reports its own errors.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Service rearchitected | Commands reference components that no longer exist | Runbook lives apart from the code that invalidated it | Keep it in the service repository so the change that breaks it appears in the same review |
| Dashboard reorganised | Panel links 404 mid-incident | Links to mutable dashboard state | Link to stable dashboard ids; check links in the same job that checks the alerts (Dashboards an Operator Can Act On) |
| Never exercised | First real use reveals three wrong steps | Written from intention rather than from execution | Exercise in game days and drills; a step nobody has run is a hypothesis |
| Mitigation automated | The manual path in the runbook is stale when the automation fails | Automation replaced the practice, not just the toil | Keep the manual path documented and rehearsed — you meet it only on the worst day (The Automation Trap) |
| Grew by accretion | Forty steps; the operator cannot find the relevant one | Every incident added a step and none removed one | Prune when adding; a runbook is read under time pressure and length is a real cost (Action Items That Change the System) |
| Assumes unavailable access | Command refused mid-incident | Written by someone with broader permissions than the on-call rotation has | Write and test as the on-call role; if elevation is needed, the runbook says how to get it (Break-Glass Access) |
How to do it properly
Most important first.
- Write one entry per symptom, and make the alert link directly to its entry. The lookup should cost zero seconds.
- Put the mitigation before the explanation. Stopping impact comes first; the operator can read the background afterwards (Stop the Harm Before You Understand It).
- State what each check is for and what each result means, not just the command.
kubectl get eventsis a command; "check forOOMKilledevents — repeated kills mean the limit is too low for current traffic, not that the process is unhealthy" is a check (OOMKilled: Over the Memory Limit). - State the risk of each mitigation. "Restarting drops in-flight requests" is the sentence that lets an operator decide rather than guess.
- Give escalation a trigger and a script: when to escalate, to whom, and what information to bring. An escalation with no threshold happens too late.
- End with verification: the specific graph, query or endpoint that shows impact has ended, and what value counts as recovered.
- Update it from incidents. The moment a runbook is found wrong during a real incident is the moment it is most cheaply fixed and most likely to be believed.
- Test entries in rehearsals — game days, failover drills, restore drills — because untested runbooks are wrong at a rate nobody expects (Restore Drills).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained to the service, but the effect is on duration rather than scope: a missing runbook makes every incident on that service longer.
What can go wrong
- A runbook written once at launch and never revisited, describing a service that has since been rearchitected.
- Steps with no reasons, which produce ritual: the operator performs them in a situation where they do not apply, because nothing told them what the step was for.
- A wall of prose with the mitigation buried in paragraph four.
- Commands that no longer work — a renamed flag, a moved dashboard, a decommissioned host — discovered during an incident.
- Every possible failure documented at equal weight, so the common ones are hard to find among the exotic ones.
- A runbook that assumes access the on-call engineer does not have, discovered at the moment the command is refused (Break-Glass Access).
- Escalation described as "contact the team" with no path, no threshold and no rotation name.
- "A runbook is documentation." Documentation explains the system to a reader with time. A runbook is an operational tool for a reader without it. Different audience, different structure, different index.
- "Every possible failure needs an entry." Cover what actually happens plus the catastrophic-but-plausible. Exhaustive runbooks are unnavigable, and navigability is the whole value.
- "If it is automated we do not need the runbook." You need it more — the operator now meets the failure only when the automation did not handle it, which is the hardest version.
- "Runbooks make on-call mechanical." They make the known parts fast so attention is available for the unknown parts, which is where it is needed.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- An engineer who has never operated this service can resolve a common failure using only the runbook.
- Alerts link to the runbook entry for their symptom, and the link works.
- Runbook entries are edited after incidents — the edit history is the proof they are used.
- A game day or drill exercised the runbook and produced corrections.
- Time to mitigate for documented failure modes is shorter than for undocumented ones, and you can see that in the incident records.
- If a runbook step turns out to be wrong or harmful, correct it during the incident, in the document, not afterwards. The correction costs a minute then and is usually never made later.
- If a runbook has grown to the point where nobody can navigate it under pressure, cut it back to the three or four symptoms that actually occur. Deleting stale entries is maintenance, not loss.
- Automate the mitigations that are unconditional and well understood — draining a bad instance, restarting a stuck consumer with a bounded retry, scaling out a queue consumer. If the answer is always the same regardless of context, a human reading it is toil (Toil).
- Automate the checks into a single command or dashboard so the operator gathers state in one step instead of eight.
- Do not automate mitigations that depend on judgement: whether to roll back or forward, whether to fail over a region, whether to shed load and from whom.
- Do not automate a response you have seen once. A rule derived from one observation will fire on the second, different failure that shares its symptom (The Automation Trap).
- Runbooks cost continuous maintenance, and the maintenance is invisible until the day it was skipped. A wrong runbook is worse than no runbook, because it is trusted.
- Detail helps a stranger and slows an expert. Structure — symptom first, mitigation early, background last — is what serves both, and it takes more effort to write than prose.
- Automating a mitigation removes toil and removes the operator's exposure to how the system fails. After a year of automation, nobody on the rotation has seen the manual path.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe six-part structure holds across platforms and organisations. What varies is the checks: the same "consumer lagging" entry is a
kubectlcommand in one environment, a managed-service console panel in another, and an SSH session in a third. - ORG-SPECIFICWhere runbooks live and how formally they are reviewed is a convention. Keeping them in the service repository means they are reviewed with the code that invalidates them, which is the strongest available defence against decay — but only where the on-call engineer can reach the repository quickly.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — game days as the mechanism that turns a runbook from a hypothesis into a tested procedure.