Runbook Anti-Patterns
"Restart the service" is ritual. "Restart only after checking X, because Y indicates Z" is understanding — and the difference decides what happens when reality does not match the entry.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why is a runbook of unconditional commands worse than no runbook at all?
Runbooks accumulate steps that worked once. Stripped of their reasons, they become rituals: performed correctly in situations where they do not apply, by an operator with no way to tell the difference.
Write down exactly what to do so the operator does not have to think. Fewer decisions at 3am means fewer mistakes.
The situation is never exactly the one the step was written for. Without the reason, the operator cannot tell whether this is that situation, and the default under pressure is to follow the instruction.
- The situation is never exactly the one the step was written for. Without the reason, the operator cannot tell whether this is that situation, and the default under pressure is to follow the instruction.
- "Restart the service" applied to a service that is healthy but blocked on a slow dependency does nothing useful and drops in-flight work. Applied to a crash-looping bad release, it hides the release as the cause and delays the rollback (Rollback: Only Useful If It Is Actually Safe).
- Ritual steps mask causes. Restarting resolves the symptom for twenty minutes, so the underlying memory leak is never diagnosed and the incident recurs on a cycle (Leak or Unbounded Cache? The Question That Picks the Fix).
- A step nobody understands cannot be safely removed, so runbooks only grow. Eventually the document is too long to navigate at the moment it is needed.
- Instruction-only runbooks train operators not to reason. After a year, the rotation can execute the documented path and cannot handle anything else — which is precisely the case where a human was needed.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A runbook step has two parts: an action and the condition under which it is correct. Anti-patterns are what you get when the condition is dropped, which happens naturally because at the time of writing the condition was obvious to the author.
- The general form of a good step is "if X, do Y, because Z" — and Z is the load-bearing part. Z is what lets an operator recognise that they are not in situation X and act appropriately anyway.
- Restarting is the canonical example because it genuinely works often. It clears leaked memory, resets a stuck connection pool, discards corrupt in-process state. It is a mitigation with a real mechanism — and the mechanism tells you exactly when it will not help: when the process is healthy and the problem is elsewhere (Connection Pool Saturation: Waiting in Front of an Idle Database).
- The other high-cost anti-pattern is a step that destroys evidence. Purging a queue, deleting a pod without capturing logs, clearing a cache before reading its state — the incident ends and the postmortem has nothing to work with (Postmortems).
- A step performed identically every time, with no condition, is not a runbook entry. It is an unimplemented automation, and leaving it in a document means paying a human to be a slow, error-prone script (Toil).
The canonical example
Restarting is the most common runbook step and the most commonly misapplied one, because it works often enough to feel unconditional. The two versions below cost the same to execute. They differ entirely in what happens when the operator is not in the situation the step assumed.
If the checkout service is erroring, restart it: kubectl rollout restart deployment/checkout-api If that does not help, escalate.
Restarting clears leaked memory and resets stuck connection pools. It helps only if the process itself is in a bad state. Check first: kubectl get pods -l app=checkout-api - `OOMKilled` or rising restart counts -> the process is degrading. Restart, and open a ticket for the leak; a restart that is needed weekly is a masked bug, not a fix. - Pods healthy, errors from one dependency -> the process is fine and blocked. Do NOT restart: it drops in-flight requests and does not touch the cause. Go to "dependency degraded". - Restart loop that began within minutes of a rollout -> this is a bad release. Roll back rather than restart; restarting reproduces the crash and hides the deploy as the cause. Cost of restarting: in-flight requests are dropped. Under load that is a visible error spike of a few seconds.
The second version is longer and it is the only one that survives contact with a situation nobody anticipated. It also encodes three real facts about the system — that restarts help process state and not dependency state, that a weekly restart is a masked bug, and that restart loops after a rollout mean the rollout — which the first version silently discards.
The anti-pattern catalogue
Each of these begins as a step that worked. The pattern is always the same: the condition was obvious at the time of writing, so it went unwritten, and what remains is an action with no test attached to it.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| "Restart the service" | Restart performed on a healthy, blocked process | Action recorded without the state it addresses | State what restarting fixes and what it costs; list the cases where it will not help |
| "Clear the cache" | Thundering herd on the origin; a second, larger incident | The load consequence of a cold cache was never written down | Say what a cold cache costs and how to warm it, or invalidate selectively (Cache Stampede: Everyone Misses at Once) |
| "Scale up" | Connection pool exhausted; the database becomes the outage | Scaling treated as free when the real constraint is downstream | Name what saturates first and the ceiling for each (The Connection Budget) |
| "Purge the queue" | Valid messages discarded; manual reconciliation for days | A destructive step written for a poison-message case, applied to a backlog | Target the specific message; never make purge the general instruction (Dead Letter Queues Are an Operation) |
| "Delete the pod" | Symptom clears; postmortem has no evidence | Mitigation destroys the state needed to diagnose | Capture logs and state first; it costs seconds (Reading a Broken Workload) |
| "Run this script" | Nobody knows what it does; it fails on a variant | A step delegated to an opaque artifact | Document what it does and its preconditions, or make it fail loudly outside them |
| "Wait 10 minutes and check again" | Ten minutes of impact with nothing learned | A delay observed once and encoded without its mechanism | Say what is expected to happen in those minutes and what to watch |
| "Escalate to the team" | Escalation late or to the wrong place | No threshold, no rotation, no handover content | Give a trigger, a named rotation and the information to bring (Telling People What Is Happening) |
What to do with a step you cannot justify
The productive move when you find an unexplainable step is not to delete it reflexively and not to keep it out of superstition. It is to work out which of four categories it falls into, because each has a different correct response.
You are reviewing a runbook and find a step nobody on the team can explain. What do you do with it?
when Someone in the organisation knows, or an old incident record explains it.
cost Archaeology time — often an hour in incident history, and sometimes the reason has expired anyway.
when It is performed identically every time with no decision in it.
cost Automation you must maintain, and the operator loses exposure to the manual path (The Automation Trap).
when Nobody can justify it and it has a plausible cost — it destroys evidence, drops work, or takes time.
cost A small chance you delete something load-bearing; you find out during the next incident, with the reason now known.
when It is cheap, harmless, and you are not confident enough to remove it today.
cost The runbook grows and the marker is honest but easy to ignore; set a date to revisit rather than leaving it forever.
How to do it properly
Most important first.
- Write every step as condition, action, reason. If you cannot state the reason, you do not yet understand the step and neither will the person following it.
- State what each mitigation costs: dropped in-flight requests, redelivered messages, a cold cache, lost diagnostic state.
- Say explicitly when not to act. "Do not restart if the process is healthy and the dependency is slow — you will drop work and it will not help" is one of the most valuable sentences a runbook can contain.
- Preserve evidence before mitigating where it is cheap: capture logs before deleting the pod, snapshot the queue depth before draining, record the version and config before rolling back.
- Delete steps you cannot justify. If nobody can say why a step exists, it is either automatable or unnecessary, and both conclusions beat leaving it.
- Automate unconditional steps rather than documenting them. A step with no decision in it belongs in code (How to Automate Something).
- Rewrite an entry every time an incident shows it was wrong, while the incident is still fresh enough that the reason is known.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained to the service, but with a specific extra cost: ritual mitigations mask causes, so the same incident recurs indefinitely on a schedule nobody investigates.
What can go wrong
- Reasons written as restatements: "restart the service because the service needs restarting" satisfies the format and teaches nothing.
- Over-correction into essays — a page of theory per step, which is unreadable under pressure and gets skipped entirely.
- Conditions stated in terms the operator cannot evaluate: "if the service is unhealthy" with no way to determine health.
- A "do not do this" warning with no alternative, leaving an operator who is watching impact grow with nothing to do.
- Automating the ritual instead of removing it — an auto-restart loop that masks a leak permanently and removes the last signal that it exists.
- Escalation used as the universal alternative, so every non-obvious situation becomes someone else's page and the rotation never develops judgement.
- "Never restart a service." Restarting is often exactly right. The anti-pattern is restarting without checking whether the condition it addresses is present.
- "Runbooks should not contain commands." They should contain commands and the conditions for running them. Commands alone are the problem, not commands.
- "If we explain everything, on-call gets slower." The explanation belongs after the action, not before it. Operators read the action, act, and read the reason when they need to deviate.
- "Automating the restart solves this." It solves the toil and can entrench the ritual. Automate the action, keep the signal, and fix the cause (Toil).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Every step in the runbook has a stated condition and reason, and a reviewer could challenge either.
- Operators have deviated from the runbook correctly, because it gave them enough understanding to recognise a variant.
- Repeat incidents from masked causes have fallen — the restart-every-Tuesday pattern is gone because the leak was fixed.
- Steps have been deleted from runbooks in the last year, not only added.
- If a rewritten runbook turns out to be harder to use under pressure than the terse version, that is real feedback: keep the conditions and reasons but move the background out of the action path. Structure, not deletion, is the fix.
- If a step was removed and an incident then needed it, restore it with the reason it was missing — that reason is now known, which is what the removal bought.
- Automate the unconditional. If a step is always correct in the situation the alert describes, the alert should trigger the action, not a human (How to Automate Something).
- Automate evidence capture before mitigation — logs, state and versions collected automatically so the fast path does not destroy the postmortem.
- Do not automate a step whose condition requires judgement. An auto-restart that cannot distinguish "crash-looping" from "healthy but blocked" will make the second case worse (The Automation Trap).
- Do not let automated mitigation hide the underlying signal. An auto-restart on memory pressure must still surface that it is happening, or the leak becomes permanent and invisible.
- Reasons make entries longer, and length is a real cost in a document read under pressure. Good structure — condition, action, reason, in that order — is the only way to have both, and it is harder to write.
- Understanding-based runbooks demand more of the operator. On a very large rotation with wide variation in experience, some entries genuinely need to be prescriptive; those should be the ones that are automated.
- Deleting unjustifiable steps occasionally deletes something that mattered for a reason nobody remembered. That risk is real and smaller than the cost of an unnavigable document.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe condition-action-reason structure is platform-independent. What varies is which steps are automatable: a platform with declarative reconciliation removes some manual steps entirely, while a hand-managed VM fleet keeps them.
- ORG-SPECIFICHow much judgement a runbook may leave to the operator depends on the rotation. A small expert rotation can carry terse entries with reasons; a large one spanning many services needs more prescription, and the prescriptive entries are the automation backlog.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — rehearsing a mitigation as the only reliable way to discover that its stated condition is wrong.