Roles During an Incident
Separating coordination from investigation so that neither starves the other — valuable at high severity, overhead at low.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Who is in charge during an incident, and when is having someone in charge worth the overhead?
Beyond two or three responders, coordination becomes a full-time job, and if nobody holds it the person debugging holds it — badly, and at the cost of the debugging.
The most senior engineer present takes charge and also does the investigating. They know the system best, so they should be doing both.
Coordination and investigation both demand full attention and interrupt each other constantly. The person switching between them does neither well.
- Coordination and investigation both demand full attention and interrupt each other constantly. The person switching between them does neither well.
- The best debugger is not necessarily the best coordinator, and the two skills are unrelated.
- With no explicit owner, decisions are made by whoever speaks last, and two people can take contradictory actions minutes apart.
- Stakeholders interrupt the person doing the work, because that is who they can find (Telling People What Is Happening).
- Nobody is tracking the elapsed clock, so an investigation runs for forty minutes without anyone asking whether it is time to mitigate regardless (Stop the Harm Before You Understand It).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The core idea is separating deciding from doing. One person holds the state of the incident — what is known, what is being tried, what has been ruled out, who is doing what — and makes or brokers decisions. Others work on the problem.
- The coordinator's value is not technical authority. It is holding context that nobody deep in a debugging session can hold, and asking the questions that get lost: how bad is it now, have we mitigated, who is not being used, when do we escalate.
- A commonly separated second role is communication: talking to stakeholders, support and customers on a cadence, so the responders are not interrupted. At high severity that is genuinely a full-time job.
- A third is scribe: recording actions and times as they happen, which makes the postmortem timeline evidence rather than recollection (Reconstructing What Actually Happened).
- The roles are positions, not people. Whoever holds one holds it explicitly, hands it over explicitly, and the handover is announced. A coordinator who is also the only person who can fix it should hand coordination to someone less technical rather than keep both.
- The overhead is real. Assigning roles takes minutes and adds ceremony, which is why for a single responder on a small incident it is pure cost.
What each role holds
Read this as a description of jobs, not of job titles. On a small incident one person holds all of them; on a large one, separating them is what keeps the investigation moving.
The last column is the diagnostic. If you see that symptom, the role is not being held — whatever the org chart says.
| Role | Holds | Decides | Symptom that it is unheld |
|---|---|---|---|
| Coordinator | The whole picture: what is known, tried, ruled out, in flight | Mitigate, escalate, severity, recovery declared | Two people take contradictory actions |
| Investigator(s) | The technical problem | What to try next, technically | Everyone is coordinating and nobody is debugging |
| Communicator | The audience: stakeholders, support, status page | What to say and when | The responder is answering direct messages |
| Scribe | The record: actions, times, observations | Nothing — records rather than decides | The timeline is written from memory afterwards |
| Subject expert | Deep knowledge of one component | Advice, not direction | Someone is guessing about a system that has an owner |
When roles pay for themselves
This is a judgement about incident size, and getting it wrong in either direction has a cost: ceremony on small incidents trains people to skip it, and no structure on large ones means the coordination happens badly inside someone's head.
The triggers below are more useful than severity levels, because they describe the thing that actually creates coordination load: the number of people who need to be kept in sync.
How much coordination load is there, right now?
when One responder, one system, no external audience.
cost None — and saying "no roles needed" explicitly is what stops someone assuming otherwise.
when Two or more responders, or stakeholders have begun asking for updates.
cost One person stops investigating; usually repaid immediately in avoided duplicate work.
when External or customer-visible impact, or leadership is watching.
cost Two people not investigating; correct when the alternative is the responder writing updates.
when High severity, multiple teams, or an incident that will certainly be reviewed in detail.
cost Significant headcount, justified by the coordination load and by needing an accurate record (Reconstructing What Actually Happened).
when The decision needed is a business one — take the feature down, notify customers, invoke a contractual process.
cost They may lack technical context, so pair them with an engineer rather than substituting.
The handover
Long incidents outlast people. A handover done badly costs more than the fatigue it was meant to relieve, because the incoming coordinator re-runs eliminated hypotheses and the outgoing one leaves believing they transferred more than they did.
A structured handover takes two minutes and is worth rehearsing. The point of writing it down is that a tired person hands over worse than they think they do.
- 1State the impact
What is broken, for whom, since when, and the current trend.
fails by Describing the investigation instead of the impact.
evidence The incoming coordinator can repeat it back in one sentence.
- 2State what is ruled out
Hypotheses eliminated and how, so they are not re-run.
fails by Only the current hypothesis is transferred, so the dead ends get revisited.
evidence A written list in the channel, not a verbal summary.
- 3State what is in flight
Who is doing what right now, and what is expected from it.
fails by Someone's work becomes invisible and is duplicated.
evidence Every responder is named against a task.
- 4State the mitigations tried
What was applied, when, and what the symptom did afterwards.
fails by A mitigation is applied twice, or an ineffective one is left in place.
evidence Times recorded, with the symptom response (Deploys on the Same Timeline as the Symptom).
- 5Transfer explicitly
Announce the change in the channel by name.
fails by Everyone still asks the previous coordinator.
evidence A message in the channel: "X is now coordinating."
- 6Leave
The outgoing coordinator stops, rather than hovering.
fails by Two coordinators, which is functionally the same as none.
evidence They are not answering questions ten minutes later.
The same structure works for handing over investigation, and for the end-of-shift handover of an ongoing incident.
How to do it properly
Most important first.
- Define the roles in advance, with a sentence each, so nobody is inventing them mid-incident.
- Assign the coordinator explicitly and say it out loud in the channel: "I am coordinating." Ambiguity here is the failure mode the role exists to remove.
- Separate coordination from investigation the moment there is more than one responder, or the moment stakeholders start asking for updates.
- Give the coordinator explicit authority to decide mitigation, escalation and severity, so decisions do not wait for consensus.
- Hand over deliberately — role, current state, what has been ruled out, what is in flight — and announce the handover (Rotations People Can Sustain).
- Rotate who coordinates. It is a learnable skill and a rotation with one capable coordinator has a single point of failure (Shared Ownership).
- Scale the roles to the incident. One responder, one small problem: no roles, and say so.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Coordination is what contains an incident that involves more people than one; its absence shows up as duplicated and contradictory action.
What can go wrong
- The coordinator starts debugging and stops coordinating, which is the commonest failure and happens because the problem is more interesting than the coordination.
- Coordination assumed rather than assigned — everyone believes someone else has it.
- A coordinator with responsibility and no authority, who can only ask people to agree.
- Roles applied ceremonially to trivial incidents until people route around the process entirely.
- The coordinator becomes a bottleneck for information they do not have, slowing the responders down.
- Handover at the end of a shift with no state transfer, so the incoming coordinator re-runs work that was already ruled out.
- Seniority conflated with the role: a senior person overrides the coordinator informally and the incident has two decision-makers.
- "The incident commander is the most senior engineer." It is whoever is holding coordination, which is frequently not the most senior person and is better when it is not the person who knows the system best.
- "The commander decides the technical fix." They decide *whether* and *when* — mitigate now, escalate, declare recovery. The technical judgement stays with the people investigating.
- "Every incident needs a commander." Roles that cost more than they return at low severity get abandoned, and then they are also absent at high severity.
- "This is copied from emergency services, so it applies as written." It is an adaptation. The parts that transfer are separating coordination from execution and making the role explicit; the ceremony around it is optional and often too heavy for software teams.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- At any moment during the last significant incident, everyone could name who was coordinating.
- The responders were not the ones writing stakeholder updates.
- The postmortem timeline was assembled from a scribe's log rather than reconstructed afterwards.
- Small incidents ran without any of this and nobody suggested otherwise.
- Roles can be collapsed mid-incident as it shrinks: "we are down to one responder, dropping the coordinator role" is a normal announcement and stops the ceremony outliving its usefulness.
- If the coordinator is not working out — too deep in the problem, too junior on the system, or exhausted — hand over. Handover is a routine move, not a judgement about the person.
- Automate the scaffolding: create the channel, invite the on-call, post a template with role slots, start the timestamped log.
- Automate reminders on a cadence — status update due, severity re-assessment due, "has anyone tried a mitigation" — which is exactly the kind of tracking a busy coordinator drops.
- Automate the collection of actions taken through tooling into the incident log, so the scribe records decisions rather than transcribing commands.
- Do not automate the coordination itself. Deciding what to try next and whether to escalate is judgement under uncertainty with incomplete information (The Automation Trap).
- Roles cost a person. At high severity that person is worth several responders; at low severity they are a responder you no longer have.
- A non-technical coordinator relays rather than reasons, which slows technical decisions; a technical one is tempted to start debugging. Both are real and there is no version without the trade.
- Practising the role requires incidents, so teams with few incidents will always be somewhat unpractised — which is a good problem with a real cost.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- ORG-SPECIFICThe spec requires this label (§78) and it matters. What varies: whether the roles exist at all, their names — incident commander, incident lead, on-call captain, duty manager — how many there are, whether the coordinator has decision authority or only convenes, whether a manager is required for high severity, and at what threshold roles are activated. What transfers is only the underlying idea: past a couple of responders, coordination is a distinct job, and someone should hold it explicitly. A three-person startup and a bank need different amounts of this, and the bank's version is not the mature form of the startup's.
- SCALE-SPECIFICBelow roughly three simultaneous responders the roles are overhead: the coordination cost is low enough that the responder can carry it. Above five or six, and especially across teams, coordination becomes the constraint and an unassigned role means nobody is tracking the whole.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — practising coordination during game days, since the role is the least-used skill in the module and the hardest to learn during a real outage.