IncidentsORG-SPECIFICSCALE-SPECIFIC
Severity: What It Should Reflect
A shared shorthand for how much of the organisation to wake — and a local convention, not a fact about software.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The production question
What makes one incident more severe than another, and who decides?
The problem
Severity drives response: who is paged, how fast, how much of the business stops. Without a shared definition it is set by whoever is loudest or most anxious, and the response is miscalibrated in both directions.
What teams do first
Copy a severity scale from a well-known engineering blog. SEV1 is total outage, SEV2 is major degradation, SEV3 is minor. Done.
How it breaks
The copied scale describes someone else's business. "Total outage" for a consumer app and for a payments processor imply completely different responses, and neither definition transfers.
How it breaks in production
- The copied scale describes someone else's business. "Total outage" for a consumer app and for a payments processor imply completely different responses, and neither definition transfers.
- Scales defined purely by technical symptoms miss the incidents that matter most: a small number of users, one enterprise customer, or a data correctness problem affecting nobody visibly yet (Telling People What Is Happening).
- Without explicit criteria, severity inflates. Everything becomes high severity because that is how you get attention, and then high severity stops meaning anything.
- Or it deflates: nobody wants to escalate at 2am, so a genuine outage is handled at low severity by one tired person until morning (Rotations People Can Sustain).
- A scale with no guidance on data and security impact routes a breach into the same channel as a slow endpoint (What Happens Between the Page and the Postmortem).
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Severity is a routing decision compressed into a label. Its only job is to answer: how many people, how fast, and how much else stops.
- What it should reflect is a small set of dimensions, and the numbering is just a name for a point in that space. Judged honestly, severity is about user impact (how many, how badly, which journey), business impact (revenue, contractual commitments, regulatory obligations), security and data impact (exposure, loss, corruption — often the highest severity even at zero visible user impact), scope (one tenant, one region, everyone), duration and trend (short and recovering, or long and growing), and reversibility (whether waiting makes recovery harder).
- Two of those routinely dominate and are routinely underweighted. Trend matters because a small-but-growing incident deserves the response its projected size warrants. Reversibility matters because data loss compounds: every minute of a corrupting write is a minute of extra recovery work (Partial and Logical Data Recovery).
- Severity is provisional. It is assigned on partial information at the start, and re-assessed as scope becomes clear — raising it is not an admission that the first call was wrong.
- The number of levels is a design trade-off. Too few and everything collapses into one bucket; too many and nobody remembers the boundaries. What matters is that each level maps to a distinct response, not that there is a canonical count.
The dimensions severity should reflect
ORG-SPECIFICThe weighting between these is a business decision. A B2B platform with a handful of large customers weights scope by customer rather than by user count; a consumer product does the reverse. Both are correct for their context and neither generalises.
These are the inputs. Your organisation's scale is a function over them, and writing the function down is most of the work — the numbering is the easy part.
The third column is where most scales are weakest: they describe symptoms and forget the dimension entirely.
| Dimension | The question | Commonly missed because |
|---|---|---|
| User impact | How many users, and how badly is their journey broken? | Aggregates hide a scoped failure — one tenant, one region, one client version |
| Business impact | Revenue, contractual commitments, regulatory deadlines? | The responder does not have this context at 3am and nobody wrote it down |
| Security / data impact | Exposure, loss or corruption of data? | It often presents with no user-visible symptom at all |
| Scope | One customer, one region, or everyone? | The dashboard cannot answer it, so it is guessed (Dashboards an Operator Can Act On) |
| Duration and trend | Recovering, steady, or growing? | Severity is set on the first reading and never revisited |
| Reversibility | Does waiting make recovery harder or impossible? | Corruption and data loss compound silently (Partial and Logical Data Recovery) |
| Workaround | Can users or support route around it? | A viable workaround genuinely lowers impact and is rarely accounted for |
A scale, presented as one example
This is an illustration of the *shape* a working scale has — each level attached to a response rather than to an adjective. It is not a recommendation, and the level names below are a convention this example chose, not a standard.
Build yours from your own past incidents. The most useful column is the last one, because a concrete example from your history settles arguments that a definition cannot.
| Level | Impact it reflects | Response it obliges | Anchor example (yours will differ) |
|---|---|---|---|
| Highest | Core journey unusable for most users, or confirmed data loss or exposure | Page immediately, roles assigned, stakeholder comms, other work stops | Checkout returning errors for all users |
| High | Core journey degraded, or a major customer fully blocked, or trend projects to the level above | Page immediately, responder plus one, regular updates | Payments failing for one region; error rate doubling every ten minutes |
| Medium | Secondary function broken, or a workaround exists | Handled in hours, in working time, informed rather than paged | Report exports failing; users can still transact |
| Low | Degradation with no user-visible effect yet | Ticket with an owner and a date | Disk at 80% on a replica; retries absorbing a flaky dependency |
How severity gets set wrongly
Both directions are common, and they have different causes. Inflation is usually a signalling problem; deflation is usually a cultural one, and the cultural one is more dangerous because it is quiet.
The response column is deliberately about changing the system rather than asking people to judge better.
Miscalibration, and what actually fixes it
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Attention is scarce | Everything is declared at the top level | High severity is the only reliable way to get help | Make lower levels actually deliver help; inflation is a rational response to a level that does nothing |
| High severity is discussed in performance reviews | Nobody declares anything | Declaring is perceived as a career risk | Leadership behaviour, not a scale change — praise early declaration explicitly and visibly |
| Incident starts small at 2am | One person handles a growing outage alone for hours | Reluctance to wake colleagues | Make escalation the default and time-bound it; nobody should have to decide whether they are allowed to ask for help |
| Data corruption with no error rate change | Treated as a low-severity bug for days | The scale is written entirely in terms of availability | Give data and security impact their own criteria at the top of the scale |
| Cross-team incident | Two teams respond at different intensities | The same number means different things to each | One organisation-wide scale, or explicit translation at the boundary |
| Impact clarifies mid-incident | Severity stays where it was set | No step in the process asks the question again | Re-assess at each status update; raising is routine, not a correction |
How to do it properly
Most important first.
- Define the levels in terms of impact your organisation actually cares about, with concrete examples from your own past incidents. Examples do far more work than definitions.
- Attach a response to each level, not a description: who is paged, who is informed, on what cadence, and whether normal work stops. If two levels imply the same response, you have one level.
- Make severity easy to raise and require no permission to do so. The cost of over-escalating is an hour of some people's time; the cost of under-escalating is the incident.
- Assign on projected impact, not only current impact. Something affecting 1% and doubling every ten minutes is not a small incident.
- Give data and security impact their own path — corruption, loss or exposure often warrant the highest response even when nothing is visibly broken, and Security Engineering owns the handling and disclosure obligations that follow.
- Set an explicit default. When it is genuinely unclear, start higher and downgrade, because downgrading is cheap and catching up is not.
- Review severity assignment in the postmortem: was it right at the time, given what was known? That is the question, not whether it was right in hindsight (Postmortems).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Blast radius if this is wrongEveryone
One testEveryone
What contains it
Nothing contains a mis-set severity — it under-resources the response to an incident that is already happening.
What can go wrong
Failure modes, including of the mitigation
- Severity used as a performance signal, so teams avoid declaring high severity because it looks bad. This is the most damaging failure and it is created by leadership behaviour, not by the scale.
- Severity inflation, which trains everyone to discount the label.
- Levels defined only by technical symptoms, so a correctness bug affecting one large customer has no place on the scale.
- A slow-burn incident never assigned any severity because it never crossed a threshold sharply.
- Severity assigned once and never revisited as the picture changes.
- Different teams using the same numbers with different meanings, which surfaces during a cross-team incident at the worst possible moment.
Misreads this invites
- "SEV1 means the site is down." It means whatever your organisation defined it to mean. There is no industry standard, and assuming a shared meaning across companies — or across teams — is a real source of confusion.
- "Severity measures how hard the incident is to fix." It measures impact. A trivial fix for a total outage is maximum severity.
- "Low user count means low severity." One enterprise customer, one regulator, or one corrupted table can outweigh a percentage.
- "Declaring high severity is escalating unnecessarily." Declaring is how you get help. Treating it as a career risk is the organisational failure that most reliably lengthens incidents.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
How you know it worked
- Two engineers given the same incident description assign the same severity without discussion.
- The response that actually happened matched what the level specifies.
- Severity has been raised mid-incident recently, without anyone treating it as a mistake.
- The scale references your own past incidents as examples.
How you get back
- Downgrading mid-incident is normal and should be announced explicitly, with what changed — otherwise responders drift away without knowing whether it is over.
- If a severity scale is not working — everything is one level, or nobody uses it — change the scale rather than exhorting people to use it correctly. It is a piece of the system, and it can be defective.
What to automate, and what stays human
- Automate the consequences: page the right rotation, open the channel, notify the stakeholder list, start the status page draft.
- Automate a suggested severity from the signals — how many users, which journey, which trend — as a starting point that a human confirms or overrides.
- Do not automate the assignment itself. Severity depends on business context — which customer, which contractual window, which regulator — that the monitoring system does not have.
What this costs
- Precise criteria are unambiguous and rigid; judgement-based criteria adapt and vary between people. Most teams end up with criteria plus an explicit override, which is the honest compromise.
- A scale everyone understands takes real effort to build and maintain, and it is invisible work.
- Defaulting high costs people's time on incidents that turn out to be minor. That cost is usually much smaller than it feels.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- ORG-SPECIFICThis is the spec's explicit requirement (§75) and it is not a formality. What varies: the number of levels, their names or numbers, which direction counts as most severe, the thresholds, who may declare, and what each level obliges. A payments company and a media site with identical error rates are in different severities, because the business impact and regulatory obligations differ. What does not vary is that severity should reflect user impact, business impact, security and data impact, scope, duration and reversibility — the dimensions are portable, the scale is not. Never assume another team's SEV2 means what yours does.
- SCALE-SPECIFICA single team with one service usually needs two levels — wake people, or handle it tomorrow. Multi-level scales earn their complexity once the response involves people who are not on the responding team, because the level is then how you tell them what is expected.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
Backendmulti-tenancy
Domains that do not exist yet
- — Testing & Reliability Engineering — connecting severity to error budget policy, so what counts as severe follows from what was promised.