Headroom
Why critical systems are never run at their limit, and the four separate claims on the reserve you hold.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why not run production at full utilisation, given that unused capacity is money spent on nothing?
Utilisation looks like efficiency, so there is constant, reasonable pressure to raise it — and every point of utilisation you gain is reserve you no longer have when something goes wrong.
Run the fleet near its limit. Idle capacity is waste, the graphs look responsible, and if traffic rises we will scale out.
Traffic is not smooth. A fleet sized for the average is saturated by the peaks inside the average — and the peaks that matter last seconds, far shorter than any scaling reaction (Autoscaling).
- Traffic is not smooth. A fleet sized for the average is saturated by the peaks inside the average — and the peaks that matter last seconds, far shorter than any scaling reaction (Autoscaling).
- Instances and nodes fail. When one goes, its share of traffic lands on the rest instantly; if the rest were near their limit, the failure of one becomes the failure of all.
- Deployments remove capacity on purpose. A rolling deploy takes instances out of service to replace them, so the fleet must serve full load while some of it is missing (Rolling: Two Versions, One Database).
- Degraded dependencies make every request more expensive. Retries, longer waits and held connections raise per-request cost without any change in request rate — the same traffic now needs more fleet.
- Near the ceiling, latency rises steeply for reasons that have nothing to do with your code. You are buying tail latency, not just risk (Capacity Management).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Headroom is not slack. It is capacity with named claims on it, and the claims are simultaneous rather than alternative — a node can fail during a deployment while a dependency is slow.
- Claim one, burst: the gap between the traffic you average and the traffic you actually receive in the busiest short interval. It is absorbed instantly or not at all, because no scaling mechanism reacts in seconds.
- Claim two, failure: whatever capacity is lost when a unit of failure is lost. If a fleet is spread over N failure domains and traffic is even, losing one leaves (N-1)/N of the fleet serving all of the load. That is a relationship, not a measurement.
- Claim three, deployment: a rolling deploy of k instances at a time out of N leaves (N-k)/N serving, for the duration of the rollout, plus the warm-up period of each replacement.
- Claim four, degradation: when a dependency slows, in-flight requests hold resources longer. Concurrency equals arrival rate times service time, so a doubled service time doubles the in-flight work at unchanged traffic — Performance owns this as Little's Law.
- Because the claims can coincide, headroom is sized against the worst plausible combination rather than the largest single claim.
Four claims on the same reserve
Every argument to reduce headroom comes from treating it as one undifferentiated buffer. Breaking it into claims makes the conversation concrete: which claim are we giving up, and what happens the next time it is called?
The right-hand column is the test. If a claim has no mechanism that could consume it in your architecture, you genuinely do not need to hold it.
| Claim | What consumes it | Timescale | What removes the need for it |
|---|---|---|---|
| Burst | Short-interval traffic peaks above the average | Seconds | A queue that can absorb it, or shedding at the edge |
| Failure | An instance, node or zone going away | Instant, then minutes to replace | More, smaller failure domains — losing 1 of many costs less |
| Deployment | Instances out of service during a rollout, plus warm-up | Minutes, on your schedule | Surge capacity: add before removing (Rolling: Two Versions, One Database) |
| Degradation | Longer service time holding resources per request | Minutes to hours | Aggressive timeouts and circuit breaking, which convert it into errors |
The arithmetic of losing a unit
Two of the four claims are pure arithmetic on stated assumptions, which makes them the easiest part of the policy to defend. Assume traffic splits evenly across identical units and nothing else is claiming reserve at that moment.
Read the last column as the requirement it is: to survive that loss without saturating, each remaining unit must be able to carry that share. This is where the two-region case becomes uncomfortable, and it gets a lesson of its own (Capacity During Failover).
| Situation | Units serving | Load per surviving unit | What that requires |
|---|---|---|---|
| Steady state, N units | N | 1/N of total | Each sized for its even share |
| One unit lost, N units | N-1 | 1/(N-1) of total | Reserve of 1/(N-1) - 1/N per unit |
| Rolling deploy, k of N cycling | N-k | 1/(N-k) of total | Or surge: run N+k briefly instead |
| One lost during a deploy | N-k-1 | 1/(N-k-1) of total | The combination the policy has to survive |
| Service time doubles | N | Same rate, twice the in-flight work | Twice the concurrency capacity, or timeouts |
Choosing a policy without pretending there is a right number
There is no universal headroom figure, and any lesson that gives you one is selling a number from someone else's architecture. What is general is the set of inputs, and the fact that the decision is a purchase.
A service is at its limit more often than you would like. What sets its headroom?
when The service is on the critical user path and degrades badly — errors rather than delay.
cost The most expensive option, and the reserve is idle almost always (Idle Capacity).
when Failure domains are numerous and deploys use surge capacity, so claims are unlikely to coincide.
cost A coincidence you decided was unlikely is now an incident when it happens.
when The workload has a natural low-value tier that can be dropped first.
cost Requires a working shedding path and a decision about who gets dropped (Load Shedding).
when Work is asynchronous and users tolerate delay rather than failure.
cost Latency becomes the release valve; queue age becomes the thing you must watch (Operating Queues and Scheduled Work).
when Load ramps predictably over tens of minutes and the constraint is per-instance.
cost Nothing covers the seconds-scale claims — bursts and node loss (How Autoscaling Fails).
How to do it properly
Most important first.
- Write the headroom policy down as claims with reasons, not as a single percentage handed down by tradition. A number without a claim cannot be defended when someone asks to reduce it.
- Size the failure claim from your actual failure domain granularity — instance, node, zone — rather than from a habit (Capacity During Failover).
- Cut the deployment claim by deploying with surge capacity: add the new instances before removing the old ones, so the fleet never dips below full (Rolling: Two Versions, One Database).
- Differentiate by criticality. The user-facing path and the nightly batch job do not need the same reserve, and pretending they do makes the policy expensive enough to be ignored.
- Measure real burstiness at a short interval. Per-minute averages hide the peaks that headroom exists for.
- Pair headroom with a shedding plan, because reserve is finite and the plan for exhausting it should not be improvised (Load Shedding).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing contains an exhausted reserve except shedding. Containment has to be built before it is needed, not during.
What can go wrong
- Headroom that exists on paper and is consumed by gradual traffic growth, because nobody re-checks the actual margin.
- Reserve held per-instance while the real constraint is shared — plenty of CPU headroom and no spare database connections (The Connection Budget).
- Autoscaling used as a substitute for headroom, so the reserve is "we will scale", which fails on exactly the timescale bursts and node failures occur on.
- A policy so expensive it is quietly abandoned service by service, leaving an inconsistent estate nobody can reason about.
- The mitigation failing: headroom sized for a single claim, then a node fails mid-deploy and both claims land at once.
- "Low utilisation means we are wasting money." Some of it is waste and some of it is the failure and deployment reserve. The distinction is whether the unused capacity has a named claim (Idle Capacity).
- "Autoscaling gives us headroom." Autoscaling gives capacity minutes later. Bursts and node failures happen in seconds (How Autoscaling Fails).
- "We hold 30% headroom, so we can absorb losing 30% of the fleet." Only if nothing else is claiming it at the same time — and during an incident, everything is.
- "Headroom is a platform-level default." The right reserve depends on failure domain count, deploy strategy and how the service degrades. It is a per-service decision with a platform default as a starting point.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Current utilisation of the binding constraint plotted against the policy target, per service.
- A node or instance was lost recently and no user-visible impact followed — the strongest evidence there is.
- Deploys show no latency change at the p99, which means the fleet genuinely carried full load while instances were cycling.
- A burst measurement at short resolution, recorded, so the burst claim has a basis.
- Reducing headroom is a change like any other and should be canaried: apply it to one service, watch a full traffic cycle including a deploy and a peak, then extend it.
- Reducing headroom is easy to reverse in a fleet you can scale and effectively irreversible in one you cannot — a downsized database instance may need a maintenance window to grow back.
- Automate the alert that says the margin has been eaten: utilisation of the binding constraint crossing the policy line is a ticket, not a page.
- Automate surge-capacity deployment so the deployment claim disappears without anyone remembering to think about it.
- Keep the policy human. How much reliability to buy is a business decision, and the person who signs for it should be the person who pays for it.
- Headroom is paid for continuously and used rarely. That is the deal, and it is the same deal as insurance — the objection "we never use it" is a statement that it worked.
- Reserve on the critical path competes directly with reserve everywhere else. Applying one number everywhere spends the budget on the services that need it least.
- High headroom hides inefficiency: a service that got twice as expensive per request will not show up as an incident, only as a shrinking margin nobody watches (Overprovisioning).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe four claims exist on any platform. Their sizes differ sharply: a serverless platform absorbs the burst and deployment claims for you and charges for it, while a fixed fleet of VMs makes both claims yours to hold.
- SIMPLIFIEDThe fractions here assume traffic splits evenly across identical units. Real fleets have uneven instances, sticky sessions and hot shards, all of which make the surviving units carry more than the even share.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — why failure domains are the unit that matters, and how correlated failure defeats the even-split assumption.