Headroom: The Capacity You Deliberately Do Not Use
Running at 100% utilization means every burst, every failed instance and every deploy becomes an incident. How much headroom is a real decision with a real cost, and the number comes from burst shape, scale-up time and blast radius — not from a convention.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Why the last 20% of utilization is not usable
A queueing system does not degrade linearly. At 50% utilization a request mostly walks straight into a free server. At 90% it frequently waits behind other work, and at 98% the wait dominates the service time entirely. The relationship is a curve with a knee, and the knee is why the industry converged on utilization targets in the 60–80% range rather than 95% — it is not caution for its own sake, it is where latency stops being predictable (Queueing: Why Systems Get Slow Before They Get Broken, Saturation: The Reading Utilization Cannot Give You).
This means "we are at 85% CPU and everything is fine" and "we are at 85% CPU and one more instance failure doubles p99" are the same reading. Utilization alone does not tell you which situation you are in; the distance to the latency knee does, and only a load test locates the knee.
The second reason is that utilization is measured as an average over a window. A one-minute average of 70% can contain 15-second stretches at 100%, during which requests queue and time out. The shorter your bursts, the more headroom the averages are hiding from you.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| CPU utilization (5m avg) | 68% | Comfortable-looking, and the number most dashboards show | normal |
| CPU utilization (10s max, peak hour) | 94% | The real peak; the 5-minute average smoothed it away | suspect |
| p99 latency at that moment | 820 ms vs 180 ms objective | Past the knee: queueing, not computing | smoking gun |
| Instances | 6 of 6 healthy | No redundancy consumed yet — one failure adds ~17% load to each survivor | suspect |
N+1, N+2, and the failure you have to survive
Redundancy headroom is a different quantity from performance headroom, and both have to fit. N+1 means the fleet still serves peak load with one instance gone; N+2 means two. With six instances, losing one moves 20% more traffic onto each survivor — so a fleet at 80% utilization goes to 96%, which is past the knee. The redundancy target and the utilization target constrain each other.
Smaller fleets need proportionally more headroom, which is the counter-intuitive part. Losing one instance of three is a 50% load increase on the survivors; losing one of thirty is 3.4%. This is why very small fleets often run at what looks like wasteful utilization: they are not wasteful, they are paying for the granularity of their failure domain.
Deploys consume the same budget. A rolling deploy that takes 20% of the fleet out of rotation is an intentional, scheduled 20% capacity reduction — and if it happens during peak hour without headroom for it, the deploy causes the incident. So does a node draining, a zone evacuating, or an instance-type migration.
| Fleet size | Load added to each survivor | Utilization at 70% baseline | Verdict |
|---|---|---|---|
| 3 instances | +50% | 105% — cannot serve peak | N+1 impossible at 70%; needs ~45% baseline |
| 6 instances | +20% | 84% — past the knee for most services | Workable only if the knee is high |
| 12 instances | +9% | 76% — usually survivable | N+1 comfortable |
| 30 instances | +3.4% | 72% — barely noticed | N+2 affordable at this size |
What decides your number
There is no universal headroom percentage, and anyone quoting one without qualification is repeating a convention. Four things decide it, and they are all measurable in your system.
Burst shape: how fast can traffic rise, and for how long? A service whose traffic can triple in ten seconds needs enough standing headroom to absorb the spike, because nothing you can provision arrives that fast. A service whose traffic ramps over an hour can run leaner and let autoscaling do the work (Autoscaling Lag: The Gap Where the Outage Lives has the arithmetic).
Scale-up time, blast radius and cost complete the picture. If new capacity is serving in 45 seconds you need less standing headroom than if it takes six minutes. If the service is on the critical path for checkout, the cost of being wrong is revenue, not a slow page. And headroom is a continuous bill — the honest framing is "we spend X per month to absorb events of shape Y", which lets someone decide whether that trade is worth it rather than debating a percentage.
- Burst shape — measure the fastest observed rise: peak 10-second rate divided by the rate 60 seconds earlier. That ratio is what standing headroom must absorb.
- Scale-up time — measure end to end, from metric breach to a warmed instance serving at full capacity, not from the API call.
- Blast radius — a checkout path and an internal reporting job do not deserve the same headroom, and pretending otherwise overspends on one and underprotects the other.
- Cost — state headroom as money per month, not as a percentage, when presenting the trade-off to anyone who owns a budget.
- Deploy strategy — if rolling deploys remove 25% of capacity, that 25% is part of the headroom requirement, not separate from it.
Key points
- Utilization targets sit well below 100% because latency degrades non-linearly near saturation, not out of superstition.
- Average utilization hides bursts; read the short-window maximum during peak hour, and compare it to the load-tested knee.
- Performance headroom and redundancy headroom constrain each other — losing one of six instances adds 20% load to each survivor.
- Small fleets need proportionally more headroom because their failure granularity is coarser.
- The right number comes from burst shape, scale-up time, blast radius and cost — expressed as money per month, not as a convention.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Traffic burst → utilization: a 10-second 2× spike pushes a 70%-utilized fleet to 140% of capacity for the burst duration.
- 2Utilization → queueing: past the knee, requests wait rather than execute, and p99 rises far faster than the load did.
- 3Instance loss → survivors: with six instances, one failure redistributes 20% more load onto each remaining instance.
- 4Deploy → capacity: taking 25% of the fleet out of rotation during peak is a self-inflicted version of the same event.
- 5Queue → timeouts → retries: once wait times exceed client timeouts, retries add load to an already saturated fleet (Retry Storms: The Load You Generated Yourself).
- • "Average CPU is 45%, we are massively over-provisioned" — the peak-hour short-window maximum may be 90%.
- • "We survived last month's peak at 92%, so 92% is fine" — you survived without an instance failure or a deploy coinciding with it.
- • "Headroom is waste" — it is insurance with a premium; the question is whether the premium matches the risk, not whether it is nonzero.
- • "Autoscaling means we do not need standing headroom" — true only if scale-up completes faster than your fastest burst rises.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Peak-hour utilization at short resolution (10–15s), not the 5-minute average the dashboard defaults to.
- • The latency knee from a ramp load test: the utilization at which p99 crosses the objective.
- • Fastest observed traffic rise: peak 10s rate ÷ rate 60s earlier, over the last quarter of traffic history.
- • End-to-end scale-up time from metric breach to a warm instance serving full traffic.
- • Capacity removed by a rolling deploy, measured as instances out of rotation at once.
- • Set the utilization target from the measured latency knee, then subtract what one instance failure and one rolling deploy each consume.
- • Raise standing headroom for services whose bursts are faster than their scale-up time; lower it where autoscaling can genuinely keep up.
- • Increase fleet granularity (more, smaller instances) where N+1 at a sensible utilization is otherwise unaffordable.
- • Schedule deploys away from peak hour where the deploy's capacity cost is a meaningful share of the fleet.
- • Express the headroom decision as monthly cost against the events it absorbs, and get it agreed rather than assumed.
- • Terminate one instance during peak-hour traffic (in a game day or with a canary fleet) and confirm p99 stays within objective.
- • Run a spike load test at the measured fastest burst ratio and confirm the standing fleet absorbs it without shedding.
- • Deploy during a representative load and check that the capacity dip does not move p99 past the objective.
- • Headroom is a continuous cost paid to absorb intermittent events; too much is a permanent bill, too little is an occasional outage.
- • More, smaller instances improve failure granularity but increase per-instance overhead, connection counts and cold-start frequency.
- • Scheduling deploys away from peak reduces risk but slows delivery and concentrates changes into narrower windows.
- • Alert on peak-hour short-window utilization crossing the target, not on average utilization.
- • Track "instances required for N+1 at target utilization" as a derived metric and alert when the actual fleet drops below it.
- • Re-measure the latency knee after significant code or dependency changes — the knee moves when per-request cost moves.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe utilization figures, fleet sizes and survivor-load percentages are teaching examples. The survivor arithmetic is exact; the utilization thresholds at which latency degrades are workload-specific and must be found by load testing.
- WORKLOAD-SPECIFICWhere the latency knee sits depends on service-time variability. Services with highly variable request cost hit the knee at lower utilization than uniform ones.