Capacity Management
Knowing how much load a system can carry, which resource runs out first, and what the moment of saturation looks like from outside.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How much traffic can this system safely carry, and what gives way first when it cannot?
Systems do not degrade gracefully by default. They serve traffic fine, then one resource runs out, and the failure surfaces somewhere that is not the resource that ran out.
Watch CPU. If CPU is comfortable there is capacity left; if it climbs, add instances. It is the first number on every dashboard and it has been right often enough to trust.
CPU is the constraint for compute-bound services and almost nothing else. An API that spends its time waiting on a database looks idle by CPU right up to the moment its connection pool is empty and every request is queueing for one.
- CPU is the constraint for compute-bound services and almost nothing else. An API that spends its time waiting on a database looks idle by CPU right up to the moment its connection pool is empty and every request is queueing for one.
- Saturation is not a slope you can watch. Below the knee, added load costs a little latency; above it, queueing makes latency rise far faster than load does, and the transition happens over minutes (Building a Capacity Model).
- The resource that runs out is frequently not in your service. Database connections, a downstream quota, a NAT gateway port range and a message broker's disk are all capacity limits your dashboard does not show.
- Capacity is not a single number, because the system does different work at different times. A fleet sized for read traffic can be saturated by a nightly export it also serves.
- Nobody discovers the limit deliberately. They discover it during a launch, a marketing email, or a retry storm caused by an unrelated failure.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Capacity management answers three separate questions, and teams that conflate them get confusing answers: what can we carry now, what runs out first, and what happens at the moment it does.
- Every system has exactly one binding constraint at a time — the resource that saturates before any other. Everything else has slack. Adding capacity to a non-binding resource changes nothing, which is why "we doubled the instances and it did not help" is such a common sentence.
- When the binding constraint changes — because you fixed the old one, or because traffic changed shape — the system's failure signature changes with it. Fixing a bottleneck does not remove the bottleneck; it moves it.
- Saturation converts a throughput problem into a latency problem. Work arrives faster than it is served, a queue forms somewhere, and the queue is what the user experiences. That is why the symptom is almost always latency and timeouts, not a tidy "out of capacity" error.
- The last thing a saturating system does is amplify. Slow responses cause client retries, retries add load, and the added load makes responses slower. Past the knee, the system participates in its own overload.
Three questions, not one
People say "do we have capacity?" and mean three different things. Separating them makes the work tractable, because each has a different method and a different owner.
| Question | What answers it | Who usually owns it | Failure when skipped |
|---|---|---|---|
| How much can we carry now? | A load test run to the point something gives | The service team | The limit is discovered by traffic |
| What runs out first? | Watching which resource saturates in that test | The service team | Capacity added to the wrong resource |
| What happens at the limit? | A deliberate degradation design | Service and platform together | Uniform timeouts and a retry storm |
| How much reserve do we hold? | A headroom policy with named claims | The owning team (Headroom) | A deploy or a node loss becomes an outage |
| What does it cost to hold? | A cost model per unit of served work | The owning team (Cost Per Request) | Capacity decisions made with no price attached |
What actually runs out
The binding constraint is rarely the one on the front of the dashboard. This table is a starting checklist: for each row, ask whether you know the ceiling and whether you can see the current level.
Note how many of these live outside the service. Roughly half of the real limits a team meets are in a dependency they do not operate.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Compute-bound work at high request rate | Latency rises with load; CPU near ceiling | CPU is genuinely the constraint | Scale out, or make the work cheaper per request |
| Growing working set or a leak | Process restarts; requests fail in bursts | Memory limit reached and the runtime is killed (OOMKilled: Over the Memory Limit) | Fix the growth; size the limit against measured usage, not hope |
| I/O-bound service under load | Latency climbs, CPU flat and low | Waiting on a dependency; the pool, not the CPU, is full (The Connection Budget) | Size and monitor the pool; scale the dependency, not only the caller |
| Fleet scaled out | Database rejects connections while app instances look healthy | Connections are per-instance, so the ceiling arrives as you add instances | Treat connections as a budget shared across the fleet |
| Producers outpace consumers | Queue depth and message age both rising | Consumer throughput is the constraint (Operating Queues and Scheduled Work) | Scale consumers on backlog age (Queue-Based Autoscaling) |
| Large payloads or chatty fan-out | Throughput plateaus below CPU limits | Network bandwidth or packets-per-second ceiling on the instance type | Reduce payload size or fan-out; choose a different instance class |
| Steady write growth | Sudden hard failure on write | Disk or volume full — the one constraint with no graceful degradation | Alert on projected time-to-full, not on percentage used |
| Traffic to a managed service or third party | Throttling responses at a fixed rate | A provider quota you did not set and cannot exceed | Know the quota, request increases ahead of launches, shed against it |
The knee, and why the last part happens so fast
Utilisation and latency are not linearly related. Well below the constraint, extra load is absorbed. Approaching it, each additional unit of load waits behind work already in progress, so latency grows disproportionately — this is queueing, and Performance owns the theory.
The operational consequence is what matters here: the interval between "the graph looks slightly worse" and "everything is timing out" is short, and it gets shorter as the system gets busier. Capacity decisions therefore have to be made from the shape of the curve, not from the current point on it.
Then amplification takes over. Clients time out and retry, so offered load rises exactly when served load is falling. A system past the knee is being pushed harder by its own failure.
How to do it properly
Most important first.
- Name the binding constraint for each service, explicitly, in the runbook. If nobody can say what runs out first, the capacity story is a guess (Runbooks).
- Measure the limit rather than deriving it: load test until something gives, and record what gave (Building a Capacity Model).
- Alert on saturation of the binding constraint, not on utilisation of everything — a pool at its ceiling is actionable, a CPU graph is not (Alert on Symptoms, Not on Causes).
- Re-derive capacity after any change to the work shape: a new endpoint, a new dependency call, a larger payload, a heavier query.
- Decide in advance what happens at saturation. A system with no answer picks the worst one — uniform timeouts for everybody (Load Shedding).
- Hold reserve deliberately rather than discovering it is gone (Headroom).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing contains saturation once it starts, because it is not a change you can revert — only shedding and shifting load bound it (Load Shedding).
What can go wrong
- A capacity model derived from a load test that only exercised the cheap endpoint, so the number is real and irrelevant.
- Capacity measured on a fleet that was warm, with caches full and connections established, and never re-checked cold.
- Autoscaling treated as the capacity answer, so nobody knows the limit — the fleet grows until it hits a quota or the database, and then the incident starts (Autoscaling).
- A constraint that lives in a shared dependency, so each service passes its own capacity review and the platform saturates anyway.
- The mitigation failing: a load test large enough to be meaningful runs against production and becomes the outage it was meant to prevent.
- "We autoscale, so capacity is handled." Autoscaling changes how many instances you have. It does not change the database connection ceiling, the third-party quota, or the fact that scaling takes time (How Autoscaling Fails).
- "We are at 60% CPU, so we can take 60% more traffic." Utilisation is not linear near the knee, and CPU is probably not the constraint anyway.
- "The load test passed." A load test that stopped below the limit tells you the limit is somewhere above it, and nothing else.
- "Capacity is a platform team problem." The constraint is usually inside the service — its query pattern, its pool size, its per-request memory.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A recorded load-test result naming the request rate at which a specific resource saturated, and which one it was.
- A saturation alert that has fired before a user-visible incident rather than during one.
- Utilisation of the binding constraint on the operator dashboard, alongside traffic — so the two can be read together (Dashboards an Operator Can Act On).
- The last traffic peak has an entry showing what the peak was and how much of the limit it used.
- Capacity changes made under pressure — raising a limit, adding instances — are usually reversible, but reverse them deliberately and not by forgetting: an emergency scale-up left in place becomes the new baseline cost nobody chose.
- A capacity limit raised on a shared dependency (connection ceiling, broker quota) is often not safely reversible while traffic is on it, because the clients that grabbed the new headroom will not give it back gracefully.
- Automate the measurement: a scheduled load test against a production-like environment that publishes the saturation point rather than a pass/fail.
- Automate saturation alerting on the constraint you named.
- Keep the judgement human: whether a marketing launch justifies pre-scaling, and whether to accept degradation rather than pay for peak, are business decisions wearing an engineering costume.
- Knowing your limit costs a real load test, a production-like environment and the time to run it — none of which are free and all of which compete with feature work.
- A capacity model is accurate for the traffic shape it was built from and decays as the system changes, so it is maintenance, not a document.
- Sizing for the measured peak means paying for the peak all the time unless you can scale or shed (Overprovisioning).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThat a single resource binds first, and that saturation converts throughput problems into latency problems, holds on any stack. Which resource binds is entirely workload-dependent.
- SCALE-SPECIFICBelow the traffic level where any resource is near its ceiling, this whole practice collapses into "we have plenty" — correctly. It becomes urgent at the point where a single traffic peak can reach a limit, which arrives earlier for connection-bound services than for stateless ones.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — System Design — sizing a system from a traffic estimate before it exists, which is the same arithmetic run forwards.