How Autoscaling Fails
Scaling too late, on the wrong signal, into a downstream bottleneck, or against itself — the six failures that produce most autoscaling incidents.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Autoscaling is configured and the service still fell over. What went wrong?
Autoscaling is enabled once and then trusted, so its failures are discovered during the traffic events it was configured for.
If the fleet scales when load rises, autoscaling is working. The scale events in the log prove it.
Scale events prove the controller acted. They say nothing about whether capacity arrived before or after users noticed.
- Scale events prove the controller acted. They say nothing about whether capacity arrived before or after users noticed.
- A controller acting on the wrong signal produces a tidy graph of scale events that correlate with nothing the users experienced.
- A controller that succeeds can still cause the incident, by scaling a stateless tier into a stateful dependency that cannot follow.
- A controller that succeeds twice in opposite directions is oscillating, and each cycle costs start-up, cold caches and connection churn.
- The failures compound. Scaling late causes a backlog, the backlog causes retries, retries raise the signal further, and the controller now chases a load that its own lateness helped create.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Six failure families cover nearly all of it, and they are separable by what the graphs show: too late, wrong signal, cold start, downstream bottleneck, thundering herd, and oscillation.
- Too late is the delay budget exceeding the time available. It is not a misconfiguration so much as a mismatch between the load shape and the control loop (Autoscaling).
- Wrong signal means the metric does not move when the constraint saturates. The canonical case is CPU on an I/O-bound service (Choosing the Scaling Signal).
- Cold start means new capacity is present but not yet useful: cold caches, unconnected pools, un-JITed code. Routing traffic to it can make aggregate latency worse than not scaling at all.
- Downstream bottleneck is scaling that transfers the constraint. Connections, queries and dependency calls all multiply with instance count, so the tier you scaled is fine and the one behind it is not (The Connection Budget).
- Thundering herd is the simultaneity of the new units: they all start at once, all miss the cache at once, all open connections at once, and all hit the same dependency at once.
- Oscillation is delayed feedback without damping. The controller acts, the effect arrives late, the signal overshoots the target in the other direction, and it acts again.
- The self-referential trap sits underneath several of these: scaling changes the signal being scaled on, so a metric that is itself affected by fleet size makes the loop chase itself.
Six families, and what each one looks like
Diagnosis is usually fast once you know what to compare. Put error rate, the scaling signal and fleet size on one time axis: the relative ordering identifies the family before you read a single configuration file.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Load steps up quickly | Errors rise, then capacity arrives, then errors fall | Delay budget longer than the time available — scaled too late | Raise the floor, pre-scale on schedule, shorten start-up (Autoscaling) |
| Dependency slows down | Latency climbs, CPU flat, no scale events at all | Signal does not reflect the constraint — I/O-bound work looks idle by CPU | Scale on in-flight requests or pool saturation (Choosing the Scaling Signal) |
| New instances start serving | Latency worsens after capacity arrives | Cold caches, unconnected pools, un-warmed runtime | Warm before readiness; stagger arrivals; keep a warmer floor (Scale to Zero) |
| Fleet scales out successfully | Database connection errors across the whole fleet | Connections are per-instance; the server maximum is fleet-wide | Cap the fleet by the connection budget; use a pooler (The Connection Budget) |
| Many instances start together | A spike in dependency load and cache misses at the moment of arrival | Thundering herd — synchronised startup work | Jitter startup tasks; stagger scale-out in increments (Operating a Cache) |
| Load near the scaling threshold | Fleet size sawtooths; churn without a load change | Delayed feedback with no damping | Stabilisation window, tolerance band, asymmetric scale-in |
| Scale-in removes instances | Errors and duplicated work at the moment of termination | In-flight requests and unacknowledged messages killed | Drain connections; handle termination signals (Graceful Shutdown) |
| Fleet reaches maximum | Saturation with no further scale events and no alert | Ceiling reached silently — the default is not to tell you | Alert on ceiling reached; pair the ceiling with a shedding plan (Load Shedding) |
The one where scaling causes the outage
The downstream-bottleneck failure deserves its own treatment because it inverts the operator's instinct. The service is saturated, the obvious action is more capacity, and more capacity is what breaks it.
The reconstruction below is the shape this takes. Notice that every actor behaves correctly according to its own configuration.
- T+0changeTraffic rises; per-instance CPU and latency climb
- T+1mactionAutoscaler evaluates and increases the desired fleet size
- T+3mchangeNew instances become ready; each opens a full connection pool
- T+3msignalTotal fleet connections exceed the database server maximum
- T+4msignalConnection errors appear across every instance, old and new alike
- T+4mchangeFailed requests are retried, raising offered load and the scaling signal further
- T+5mactionAutoscaler scales out again, opening still more connections
- T+7mactionOperator pins the fleet at a fixed size and enables shedding for background traffic
- T+9mrecoveryConnection count falls below the maximum; error rate drops
- T+20mrecoveryFleet capped by a connection budget; ceiling set from that budget rather than from the account quota
The controller did exactly what it was configured to do at every step. The configuration error was a ceiling derived from what the platform allowed rather than from what the database could accept — which is the general form of this failure.
Damping, and what it costs
Oscillation is fixed by making the loop less eager, and every mechanism for doing that also makes it slower to respond. Choosing among them is choosing which failure you would rather have.
evaluate every interval no tolerance band symmetric scale-out and scale-in act on the instantaneous value => load hovers near threshold out, in, out, in each cycle: boot cost, cold cache, connection churn, warm-up latency
evaluate every interval tolerance band around the target scale out fast, scale in slowly stabilisation window on scale-in bounded change per interval => fewer, larger, deliberate moves capacity retained through short dips cost: slower to release capacity, and slower to react to a real drop
The asymmetry is the design principle: being too large is a cost problem and being too small is an availability problem, so the loop should be biased towards being too large. Undamped policies are symmetric, which treats those two outcomes as equally bad — and they are not.
How to do it properly
Most important first.
- Compare capacity arrival with error onset for every real traffic event. That single comparison distinguishes "too late" from every other failure (A Successful Deploy Is Not Evidence of a Healthy System has the same shape).
- Verify the signal moves when the constraint saturates, by load testing to saturation and watching whether the metric responded.
- Give the fleet a floor sized to survive the delay budget, and pre-scale for anything predictable.
- Bound the ceiling by what the downstream can survive, and treat the connection budget as a fleet-level property rather than an instance-level one (The Connection Budget).
- Stagger and jitter startup work — cache warming, connection establishment, scheduled fetches — so a batch of new instances does not arrive as a synchronised spike (Operating a Cache).
- Damp the loop: stabilisation windows, asymmetric scale-out and scale-in, a tolerance band, and a limit on how much the size can change per interval.
- Drain properly on scale-in, so removing capacity does not fail in-flight work (Draining: Stopping Without Dropping, Graceful Shutdown).
- Have a documented way to pin the fleet during an incident and take the loop out of the picture (Runbooks).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A ceiling contains the downstream damage and a floor contains the lateness. Both are configuration set in advance; during the incident the only containment is pinning the fleet.
What can go wrong
- Damping added to stop oscillation, set so conservatively that the system can no longer respond to real load.
- A ceiling added to protect the database, then raised during an incident and never lowered.
- Scale-in disabled to avoid churn, so the fleet ratchets upward permanently and nobody notices until the bill (Idle Capacity).
- Warm-up handled by delaying readiness, which makes the effective scaling delay longer and pushes the system back into the "too late" family (Probes: Readiness, Liveness and Startup).
- Multiple autoscalers acting on the same workload — a horizontal one and a vertical one, or a platform one and a custom one — fighting each other.
- The mitigation failing: pinning the fleet during an incident and forgetting to re-enable the loop afterwards.
- "The autoscaler failed." More often the autoscaler did exactly what it was told, on a signal that did not reflect the constraint. Check the signal before the controller.
- "We need faster scaling." Often you need a higher floor. Scaling faster is bounded by physics; standing capacity is bounded by budget.
- "Adding instances always helps." It helps only when the constraint is per-instance. When it is shared, adding instances is the mechanism of the outage (Building a Capacity Model).
- "Oscillation is cosmetic." Each cycle pays start-up cost, discards warm caches and churns connections, so an oscillating fleet is slower and less reliable than a static one of the same average size.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For the last traffic event: a graph of error rate, signal, and fleet size on one time axis. The ordering of the three tells you the family.
- Scale events annotated on the incident timeline next to deploys and alerts (Reconstructing What Actually Happened).
- Downstream connection count and saturation plotted against fleet size, which makes the bottleneck-transfer failure obvious.
- A count of scale-out followed by scale-in within a short window — a direct oscillation measure.
- Per-instance latency for the first minutes after start, which quantifies cold start rather than assuming it.
- Pin minimum equal to maximum at a size you know works. This is the single most useful autoscaling incident action and it should be a documented one-liner, not an improvisation.
- Policy changes made during an incident should be recorded and reviewed afterwards; the raised ceiling and the disabled scale-in are the two that persist and cause the next problem.
- If scaling caused a downstream outage, rolling back the fleet size does not immediately restore the dependency — connection storms and cache misses have their own recovery time (Operating a Cache).
- Automate the damping and the drain, both of which are mechanical and both of which humans get wrong under pressure.
- Automate an alert for the ceiling being reached and for oscillation, since neither is visible without looking for it.
- Keep the incident decision human: whether to pin the fleet, raise the ceiling or shed load depends on which failure family you are in, and that is a diagnosis rather than a threshold.
- Every mitigation for oscillation slows the response to genuine load. Damping is a direct trade between stability and responsiveness.
- A ceiling that protects the database also guarantees saturation once it is reached, which means you must pair it with a shedding plan (Load Shedding).
- Staggering startup work reduces herd effects and lengthens the time until new capacity is fully useful.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALAll six families occur on every platform. The mechanics differ: on serverless, cold start and downstream connection storms dominate, while on a VM group lateness dominates because boot time is the largest term.
- SIMULATEDThe timeline in this lesson is a composite reconstruction produced for teaching, not a report of a specific incident. The ordering of events is what transfers; the timings are illustrative.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — synchronised behaviour across independent clients, and why jitter is the standard defence.