Autoscaling Lag: The Gap Where the Outage Lives

The scaling API call is the fastest part of scaling, which is why measuring from that point produces such optimistic numbers. The honest measurement starts when load rises and ends when a new instance is serving at full capacity. Everything in between is a delay you can measure, and most of it happens before or after the part people time.

The metric window comes first: a 60-second window means the metric reflects load that is already up to a minute old. Then the evaluation interval, where the policy checks the metric on a schedule and may require several consecutive breaches. Then provisioning, where the platform allocates and starts an instance. Then boot: process start, dependency connections, health check passing. And finally warm-up — the part almost nobody counts — where the instance has a cold cache, a cold connection pool and, in JIT runtimes, unoptimized code (JIT and Warm-Up: The First Thousand Requests Are a Different Program).

Add them up and three minutes is a realistic total for an ordinary containerized service. If your traffic spikes last ninety seconds, autoscaling contributes nothing to that event: the spike is over before the capacity is useful. This is not a criticism of autoscaling, it is a statement about what problem autoscaling solves — it handles sustained load changes, not bursts.

During those three minutes the existing fleet absorbs the entire spike alone. If the spike exceeds standing headroom, the fleet passes its latency knee, requests queue, and wait times grow. Once wait times exceed client timeouts, clients retry — and retries arrive as new load on a fleet that is already saturated (Retry Storms: The Load You Generated Yourself). The gap is where a manageable traffic increase turns into a self-amplifying incident.

Worse, the arriving capacity briefly makes things harder before it makes them better. A cold instance opens new database connections, misses on every cache lookup, and runs unoptimized code paths. For its first seconds it consumes shared resources while serving fewer requests than an established instance. Scale out ten instances at once into a struggling database and the cold-start cost can deepen the incident it was meant to relieve (Cache Stampede: Everyone Misses at Once describes the cache side of this).

This is why the fleet you start the spike with matters more than the fleet you can eventually reach. Standing headroom (Headroom: The Capacity You Deliberately Do Not Use) is what carries you across the gap; autoscaling is what stops you from paying for that headroom permanently at a level sized for the sustained peak.

Every part of the chain can be shortened, and every shortening has a price. Reduce the metric window from 60s to 15s and you react a minute sooner at the cost of noisier signals and more flapping. Drop the consecutive-breach requirement and you react faster and scale on transients. Keep a pre-warmed pool of instances and you skip provisioning and boot entirely, but you pay for idle capacity — which is standing headroom wearing a different hat.

Predictive and scheduled scaling attack the problem from the other side: if the spike is foreseeable — a daily peak, a marketing send, a scheduled batch — scale before it rather than in response to it. This is the highest-leverage fix available for predictable traffic, and it does nothing for genuine surprises.

Then there is admission control. If you cannot get capacity in time, decide deliberately what happens instead: shed low-priority traffic, serve a degraded response, or queue with a bounded wait so that some requests succeed properly rather than all of them timing out (Concurrency Limits: An Unbounded Server Is a Slower Server). Choosing the degradation is far better than discovering it.