The question this answers
How does the amount of capacity running follow the amount of demand arriving, without a human watching a graph?
The checkout API serves 400 req/s on an ordinary Tuesday and 6,000 req/s in the ninety minutes after the marketing email goes out. Provisioning for the peak wastes every Tuesday; provisioning for Tuesday loses the campaign.
A control loop that adds and removes capacity against an observed signal, so the fleet is sized for the load currently arriving rather than for the worst load anyone remembers.
Four different things are called autoscaling
The word covers four mechanisms that solve different problems and fail in different ways. Conflating them is why teams end up with a horizontal scaler attached to a workload that cannot run twice, or a scheduled rule that was correct for last year's traffic shape.
Horizontal changes how many instances exist. Vertical changes how large one instance is. Scheduled changes the count on a clock. Metric-based changes the count from an observed signal. Only the last two are what people usually mean, and only horizontal scaling is available to a workload that keeps state in local memory — see Stateful Workloads: Databases Are Not Stateless APIs for what has to move first.
A fifth option is worth naming because it is often the right one: scale nothing, and let a queue absorb the burst. If the work is asynchronous, a backlog that drains in twenty minutes is cheaper and far more predictable than a fleet that triples and then has to shrink.
| Mechanism | What changes | How fast | Requires | Where it breaks |
|---|---|---|---|---|
| Horizontal | Instance count | Minutes — see Startup Time & Cold Start | A stateless workload behind a load balancer | The shared database it all points at does not scale with it |
| Vertical | CPU/memory of one instance | A restart, so a gap in service unless replaced first | A workload that cannot be split — a primary database, a single-writer job | There is a largest instance, and you will meet it |
| Scheduled | Count, on a clock | Instant, because it provisions *before* the load | A demand pattern that genuinely repeats | The day the pattern changes and nobody updates the schedule |
| Metric-based | Count, from a signal | Always lagging — the metric window alone is a minute | A signal that represents workload pressure (Autoscaling Signals) | CPU chosen by default on an I/O-bound service |
| Queue + workers | Nothing — latency absorbs the burst | Immediate, by design | Work that may complete later | Interactive requests, where a backlog is an outage |
The loop, and where the time actually goes
A metric-based scaler is a control loop, and every stage of it costs wall-clock time. The scaler does not see the current load; it sees an aggregate over a window that already closed. It does not act on the first breach; it waits for a breach to persist, because otherwise a single garbage-collection pause would double the fleet. Then it asks for capacity, and capacity takes as long as it takes to boot.
Add the stages up honestly and the number is uncomfortable: a scaler with a 60-second metric window, a two-period breach requirement and a 90-second instance boot does not deliver usable capacity until roughly three and a half minutes after the load arrived. For a traffic spike that peaks in ninety seconds, the new instances land after it is over — they are paying for the *next* spike, not this one.
This is why cooldown matters as much as threshold. Without it the loop oscillates: capacity arrives late, the metric is now below target because the spike passed, the scaler removes capacity, load returns, and the fleet flaps. Flapping costs more than being slightly over-provisioned, because every cycle pays the full provisioning and warm-up bill again.
- 1Metric window closes~60s
The scaler receives an aggregate for a period that has already ended.
You are steering by a signal that is at minimum one window old.
- 2Breach persists~60s
The scaler waits for N consecutive periods above target so noise does not trigger it.
Tuned too tight and the fleet flaps on garbage-collection pauses.
- 3Decisionseconds
A target quantity is computed and a capacity request is issued.
Step size too small means several loops to catch up; too large means overshoot.
- 4Capacity allocated~30s
The provider assigns a host, attaches network and storage, starts the instance.
Insufficient-capacity errors in one zone, or a quota you forgot you had.
- 5Boot and warm-up~60–120s
Image pull, runtime start, connection pools filled, caches populated.
A large image or a slow dependency dominates everything above it.
- 6Health check passes~30s
The load balancer marks the target healthy and begins sending traffic.
A readiness check that returns 200 before warm-up finishes routes traffic into a cold process.
- 7Cooldown~300s
Further scaling decisions are suppressed while the effect of this one is observed.
No cooldown means oscillation; too much means the second, larger spike is unanswered.
Autoscaling only saves money if scale-in works
The business case for autoscaling is the trough, not the peak. Scaling out protects availability; scaling *in* is what pays for it. In practice scale-in is the half that gets disabled — someone was burned by an aggressive removal during an incident, set the scale-in threshold to something unreachable, and the fleet has run at peak size ever since. The bill looks exactly like a fixed fleet, with the operational complexity of a dynamic one on top.
Scale-in also needs the workload to tolerate being told to stop. An instance chosen for removal must drain in-flight requests, deregister from the load balancer, and finish or hand back whatever it was working on. Without that, every scale-in event produces a handful of 502s that nobody attributes to autoscaling. See Graceful Shutdown: The 502 Spike Nobody Investigates.
And note the second-order effect: a fleet that scales freely multiplies its dependencies. Two hundred instances mean two hundred sets of database connections. The autoscaler is happy; the database connection limit is the thing that actually falls over — see Serverless and Database Connections for the extreme version of the same failure.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Key points
- Horizontal, vertical, scheduled and metric-based are four different mechanisms with different reaction times and different prerequisites.
- A metric-based scaler is always reacting to load that has already arrived: window + breach + provision + boot + health check is minutes, not seconds.
- Scheduled scaling is the only kind that provisions *before* the load, which is why predictable peaks should use it.
- Autoscaling pays for itself through scale-in. A fleet where scale-in was quietly disabled is a fixed fleet with extra moving parts.
- Scaling the stateless tier multiplies pressure on everything it depends on — connections, downstream APIs, the NAT path.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • A scaling group holds a desired count, a minimum and a maximum, plus a template describing what an instance is.
- • A metric source aggregates a signal over a window; the scaler compares that aggregate against a target.
- • A sustained breach produces a new desired count. The group provisions or terminates instances to converge on it.
- • New instances register with the load balancer and receive traffic only after passing a health check (Health Checks).
- • A cooldown suppresses further decisions while the previous one takes effect, which is what stops the loop oscillating.
- • Own the minimum: it is your real availability floor, and it must survive the loss of one zone with the remainder.
- • Own the maximum: it is the last line of defence against a runaway loop or a retry storm scaling you into a five-figure hour.
- • Own the launch template lifecycle — a stale image in the template means every scale-out event deploys last month's code.
- • Own scale-in safety: draining, deregistration delay, and protection for instances doing long-running work.
- • Re-tune after every material change to the workload. Thresholds set for a synchronous API are wrong for the same service once it grows a background job.
- • Scaling lag: the fleet doubles four minutes after the spike, so users saw errors and the extra capacity serves a trough.
- • Flapping: no cooldown, so the loop adds and removes capacity every few minutes and bills for both.
- • Ceiling hit: maximum reached, requests queue, latency climbs, and the dashboard shows a scaler that believes it is finished.
- • Scale-in during an incident: a metric drops because requests are failing fast, the scaler removes capacity, and recovery is now slower.
- • Provisioning failure: the zone has no capacity of that instance type, so the group silently stays under target while the metric screams.
- • Downstream collapse: the tier scaled, and the database connection limit or a third-party rate limit became the outage instead.
- • The stateless tier scales close to linearly; the first thing to stop scaling with it is almost always the shared datastore.
- • Reaction time does not improve with fleet size — a 200-instance fleet still takes the same three minutes to add the 201st.
- • Very large groups hit provider-side limits first: instance quotas, IP addresses in the subnet, capacity of one instance family in one zone.
- • Beyond a certain rate of change, pre-provisioning (scheduled or warm pools) is the only mechanism fast enough. Reactive scaling cannot out-run a step function.
- • Every launched instance receives an identity from the launch template. Over-broad instance roles are inherited by every instance the scaler ever creates — see Roles vs Static Keys.
- • The maximum count is a security control as much as a cost control: it bounds what an attacker can spend during an abusive traffic pattern.
- • New instances must fetch secrets at boot rather than baking them into the image, because the image is copied to every instance the scaler starts.
- • A scaler with permission to terminate instances is a destructive capability. Scope who and what may modify the group (Least Privilege in Infrastructure).
- • Baseline instance-hours dominate; the elastic part is usually the smaller number, which is why right-sizing the baseline beats tuning thresholds.
- • Oscillation and short-lived instances waste money through provisioning overhead and minimum billing increments.
- • Scaling out multiplies per-instance costs you may not associate with capacity: image pulls, agent licences, log volume, NAT-processed bytes.
- • The honest comparison is against a fixed fleet sized for peak. If peak is only 1.5× trough, autoscaling may cost more than it saves once you price the complexity.
- • Desired vs running vs healthy instance count on one chart — the gap between the three is where every scaling incident lives.
- • Scaling activity events with their reason strings, which tell you whether the loop is converging or flapping.
- • Time from scaling decision to first request served, measured per launch. This is the number people guess at and are wrong about.
- • Saturation of the resource that actually limits the workload, not just the one you scale on (Autoscaling Signals).
- • The signal that lies: average CPU across the fleet. It falls the moment new instances join, which makes a failing scale-out look like a successful one.
- • A fixed fleet sized for peak. If the peak-to-trough ratio is under about 2× — or the fleet is three instances — this is simpler, cheaper to reason about, and has no lag. Start here.
- • Scheduled scaling alone, when demand is predictable. It has none of the lag and none of the signal-choice problems, and it is the right answer for a business-hours workload.
- • One larger instance. Vertical scaling costs a restart and has a ceiling, but for a workload that cannot be split it is the only honest answer.
- • A queue with a fixed worker pool, when work may complete later. The backlog absorbs the burst and the capacity never changes.
- • A platform that scales for you — serverless, or a managed container service with request-based scaling — when you would rather buy the loop than tune it. See Serverless Trade-offs for what that costs.
- • Buys elasticity; costs a control loop with its own failure modes, tuning burden and misleading metrics.
- • Scale-in saves money and creates a new class of incident: instances removed while doing work, and capacity removed during a recovery.
- • The elastic tier makes fixed downstream limits sharper — you have moved the bottleneck rather than removed it.
- • A wide range between minimum and maximum gives resilience to surprise and exposes you to a large bill from a retry storm.
Autoscaling: capacity is always late
desired = ceil(running × util ÷ target) → ceil(2 × 0.82 ÷ 0.70) = 3 capacity gap: traffic peaks at 2.6k rps = 13 instances; each one takes 8s to boot, so the fleet is 8s behind the truth at all times.
What people believe, and what is true
Autoscaling means we never run out of capacity.
It means capacity arrives some minutes after you needed it. Anything faster than that window must be pre-provisioned.
Autoscaling always saves money.
It saves money in the trough. With scale-in disabled or a modest peak-to-trough ratio, a fixed fleet is cheaper and simpler.
If the fleet scaled, the incident is over.
The tier that scaled is rarely the one that broke. Check the database connections, the downstream rate limit and the NAT before declaring victory.