Autoscaling: Scaling on the Right Signal
An autoscaling policy is a claim about what your bottleneck is. Scale on CPU and you have claimed the service is CPU-bound; when it is actually waiting on a database, the policy never fires while users time out.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The scaling signal is a model of your bottleneck
Every autoscaling policy encodes a hypothesis: "when this number is high, we need more instances". CPU-based scaling says the bottleneck is compute. Queue-depth scaling says the bottleneck is worker throughput. Concurrency-based scaling says the bottleneck is in-flight capacity, whatever the resource. If the hypothesis is wrong, the policy is not merely imprecise — it is silent during exactly the events it exists for.
This is why the choice of signal matters more than the threshold. A well-tuned threshold on the wrong metric is a policy that will never fire. And the failure is quiet: no alarm says "your scaling metric is uncorrelated with your latency", so teams discover it during an incident, when the fleet stayed at six instances through the whole event.
The test is simple and worth running once per service: during a load test, plot the scaling metric against p99 latency. If p99 can cross your objective while the scaling metric stays below its threshold, you have found the gap before production did.
| Signal | Bottleneck it assumes | Scales well for | Blind to |
|---|---|---|---|
| CPU utilization | Compute | Serialization, compression, rendering, computation-heavy handlers | I/O waits, lock contention, pool exhaustion — CPU stays low while latency explodes |
| Memory utilization | Working set size | Caches, in-memory datasets | Almost all latency problems; memory rarely correlates with request slowness |
| In-flight requests / concurrency | Whatever holds requests open | Nearly everything — it rises whether the wait is CPU, I/O or a lock | Distinguishing *why* requests are held; it scales even when scaling will not help |
| Queue depth or oldest-message age | Worker throughput | Async workers, batch pipelines (The Backlog Arithmetic: Four Levers and a Drain Time) | Synchronous request paths that have no queue to observe |
| Requests per second per instance | Uniform per-request cost | Homogeneous workloads | Traffic-mix shifts — the same RPS with heavier requests needs more capacity |
| Custom work metric | Whatever you defined | Domain-shaped load (tokens/s, rows/s, jobs/s) | Only what you thought to measure; needs maintenance as the workload evolves |
The I/O-bound service that never scales
Here is the canonical failure, and it is common enough to be worth recognizing on sight. A service spends most of its request time waiting on a database. Traffic rises, the database slows, requests pile up waiting for connections, and p99 goes from 180ms to eight seconds. CPU, meanwhile, barely moves — the process is not computing, it is waiting, and waiting consumes no CPU.
A CPU-based policy with a 70% target sees 22% and does nothing. The fleet stays flat for the entire incident. Afterwards the postmortem often concludes "autoscaling did not keep up", which is the wrong lesson — the policy worked exactly as configured, and the configuration encoded a false belief about the bottleneck.
The subtler point: adding instances here might not have helped anyway. If the constraint is database connections, more application instances open more connections and make it worse (Connection Pool Saturation: Waiting in Front of an Idle Database). A concurrency-based policy would at least have fired, which is better because it makes the constraint visible — but scaling the wrong tier is a separate mistake from scaling on the wrong signal, and both are worth naming out loud.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| CPU utilization (scaling metric) | 22% | Far below the 70% target — policy at rest all incident | suspect |
| p99 request latency | 8.2 s vs 300 ms objective | Users are timing out while the fleet holds steady | smoking gun |
| In-flight requests per instance | 410 (steady state: 30) | The signal that would have fired — requests held open, not executing | smoking gun |
| DB connection pool waiters | 300+ | The actual constraint; more app instances would add connections, not throughput | smoking gun |
| Fleet size | 6 → 6 | No scaling event occurred during a 40-minute degradation | suspect |
Scaling on work, and keeping the policy stable
For request-driven services, in-flight concurrency is usually the more honest signal than CPU: it rises whether requests are held by computation, I/O or a lock. Little's Law gives you the target directly — if one instance can hold capacity concurrent requests at your latency objective, target somewhat below that (Little's Law as Working Intuition, Concurrency Limits: An Unbounded Server Is a Slower Server). For async workers, queue depth and oldest-message age are the natural signals because the queue *is* the backlog (Depth Is Not an Emergency; Age Is).
Then there is stability. A policy that reacts to every fluctuation oscillates: it scales out, the metric drops, it scales in, the metric rises. Flapping costs money and, worse, keeps the fleet permanently cold because instances never live long enough to warm up. Asymmetric behavior is the usual remedy — scale out quickly and aggressively, scale in slowly and conservatively, since the cost of being briefly over-provisioned is much lower than the cost of being under.
Minimum and maximum matter as much as the target. The minimum is a capacity decision — it is the fleet that absorbs a burst before scaling can respond, which is exactly the standing headroom from Headroom: The Capacity You Deliberately Do Not Use. The maximum is a blast-radius decision: it stops a runaway loop or a traffic flood from scaling into a downstream tier's failure, or into a very large bill.
- Prefer concurrency for request services — it rises regardless of what is holding the request open.
- Prefer queue age for workers — depth alone cannot distinguish a million fast jobs from ten thousand slow ones.
- Scale out fast, scale in slow — asymmetry prevents flapping and keeps instances warm.
- The minimum is your standing headroom, not a cost-saving floor to be minimized.
- The maximum is a blast-radius control that protects downstream tiers and the invoice.
1# Signal: in-flight requests per instance (rises for CPU, I/O and lock waits alike)2# Target derived from Little's Law at the latency objective:3# per-instance capacity 220 req/s x 0.120 s objective ~= 26 concurrent4# target below that to leave room: 18 concurrent5 6scale_on: in_flight_requests_per_instance7target: 188metric_window: 30s # shorter -> faster reaction, more noise9 10scale_out:11 when: metric > target for 1 datapoint12 add: max(1, ceil(fleet * 0.5)) # aggressive: halve the gap fast13 cooldown: 60s14 15scale_in:16 when: metric < target * 0.6 for 5 consecutive datapoints17 remove: 1 # conservative: one at a time18 cooldown: 300s19 20min_instances: 6 # standing headroom: absorbs a burst before scaling reacts21max_instances: 40 # blast radius: protects the database tier and the billKey points
- An autoscaling policy encodes a hypothesis about your bottleneck; if the hypothesis is wrong, the policy stays silent during the incident.
- CPU-based scaling is blind to I/O waits, lock contention and pool exhaustion — the cases where latency degrades with CPU at 20%.
- In-flight concurrency is the more honest signal for request services; queue age is the right one for async workers.
- Verify the policy by plotting the scaling metric against p99 in a load test: if p99 can breach while the metric stays flat, the policy will never fire.
- Scale out aggressively and in conservatively; the minimum instance count is a headroom decision and the maximum is a blast-radius decision.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Traffic → dependency: request rate rises, database service time rises with it, and requests begin waiting rather than executing.
- 2Waiting → CPU: waiting consumes no CPU, so the scaling metric stays at 22% while in-flight requests climb from 30 to 410.
- 3Scaling metric → policy: the policy compares 22% to a 70% target and takes no action for the full duration of the event.
- 4No scaling → queue: in-flight work accumulates against a fixed fleet, pushing p99 to 8s and triggering client timeouts.
- 5Timeouts → retries: clients retry, adding load to a fleet that the scaling policy still considers idle (Retry Storms: The Load You Generated Yourself).
- • "Autoscaling failed" — the policy did what it was told; the signal was uncorrelated with the problem.
- • "CPU is low so the service is healthy" — for an I/O-bound service, low CPU during a latency incident is expected, not reassuring.
- • "We just need a lower CPU threshold" — a lower threshold on an uncorrelated metric fires at random times, not the right ones.
- • "Scaling out will fix it" — when the constraint is downstream, more instances add connections and pressure, not throughput.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Correlation between the scaling metric and p99 latency during a ramp load test — the plot that proves the policy can fire.
- • In-flight request count per instance at steady state and at peak, from an active-requests gauge.
- • Scaling event history: how many times the policy fired during the last three incidents, and at what point relative to the latency breach.
- • Flapping rate: scale-out events followed by scale-in within one instance lifetime.
- • Switch request-driven services to a concurrency-based signal with the target derived from Little's Law at the latency objective.
- • Switch async workers to oldest-message age, which distinguishes a deep fast queue from a shallow slow one.
- • Validate the policy in a load test by confirming it fires before p99 breaches the objective, not after.
- • Make the policy asymmetric — fast out, slow in — and set the minimum from standing headroom rather than from cost.
- • Set a maximum that protects the downstream tier, and alert when the fleet reaches it, since that is a capacity signal in its own right.
- • Re-run the ramp load test and confirm a scale-out event occurs before p99 crosses the objective, recording the margin.
- • Confirm the fleet returns to baseline within the expected window after load subsides, without oscillation.
- • Check scaling event counts over a week: repeated out-in pairs within a few minutes indicate flapping that needs a wider deadband.
- • Concurrency-based scaling fires reliably but will happily scale a tier that is not the constraint, spending money without fixing latency.
- • Aggressive scale-out costs money during transient spikes that would have resolved on their own.
- • Shorter metric windows react faster and flap more; longer windows are stable and arrive late.
- • Alert when p99 breaches the objective while no scaling event has occurred in the preceding window — the signature of an uncorrelated signal.
- • Alert when the fleet sits at max_instances, which means the policy has run out of room rather than solved the problem.
- • Re-validate the scaling metric after changes to the request path that shift where time is spent.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe incident numbers, targets and policy values are teaching examples. The Little's Law derivation of the concurrency target is exact arithmetic; the specific values are not from any real service.
- ENVIRONMENT-SPECIFICAvailable scaling signals, cooldown semantics and metric resolution differ substantially across orchestrators and cloud providers. The policy shape transfers; the syntax and the achievable reaction time do not.