The question this answers
Which measurement should decide how much capacity is running — and why is the easiest one to read usually the wrong one?
The document-processing API spends most of its time waiting on an external OCR service. At 30% CPU it is already refusing work, because every one of its 50 pooled connections is in use. The users see timeouts; the CPU chart is flat and green.
A scaling decision driven by the resource that actually runs out first, so capacity is added when the workload is under pressure rather than when one convenient number happens to move.
Scale on the signal that represents workload pressure
That sentence is the entire lesson. The scaler is a controller, and a controller is only as good as its input. If the input does not correlate with the thing users experience, no amount of threshold tuning will help — you have built a fast, automatic way to make the wrong decision.
"Workload pressure" means: the resource that is closest to exhausted. For a compute-heavy image transcoder that genuinely is CPU. For a service that spends 90% of its time waiting on a database, an external API or a disk, CPU is nearly meaningless — the process is blocked, not busy. Its pressure lives in concurrency: in-flight requests, pooled connections, thread or event-loop occupancy. For an asynchronous worker, pressure is the backlog and how fast it is growing.
CPU became the default because it is the one metric every platform emits without instrumentation. That is a statement about tooling convenience, not about your workload. The question to ask before setting any target is: *when this service is in trouble, which number moves first?* Scale on that one.
| Signal | Represents | Right for | Wrong for | How it misleads |
|---|---|---|---|---|
| CPU utilization | How busy the processor is | Transcoding, compression, inference, parsing | Anything I/O-bound | Stays low while the process is blocked waiting; also drops the instant new instances join |
| Memory utilization | How much of the heap is committed | Caches, in-memory joins, large working sets | Runtimes with lazy garbage collection | A JVM or Go heap sits near its ceiling by design; you scale on the collector, not on demand |
| Request concurrency | In-flight requests per instance | Synchronous APIs, especially I/O-bound ones | Very long-lived connections | Usually the best default for a web tier, and it maps directly to user-visible latency |
| Queue depth | Unprocessed work waiting | Workers, batch, event consumers | Interactive request paths | Depth alone hides direction — a stable 10,000 is fine, a rising 200 is an incident |
| Connection-pool saturation | Fraction of the pool checked out | Anything in front of a database | Pool-free architectures | The signal in the opening example. Frequently the true limit and almost never graphed |
| Custom application metric | Whatever the workload is really limited by | Domain-specific pressure: tokens/s, active sessions, GPU memory | Teams with no metrics pipeline | Requires plumbing, and a metric nobody maintains becomes a scaler nobody trusts |
The I/O-bound service at 30% CPU
Here is the incident in full, because it is the most common autoscaling failure there is. A service accepts uploads and calls an external OCR provider. Each request holds a worker slot for two to eight seconds while that call is outstanding. The instance has 50 slots. Traffic doubles, all 50 slots fill, the 51st request queues, and once the queue exceeds the accept backlog, connections are refused. Latency goes from 300 ms to 30 seconds and then to errors.
Throughout, CPU sits at 28%. The scaler is configured for a 70% CPU target, so it does nothing. The fleet dashboard is green. The load balancer reports every target healthy, because the health endpoint returns from a dedicated path that never touches the pool. The first person to understand it is whoever thinks to look at in-flight request count per instance — a number nobody had graphed.
The fix is to scale on concurrency: target roughly 60–70% slot occupancy per instance, and the scaler acts while the queue is still short. The deeper fix is to know your limiting resource before an incident tells you. Load-test the service and watch which number reaches its ceiling first; that is your scaling signal, and it is also the answer to "how many requests can one instance take?"
metric t=0 t=60s target verdict ----------------------------------------------------------------------- cpu.utilization 24% 28% 70% quiet memory.utilization 41% 43% 80% quiet http.requests_in_flight 19 / 50 50 / 50 35 / 50 SATURATED http.accept_queue_depth 0 214 0 SATURATED db.pool.checked_out 6 / 20 20 / 20 14 / 20 SATURATED upstream.ocr.latency_p95 310 ms 6400 ms < 1000 ms degraded lb.target_health healthy healthy healthy MISLEADING The scaler is watching row 1. The users are living in rows 3-6.
Queue depth is a rate problem, not a level problem
Worker fleets get a better signal than web tiers because the backlog is directly observable — but depth on its own is a trap. A queue holding 10,000 messages that drains at exactly the rate it fills is healthy. A queue holding 200 that has grown every minute for an hour is an outage in twenty minutes. The number that matters is the derivative, or better, the *estimated time to drain*: backlog divided by current throughput.
Scaling on time-to-drain gives the scaler a target expressed in the language the business actually uses. "Keep the backlog under five minutes of work" is a service-level objective a product owner can agree to, and it translates mechanically into a worker count. Scaling on raw depth requires someone to guess a number that silently becomes wrong the moment per-message processing time changes.
The comparison below is the same autoscaler configuration written twice. Both are valid; only one of them reacts before users notice.
autoscaling:
min: 2
max: 20
metric:
type: resource
name: cpu
targetAverageUtilization: 70
# Workers spend their time blocked on an HTTP call to the OCR
# provider. Fleet CPU peaks around 30% no matter how deep the
# backlog gets, so this scaler stays at min: 2 during an incident.autoscaling:
min: 2
max: 20
metric:
type: external
name: queue_seconds_of_backlog # depth / throughput
targetValue: 300 # keep under 5 minutes of work
behavior:
scaleUp: { stabilizationWindowSeconds: 0 } # react immediately
scaleDown: { stabilizationWindowSeconds: 600 } # retreat slowlyThe first configuration scales on a number that does not move when the workload is in trouble. The second scales on the thing the workload is actually limited by, with an asymmetric window: fast to add capacity, slow to remove it, because being wrong in those two directions costs very different amounts.
Key points
- Scale on the signal that represents workload pressure — the resource closest to exhausted, not the one easiest to read.
- CPU is the default because every platform emits it, and it is wrong for any workload that spends its time waiting.
- For synchronous services, in-flight request concurrency is usually the better default; it maps directly to user-visible latency.
- For workers, use time-to-drain (backlog ÷ throughput) rather than raw queue depth — depth without direction says nothing.
- Make scale-up fast and scale-down slow. The two errors have very different costs.
- The limiting resource is often a connection pool nobody has graphed.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The workload or the platform publishes a metric; the scaler reads an aggregate of it over a window.
- • The scaler computes desired capacity, typically as
current × (observed ÷ target), and clamps it to the min/max range. - • Custom and external metrics require a bridge — an adapter or metrics API — that turns an application metric into something the scaler can query.
- • Multiple metrics are usually combined by taking the maximum of the individual recommendations, so any one signal can scale out but no single one can scale in alone.
- • Stabilization windows apply asymmetrically: a short one on the way up, a long one on the way down.
- • Load-test to find the ceiling before choosing a target: run one instance to failure and record which number moved first.
- • Own the metric pipeline. A custom scaling metric that stops being published is an autoscaler that silently stops working — and most fail open, staying at their last count.
- • Re-derive the target when the workload changes shape. Adding a synchronous call to a new dependency can move the limit from CPU to concurrency overnight.
- • Graph the candidate signals side by side during incidents, not just the one you scale on. That is how you discover the pool.
- • Set targets with headroom for the provisioning delay: targeting 90% occupancy means the queue forms before capacity can possibly arrive.
- • The wrong signal: CPU on an I/O-bound service, so the scaler never fires while users time out.
- • Metric pipeline failure: the custom metric stops arriving and the scaler freezes at its current size with no alarm.
- • Averaging hides the problem: mean CPU across 40 instances stays comfortable while four instances are pinned and dropping requests.
- • Feedback loop: latency-based scaling adds instances, the new instances warm caches by hammering the database, database latency rises, the scaler adds more.
- • Scale-in on a metric that fell for the wrong reason: requests are failing fast, concurrency drops, the scaler removes capacity mid-incident.
- • A stale target inherited from a previous architecture, defended because "it has always been 70".
- • Signal quality degrades with fleet size: an average over 200 instances smooths away exactly the outliers you need to see. Prefer percentile or per-instance saturation.
- • Metric cardinality and publishing cost grow with the fleet; custom per-instance metrics at large scale become a real observability bill.
- • The scaling target itself must move as the workload changes — a per-instance concurrency limit derived at 4 vCPUs is wrong after a resize.
- • Multi-metric scalers become hard to reason about past three or four signals. If you cannot explain a decision, you cannot debug it at 3 a.m.
- • A scaling metric published by the application is an input that can move infrastructure. Treat the publishing path as privileged — an attacker who can inflate it can inflate your bill.
- • Request-rate-based scaling scales *with* an attack. Pair it with rate limiting at the edge, or the abuse is absorbed at your expense.
- • The metrics adapter needs read access to the metric source and write-adjacent influence over the scaling group; scope it narrowly (Least Privilege in Infrastructure).
- • Queue-depth scaling on a queue an external party can write to is an amplification path worth thinking about explicitly.
- • A target utilization is a direct cost dial: targeting 50% roughly doubles the fleet compared with targeting 90%, in exchange for latency headroom.
- • Custom metric publication and retention cost money at scale, and the scaling metric usually needs a shorter resolution than everything else.
- • The wrong signal costs twice: over-provisioning against a metric that moves for unrelated reasons, and outages against one that does not move at all.
- • Time-to-drain targets make the cost conversation concrete: "five minutes of backlog" and "one minute of backlog" have a comparable, defensible price difference.
- • The chosen signal, the scaler decision it produced, and the resulting instance count on one timeline. Debugging a scaler without all three is guesswork.
- • Per-instance saturation distribution, not the fleet average — the average is the metric that lies here.
- • Freshness of the scaling metric itself, alerted on. A metric that stops arriving is invisible unless you watch for absence.
- • Correlation between the scaling signal and user-visible latency. If they do not move together, you are scaling on the wrong thing and this chart proves it.
- • Scheduled scaling, when the demand pattern is predictable. No signal to choose, no lag, and far fewer ways to be wrong.
- • A fixed fleet with a concurrency limit and clean backpressure. Rejecting excess work quickly is often better than absorbing it slowly — see Serverless and Database Connections for why unlimited absorption ends badly.
- • A platform with built-in request-based scaling (managed container services, serverless) that removes the signal choice entirely, at the cost of control.
- • No autoscaling at all for a three-instance service. The blast radius of a bad signal exceeds anything you save.
- • The best signal usually requires application instrumentation; the worst one is free. That gap is why so many teams ship the wrong one.
- • Aggressive targets save money and remove the headroom you need to survive the provisioning delay.
- • Multi-metric scaling handles mixed workloads and makes every scaling decision harder to explain.
- • Latency-based scaling reflects user experience most directly and is the most prone to feedback loops.
What people believe, and what is true
CPU is the standard autoscaling metric.
It is the standard *available* metric. For most web services it is a proxy for a proxy, and it is flat during the exact incident you wanted to scale through.
A deep queue means we need more workers.
A deep but stable queue is fine. A shallow but growing one is the emergency. Scale on the rate, or on time-to-drain.
If we scale on latency we are scaling on what users feel.
True, and it is the signal most prone to feedback loops, because added capacity can raise latency before it lowers it.