Autoscaling a Backend
Choosing a signal that actually reflects load — and understanding why CPU is the wrong one for a service that spends its time waiting.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What should the number of instances be a function of, and how fast can that number honestly change?
Traffic triples every weekday morning and halves overnight. Capacity should follow it without anyone being paged and without paying for the peak all night.
Scale on CPU: target 70% average utilisation, minimum two instances, maximum twenty. That is the default everywhere, so it must be the sensible choice.
The service is I/O-bound: most of each request is spent waiting on the database and on a third-party API. CPU stays around 20% while every request queues, so the autoscaler sees an idle fleet and never scales up. The service is saturated and the metric says it is bored (Computing or Waiting? in Observability & Performance).
- The service is I/O-bound: most of each request is spent waiting on the database and on a third-party API. CPU stays around 20% while every request queues, so the autoscaler sees an idle fleet and never scales up. The service is saturated and the metric says it is bored (Computing or Waiting? in Observability & Performance).
- When it finally does scale, the new instances take two minutes to pull an image, start, build a pool and warm caches — by which time the morning spike has already caused the incident (Autoscaling Lag: The Gap Where the Outage Lives in Observability & Performance).
- Scale-up multiplies connections to the database, and the database hits its connection limit before the fleet reaches its maximum (Connection Pools).
- The metric is noisy, so the autoscaler adds and removes instances repeatedly, and every removal is a shutdown that drops requests because the service never implemented drain (Graceful Shutdown).
- The queue consumer fleet scales on CPU too, so a backlog of a million messages produces no scaling at all, because polling and waiting is not CPU work (Queue Backlog).
What is actually happening
- An autoscaler is a control loop: observe a metric, compare it to a target, change the replica count, wait, observe again. Every property of the system follows from that being a feedback loop with delay.
- The delay is the sum of several things and is usually underestimated: metric collection interval, the autoscaler's evaluation period, scheduling, image pull, process start, dependency connection, cache warm-up. Capacity arrives well after the decision to add it (Startup Time & Cold Start in Cloud & Infrastructure).
- The signal must correlate with the resource that saturates. CPU is a proxy for load only when CPU is what runs out. For a backend that spends most of each request waiting on I/O, the resource that runs out is *concurrency* — worker slots, pool connections, event-loop capacity — and CPU is nearly flat while all of them are exhausted.
- Better signals for a request-serving backend: in-flight requests per instance (concurrency), request rate against a known per-instance capacity, utilisation of the actual bounded resource (pool waiters, worker-pool saturation, event-loop lag).
- For asynchronous workers, the correct signal is almost always queue depth or queue age, because it measures the backlog directly rather than inferring it (Depth Is Not an Emergency; Age Is in Observability & Performance).
- Latency-based scaling is a tempting signal and an unstable one: latency rises for many reasons that more instances cannot fix — a slow dependency, a lock, a bad query — and scaling up in those cases adds load to the thing that is already slow (Cascading Failure).
- Scaling down is a termination event and uses the same graceful path as a deploy. An autoscaler on a service without drain is a machine for dropping requests all day (Graceful Shutdown).
- Every autoscaler has a stabilisation or cooldown behaviour, usually asymmetric: scale up quickly, scale down slowly. That asymmetry is deliberate — being briefly overprovisioned is much cheaper than being briefly under.
Why CPU lies about an I/O-bound backend
Take a request that spends 15 ms executing application code and 200 ms waiting — on the database, on a payment API, on a cache round trip. While it waits, it consumes a worker slot, a pool connection and a socket, and consumes essentially no CPU. Fill every worker slot in the process with requests like that and the instance is completely saturated: it cannot accept another request, and every new arrival queues.
What does CPU utilisation read? Roughly the fraction of time spent executing, which is small. The autoscaler compares that to a 70% target, concludes the fleet is over-provisioned, and may even scale down while requests queue. The metric is not wrong about CPU; it is measuring the wrong resource.
The fix is to scale on the resource that actually runs out. For a request-serving backend that is concurrency — how many requests are in flight per instance relative to how many it can hold. That number rises exactly when the service saturates, whether the work is CPU-bound or I/O-bound, which is what makes it the better default.
metrics:
- type: Resource
resource:
name: cpu
target: { averageUtilization: 70 }
# Request: 15ms CPU, 200ms waiting on the DB.
# 50 concurrent requests fill every worker slot.
#
# worker slots: 50/50 SATURATED
# pool waiters: 31 queueing
# p99 latency: rising steeply
# CPU utilisation: ~18% <-- the signal
#
# 18% is far below 70%.
# The autoscaler does nothing.
# It may scale DOWN.# In-flight requests per instance, target derived
# from a load test (where does p99 turn upward?).
metrics:
- type: Pods
pods:
metric: { name: http_requests_in_flight }
target:
type: AverageValue
averageValue: "35" # capacity 50, headroom for the ramp
# Same moment:
# in-flight per instance: 50 vs target 35
# -> scale up now
#
# Works for CPU-bound work too: a CPU-bound
# request holds its slot while it computes, so
# in-flight rises there as well.Concurrency measures occupancy of the resource that actually limits the service — the slot a request holds from arrival to response — regardless of whether that time is spent computing or waiting. CPU measures only one of the two. This is also why Little's Law is the right mental model: in-flight requests equal arrival rate times latency, so concurrency rises when either the traffic grows or the dependencies slow down, and both are cases where more instances buy time (Little's Law as Working Intuition in Observability & Performance).
Choosing the signal for the workload
The signal should be the thing that saturates, and it differs by workload type. A request-serving API, a queue consumer and a batch processor have three different answers, and using one answer for all three is how a fleet ends up scaling on something it cannot influence.
The final row is the trap: any signal that a scale-up cannot change is a signal that will drive the autoscaler to its maximum and stay there while the actual problem is untouched.
What runs out first in this workload?
when HTTP services, especially I/O-bound ones.
cost Needs a custom metric and a capacity number from a load test (Performance Testing a Backend).
when Uniform request cost and a known per-instance capacity.
cost Breaks when request cost is heterogeneous or dependencies slow down.
when Asynchronous workers and batch consumers.
cost Must be paired with graceful shutdown so scale-down does not lose claimed jobs (Worker Scaling).
when Genuinely CPU-bound work: encoding, compression, rendering, model inference.
cost Useless for I/O-bound services, and it is the default everywhere.
when Almost never — memory usually indicates a leak, not load.
cost Scaling on it hides the leak and multiplies it (Memory Leaks in Backend Services).
when The bounded resource is explicit and instrumented.
cost Very accurate, and the most instrumentation work.
when Rarely, and only with a saturation signal alongside it.
cost Rises for reasons instances cannot fix; scaling up then adds load to the slow dependency (Cascading Failure).
when Predictable daily or weekly patterns.
cost No reaction to the unexpected; use alongside a reactive signal, not instead of one.
The delay budget: capacity always arrives late
Autoscaling is a feedback loop with a lag, and the lag is the sum of several intervals that are each easy to overlook. Adding them up gives an honest answer to "how fast can we respond", and that number determines how much permanent headroom the service needs.
Most of this budget is application-controlled: image size, initialisation work, whether readiness waits for a warm cache. A team that halves its startup time has improved its autoscaling more than any tuning of thresholds could.
- 1Load rises
Requests arrive faster than the fleet can serve.
fails by Nothing yet — latency has already started rising.
- 2Metric collected
Scrape or aggregation interval elapses.
fails by A long interval or an averaging window that smooths the spike away.
- 3Autoscaler evaluates
Compares metric to target, computes desired count.
fails by Stabilisation window deliberately delaying the reaction.
- 4Instance requested
Scheduler places it; a node may need to be added first.
fails by No capacity available: cluster autoscaling adds its own delay.
- 5Image pulled
Registry fetch, unless cached on the node.
fails by Large images making this the dominant term (Containerizing a Backend).
- 6Process starts
Runtime boot, module load, config validation.
fails by Expensive startup work, blocking dependency checks.
- 7Dependencies connected
Pool built, caches primed, clients created.
fails by A thundering herd of new connections at the database (Thundering Herd in Concurrency & Parallelism).
- 8Readiness passes
Instance joins the load balancer.
fails by Passing readiness before it can actually serve, so it takes traffic cold (Health Checks: Startup, Readiness, Liveness).
- 9Warm
Caches populated, runtime optimised, latency normal.
fails by Full traffic share on a cold instance, raising the tail for everyone.
The total is your minimum reaction time. Traffic that rises faster than this is handled by headroom, queueing and load shedding — never by the autoscaler, which is still starting instances when the spike ends.
How to build it
Most important first.
- Pick the signal by asking what actually runs out first. If the answer is "connections to the database" or "worker slots", scale on concurrency or on saturation of that resource — not on CPU.
- For HTTP services, prefer concurrent in-flight requests per instance, with a target derived from a load test rather than guessed. It degrades correctly for both CPU-bound and I/O-bound work.
- For workers, scale on queue depth or the age of the oldest message, which measures the backlog you actually care about (Worker Scaling).
- Set the minimum from availability and warm-up needs, not from cost. Two is a floor for availability; a higher floor is often justified by the time it takes to add capacity.
- Set the maximum from what the downstream dependencies can absorb, not from what the platform will allow. The maximum instance count times the pool size must fit inside the database's connection budget (Connection Pools).
- Make instances start fast: small images, cheap initialisation, no long warm-up before readiness. Scale-up latency is mostly application-controlled (Containerizing a Backend).
- Implement graceful shutdown before enabling autoscaling. Scale-down happens far more often than deploys (Graceful Shutdown).
- Use scheduled scaling for predictable patterns — a weekday morning ramp is known in advance, and pre-scaling removes the reaction delay entirely.
- Add stabilisation windows to prevent flapping, and expect them to be asymmetric: react quickly upward, slowly downward.
- Combine autoscaling with load shedding. When scaling cannot keep up, rejecting excess work quickly is better than accepting it into an unbounded queue (Backpressure).
What can go wrong
- CPU-based scaling on an I/O-bound service — the single most common misconfiguration in this area. It looks correct, it is the platform default, and it never scales when it should.
- Scale-up that saturates a shared dependency, converting a capacity problem into a dependency outage that affects every service using it.
- Flapping: the fleet oscillates, every cycle terminates instances, and each termination is a small burst of errors.
- A maximum set too high, letting the autoscaler exhaust a database connection limit or a third-party rate limit (Rate Limiting).
- A maximum set too low, so the autoscaler quietly stops scaling and the service saturates with no alert, because the autoscaler itself is not failing.
- Scaling on a metric that a scale-up cannot influence — external API latency, lock contention, a slow query — so the loop adds instances forever with no effect (Why Is My API Slow?).
- Scale-down that terminates a worker holding an unacknowledged job, causing redelivery or loss (Job Idempotency).
- A scale-down decision can select an instance that has just been given work; the drain path must complete that work before exiting (Graceful Shutdown).
- Scale-up during a rolling deploy creates new instances on whichever version the deployment currently specifies, so the fleet can contain both versions in a proportion nobody chose (Rolling Deployments).
- Two scaling signals (an HPA and a queue-based scaler on the same deployment) can issue conflicting decisions and oscillate.
- Rapid scale-up can exceed a third-party API's rate limit and get the whole service blocked — a self-inflicted denial of service against a dependency you do not control (Rate Limiting).
- Each new instance obtains credentials at startup; prefer short-lived workload identity so a fleet that grows and shrinks all day does not spread long-lived keys (Secrets Are Not Configuration).
- An autoscaler is an amplifier for anything that generates load, including an attack. Rate limiting and load shedding must sit in front of it, or an attacker sets your capacity (Defence in Depth).
- "CPU is a good default signal." It is a good signal for CPU-bound work only. A backend waiting on a database is not busy by CPU, and will not scale up when it should — which is the single most important sentence in this lesson.
- "Autoscaling handles traffic spikes." It handles ramps. A spike faster than your scale-up latency is handled by headroom, queueing and load shedding, not by the autoscaler.
- "More instances means more capacity." Only if the bottleneck is in the instances. If it is a shared database or an external API, more instances make it worse (Cascading Failure).
- "Scale to zero saves money." It also guarantees that the next request pays a full cold start, which is a product decision rather than a cost one (Serverless Backends).
- "The autoscaler is working — replica count is changing." Changing is not the same as correct. Compare the count against a saturation metric you trust before believing it.
Operating it
- Plot replica count together with the scaling metric and with a saturation metric you trust. If the fleet grows while saturation stays high, the signal is not the constraint.
- Measure the full scale-up latency directly: from the metric crossing its target to a new instance serving its first request. This number is the honest limit on how fast you can respond (Autoscaling Lag: The Gap Where the Outage Lives in Observability & Performance).
- Track scale events per hour. A high rate means flapping, and each event has a cost in dropped requests and cold caches.
- Alert on sustained operation at the maximum replica count — the autoscaler is not failing, it has simply run out of room, and nothing else will tell you.
- For workers, alert on queue age rather than depth: a large but shrinking queue is fine, and a small but growing one is not (Depth Is Not an Emergency; Age Is in Observability & Performance).
- At larger fleets the per-instance overhead becomes the limit: each replica holds connections, warms its own cache and makes its own outbound calls, so scaling multiplies pressure on every dependency (Horizontal vs Vertical Scaling).
- At 100x, autoscaling and capacity planning merge: the autoscaler handles the shape of the day and a human decides the envelope it operates within (Capacity Planning: Traffic to Machines in Observability & Performance).
- Very fast scaling makes the dependencies the bottleneck, which is why the maximum should be derived from downstream capacity rather than from platform limits.
- Autoscaling trades steady-state cost for reaction delay. It is always behind, so the choice is how much headroom to keep permanently to cover the gap.
- A sensitive autoscaler responds quickly and flaps; a damped one is stable and late. There is no setting that is both.
- Better signals (concurrency, queue depth, pool saturation) require instrumentation and a metrics pipeline the autoscaler can read. CPU is available for free, which is exactly why it is the default and why it is so often wrong.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe control-loop model, the delay budget and the signal-choice reasoning apply to every autoscaler.
- CLOUD-SPECIFICAvailable metrics, stabilisation behaviour and scale-in protection differ. Kubernetes HPA scales on resource or custom metrics with configurable stabilisation windows; instance-group autoscalers often expose fewer signals; request-based container platforms scale on concurrency natively and need no metric pipeline at all.
- RUNTIME-SPECIFICWhat saturates first depends on the runtime: a Node service saturates the event loop (visible as loop lag, not CPU), a thread-per-request server saturates its thread pool, and a pre-fork Python fleet saturates worker slots. The right signal is whichever of those runs out (Backend Runtime Models).
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.