Worker Scaling
More workers help until the shared dependency saturates, at which point they make everything worse — including the requests still in the path.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
How many workers should there be, what signal decides that, and when does adding more stop helping?
The queue drains fast enough that nobody notices, at 3pm and at 3am, without the workers becoming the reason the database is slow.
Scale on CPU, like the API. When worker CPU is high, add workers; when it is low, remove them. It is the default autoscaling metric and it is already configured.
Most job workers are IO-bound: waiting on a database, a third-party API, object storage. CPU stays low while the queue grows, so a CPU-based policy never scales up during exactly the backlog it exists to prevent (Computing or Waiting?).
- Most job workers are IO-bound: waiting on a database, a third-party API, object storage. CPU stays low while the queue grows, so a CPU-based policy never scales up during exactly the backlog it exists to prevent (Computing or Waiting?).
- When it does scale, it scales past the real ceiling. Twice the workers means twice the concurrent database connections, and the pool or the database saturates — so the queue drains no faster and the API, sharing that database, gets slower (Connection Pool Exhaustion).
- Scaling on queue depth alone is nearly as bad. A large depth of fast jobs is fine; a small depth of slow jobs may already be violating expectations. Depth without duration is not a workload description (Depth Is Not an Emergency; Age Is).
- Scale-in kills workers mid-job. Without a drain period, in-flight messages return to the queue and are processed again, so aggressive scaling multiplies duplicate deliveries (Job Idempotency).
- Scaling has lag: detect, decide, provision, boot, warm, connect. During that window the backlog grows regardless, which is why autoscaling is not a substitute for backpressure (Autoscaling Lag: The Gap Where the Outage Lives).
What is actually happening
- Worker capacity is worker count multiplied by per-worker concurrency. Those are two separate knobs with different costs: more processes cost memory and connections; more concurrency per process costs contention within a process and, on a single-threaded runtime, nothing at all for IO-bound work.
- The useful scaling signal is oldest-message age or the derived time-to-drain, not depth and not CPU. Age is the thing a user experiences, and time-to-drain (depth divided by completion rate) states whether the current fleet is sufficient.
- The relationship between them is Little's law: the average number of items in the system equals the arrival rate times the average time in the system. It tells you that halving processing time and doubling worker count have the same effect on backlog — which is why profiling a job is often cheaper than scaling it (Little's Law as Working Intuition).
- Throughput is capped by the most constrained shared dependency, not by the worker count. Past that point additional workers add queueing at the dependency, and the queueing is shared with everything else that uses it (Connection Pools).
- Scale-in requires cooperation from the application: catch the termination signal, stop claiming new messages, finish or release the in-flight ones, then exit (Graceful Shutdown).
- Scaling to zero is possible for queue-driven work and introduces cold start into the latency of the first message after an idle period (Startup Time & Cold Start).
Pick the signal that describes the problem
Each of these signals is correct for something, and the failure is using one that answers a different question. Ask what you would tell a user: "your export will start within a minute" is a statement about age, not about CPU.
Time-to-drain is the most useful derived signal because it combines both halves — depth and completion rate — into the one number a policy can act on and an alert can threshold.
Which signal actually describes "we need more workers"?
when The default for almost every job queue. It is what a waiting user experiences.
cost A lagging indicator: by the time age is high, some delay has already been suffered.
when You want a leading indicator that accounts for both backlog size and current throughput.
cost Requires a stable completion-rate measurement; noisy when job durations are wildly mixed.
when Job durations are uniform, so depth is a proxy for time.
cost Meaningless across mixed durations, and it says nothing about whether workers are actually blocked (Queue Backlog).
when Distinguishing "not enough workers" from "workers are all waiting on something downstream".
cost Not a scaling trigger on its own — low utilisation with a growing queue means scaling up will not help (Twenty Workers, All Busy, Five Hundred Waiting).
when Genuinely CPU-bound jobs: image processing, compression, rendering.
cost Near-useless for IO-bound work, which is most job work (Computing or Waiting?).
when Steady arrival rate, predictable durations, no meaningful idle period.
cost Pays for peak all the time; buys no scale-in duplicates, no cold starts and no flapping.
Where adding workers stops helping
Every worker fleet has a ceiling that has nothing to do with the fleet. Finding it before an incident is a short exercise: list what every job touches, and write down the concurrency limit of each. The smallest number on that list is your maximum useful worker count.
The final column is the part that makes this urgent rather than academic. When workers push past a shared limit, the damage lands on the request path — the API and the workers are sharing a database, and the users who were not waiting for a background job are now waiting for everything.
| Shared constraint | What saturates | Symptom when workers exceed it | Who else is harmed |
|---|---|---|---|
| Database connection pool | Connections in use; wait queue grows | Jobs and requests both block acquiring a connection | Every API request using the same pool (Connection Pool Exhaustion) |
| Database capacity | CPU, IO, or lock contention on hot rows | Query latency rises for everyone; throughput flat | The entire application (Low CPU, High Latency: Lock Contention) |
| Third-party rate limit | The provider's quota | Workers spend their time receiving 429s and retrying | Any other feature calling the same provider (Rate Limiting) |
| Object storage / network bandwidth | Throughput per account or per instance | Transfers slow; job duration grows with fleet size | Uploads and downloads in the request path (Object Storage) |
| The broker itself | Connections or per-partition consumer limits | Consumers idle or rebalance repeatedly | Every consumer of that broker (Queue Semantics) |
| Downstream service capacity | Its own pool and CPU | It slows, then sheds, then fails | Everything depending on it (Cascading Failure) |
Scale-in is a shutdown problem
Scaling out is easy and scaling in is where the correctness bugs are. A terminated worker holding an in-flight message has not lost the work — the lease expires and it is redelivered — but it has guaranteed a duplicate, and on an aggressive policy that happens every few minutes.
The fix is the same drain sequence as a deploy: stop claiming, finish or explicitly release what is in flight, then exit. The platform must give you long enough to do it, which means the termination grace period has to exceed your p99 job duration.
process.on('SIGTERM', () => process.exit(0))
// The message in flight is never acked.
// It is redelivered after the lease expires and runs again.
// With a 30-second scale-in policy this happens continuously,
// so duplicate rate tracks the autoscaler, not the failure rate.let draining = false
process.on('SIGTERM', async () => {
draining = true // 1. claim nothing new
await Promise.race([ // 2. let in-flight work finish
inFlight.settled(),
delay(GRACE_MS - 5_000),
])
await releaseUnfinished() // 3. return what did not finish,
await broker.close() // so it is retried immediately
process.exit(0) // rather than after a full lease
})
while (!draining) {
const messages = await broker.claim({ max: 10 })
await Promise.all(messages.map(handle))
}Draining converts a guaranteed duplicate into a completed job. Explicitly releasing what could not finish is the second half: without it, an unfinished message waits out the whole lease before anyone can retry it, which is the difference between seconds and minutes of delay. The grace period must exceed p99 job duration, or the platform kills the process mid-drain and you are back to the left-hand column (Graceful Shutdown).
How to build it
Most important first.
- Scale on oldest-message age or time-to-drain. Both express the thing you actually care about; CPU and depth do not (Autoscaling Signals).
- Set a maximum worker count derived from the downstream limit — connection pool size, third-party rate limit, database capacity — not from what the platform will let you run (Resource Limits).
- Bound concurrency toward each dependency independently of worker count, so a scale-up cannot exceed a third party's quota (Rate Limiting).
- Tune per-worker concurrency before adding workers for IO-bound jobs. On an event-loop runtime, one process can hold many concurrent in-flight IO operations at almost no cost (Backend Runtime Models).
- Implement drain-on-shutdown and set the platform's termination grace period longer than your p99 job duration (Graceful Shutdown).
- Separate fleets by job class so a CPU-bound image job and an IO-bound webhook sender are not scaled by the same signal against the same limit (Bulkheads).
- Profile before scaling. A job that spends most of its time on an N+1 gets cheaper to run and faster to drain from one fix, permanently, at no infrastructure cost (The N+1 Query Problem).
- Set scale-in more conservatively than scale-out. Scaling up early costs money; scaling down early costs duplicate deliveries and a re-warming fleet.
What can go wrong
- Scaling on CPU for IO-bound workers, so the fleet never grows during the backlog it was meant to absorb.
- Scaling past the database's capacity, turning a worker backlog into a site-wide slowdown (Cascading Failure).
- Scale-in without drain, producing duplicate processing on every scale event.
- Flapping: aggressive thresholds causing continuous scale-out and scale-in, with cold starts and duplicate deliveries on every cycle.
- A maximum worker count set from platform quota rather than from downstream capacity, so the safety limit is not a safety limit.
- Scaling to zero on a queue with sporadic latency-sensitive work, so the first message after idle pays a cold start.
- One fleet handling both fast and slow job types, so the scaling signal describes neither (Queue Backlog).
- Workers scaled up while the actual constraint is a third-party rate limit, so extra workers spend their time receiving 429s (Retry Storms).
- Scale-in terminating a worker mid-job, so the message is redelivered and processed twice.
- Scale-out and scale-in triggering in the same window, so workers are created and destroyed continuously.
- Many workers starting at once and opening connections simultaneously, exhausting the pool at the moment capacity was supposed to increase (Connection Pool Exhaustion).
- Two workers claiming the same message when a lease expires under load, made more likely by higher concurrency (Queue Semantics).
- Every additional worker is another process holding production credentials. Autoscaling multiplies credential exposure, which is an argument for short-lived workload identity rather than static keys (Roles vs Static Keys).
- Per-tenant fairness does not come from scaling. Without per-tenant limits, one tenant's bulk work consumes the whole fleet regardless of its size (Multi-Tenancy).
- Scaling policy driven by an attacker-controllable signal is a cost-amplification attack: enqueue cheaply, force expensive scale-out (Rate Limiting).
- A worker fleet is a second deployment surface with its own image, dependencies and supply chain. It is easy to forget when patching, because it has no public endpoint (Dependency Security).
- "More workers means faster." Only up to the shared constraint. Past it, more workers add queueing at the database and slow down the request path too (Cascading Failure).
- "CPU is the standard autoscaling metric." It is the standard metric for CPU-bound work. For an IO-bound worker it is close to a constant, and constants make poor signals (Computing or Waiting?).
- "Queue depth is the right signal." Depth without processing rate says nothing about delay. A thousand 10ms jobs and ten 10-minute jobs are very different situations with similar depths.
- "Autoscaling handles spikes." It handles sustained changes with a lag. Instantaneous spikes are absorbed by the queue and by backpressure, not by provisioning (Backpressure).
- "Workers are stateless, so scale-in is safe." Workers hold in-flight jobs, which is state with a lease attached. Killing one without draining guarantees redelivery (Graceful Shutdown).
Operating it
- Oldest-message age, per queue, as the primary signal — for alerting and for the scaling policy itself (Depth Is Not an Emergency; Age Is).
- Completion rate against arrival rate. Their difference predicts time-to-drain and is the earliest indicator that the fleet is undersized (Little's Law as Working Intuition).
- Worker utilisation: fraction of time actually processing versus idle-polling. Low utilisation with a growing queue means the constraint is downstream, not the fleet (Twenty Workers, All Busy, Five Hundred Waiting).
- Downstream saturation — pool wait time, dependency latency, third-party 429 rate — plotted alongside worker count. Where the first rises and throughput does not, you have found the ceiling (Connection Pool Saturation: Waiting in Front of an Idle Database).
- Duplicate-processing rate correlated with scale-in events, which is how you find a missing drain (Job Idempotency).
- Scale events as annotations on the queue charts, so cause and effect are visible ("What Changed?" — Deploy Markers and the Invisible Deploys).
- At 10x, the constraint moves from worker count to the database or the third party, and the interesting work becomes making each job cheaper rather than running more of them.
- At 10x, per-queue fleets stop being optional: one policy cannot serve a 50ms webhook sender and a 20-minute export.
- At 100x, scaling decisions become per-tenant and per-priority, and fairness matters more than aggregate throughput.
- Sometimes the right answer is a fixed fleet. A steady arrival rate with predictable job duration does not need autoscaling, and a fixed fleet has no scale-in duplicates, no cold starts and no flapping (Capacity Planning: Traffic to Machines).
- Scaling on age is the right signal and reacts later than a leading indicator would — by the time age is high, some delay has already happened.
- More workers drain faster and consume more connections, more memory and more of a shared dependency everyone else needs.
- Higher per-worker concurrency is cheap for IO-bound work and makes a single worker's failure lose more in-flight jobs at once.
- Scaling to zero saves money and puts a cold start in front of the first message after every idle period.
- A fixed fleet is predictable and simple, and it pays for peak capacity all the time.
- Separate fleets per job class give isolation and multiply the number of scaling policies to tune and alerts to maintain.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALCapacity equals workers times per-worker concurrency, bounded by the most constrained shared dependency. True everywhere.
- RUNTIME-SPECIFICPer-worker concurrency means different things. On an event loop, one process can hold hundreds of concurrent IO operations, so the first knob to turn is concurrency, not process count — but a single CPU-bound job blocks all of them (Blocking the Event Loop). On a thread-per-job model, concurrency costs a thread and its stack. In CPython, threads help IO-bound jobs and not CPU-bound ones, so CPU-bound work scales by process (Python Runtime Models).
- CLOUD-SPECIFICWhat you can scale on differs by platform. Kubernetes HPA scales on CPU and memory natively and needs an adapter to scale on a queue metric (Horizontal Pod Autoscaling — and Why New Capacity Is Always Late); event-driven autoscalers exist specifically to close that gap. Serverless consumers scale per message with concurrency controls and a very different cold-start and connection profile (Serverless and Database Connections). Termination grace periods, and whether the platform sends a signal your process can catch, also differ.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.