Twenty Workers, All Busy, Five Hundred Waiting
Every worker is occupied and the queue is 500 deep, so the obvious move is more workers. Whether that helps depends entirely on what the workers are busy *doing* — and if they are waiting on a shared dependency, adding workers makes things worse.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Busy is not the same as working
A worker pool reports two very different states with the same number. In one, twenty workers are each burning a core computing something expensive: the pool is genuinely the constraint, and adding workers adds throughput more or less linearly until some other resource runs out. In the other, twenty workers are each blocked on a database call, holding a connection and consuming almost no CPU: the pool is a queue in front of the real bottleneck, and adding workers adds contention rather than throughput.
The distinction is exactly Computing or Waiting? applied to a pool, and it is decidable in about a minute. Look at per-worker CPU utilization while workers report 100% busy. High CPU means CPU-bound: more workers (or more cores) helps. Low CPU means the workers are waiting, and the only question worth asking next is *what* they are waiting on — a database (Connection Pool Saturation: Waiting in Front of an Idle Database), an external API, a lock, or disk.
Getting this wrong is expensive in a specific way: adding workers against an I/O-bound pool usually makes latency worse rather than merely failing to help. Forty workers contending for twenty database connections means the wait moves from the queue into the connection pool, where it is harder to see, and the extra concurrency can push the shared dependency past its own knee.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| workers busy (pool A) | 20 / 20 | Fully occupied. On its own this says nothing about the constraint. | suspect |
| worker CPU (pool A) | 94% | Workers are computing, not waiting. The pool really is the bottleneck. | smoking gun |
| workers busy (pool B) | 20 / 20 | Identical reading to pool A — which is exactly why busy-ness cannot be the diagnostic. | suspect |
| worker CPU (pool B) | 7% | Workers are blocked, not busy. Something downstream is the constraint. | smoking gun |
| DB pool wait (pool B) | 340 ms p50 | Workers spend most of each job waiting for a database connection. | smoking gun |
| DB CPU (pool B) | 38% | The database is not overloaded either — the constraint is the connection pool size, not the database. | normal |
Where the latency actually accrues
Job latency as users experience it is queue wait plus service time, and the two respond to completely different interventions. Adding workers reduces queue wait and does nothing to service time. Making the job faster reduces service time and — because faster service raises μ — also reduces queue wait. That asymmetry is why "make the work cheaper" is a more durable fix than "add workers", even though it is slower to deliver.
Splitting the two on a waterfall makes the argument concrete for a room full of people. If a job takes 2.4 seconds end to end, of which 2.1s was spent sitting in the queue and 0.3s doing work, more workers is exactly right. If 0.2s was queue wait and 2.2s was a downstream call, adding workers changes 8% of the number and multiplies pressure on whatever is producing the other 92%.
This is also the practical use of Little's Law as Working Intuition. Concurrency ≈ throughput × latency, so a pool of 20 workers serving jobs that take 0.3s can sustain roughly 20 ÷ 0.3 ≈ 66 jobs/s. If you need 200/s, you need about 60 workers *at that service time* — or you need to get service time down, which reduces the worker count required proportionally. The law turns "how many workers?" from a guess into a calculation with stated assumptions.
Sizing the pool on purpose
The default pool size in most frameworks is a guess someone made about a different workload. Sizing it deliberately takes two numbers you already have: target throughput and measured service time. Little's Law gives the concurrency needed; the constraint check tells you whether that concurrency is achievable.
For CPU-bound work the ceiling is cores — running 200 CPU-bound workers on 8 cores does not give 200-way parallelism, it gives 8-way parallelism plus a great deal of Context Switching overhead and much worse tail latency. For I/O-bound work the ceiling is whatever shared resource the workers wait on: if the database pool has 20 connections, worker concurrency above 20 simply relocates the queue.
The honest conclusion is often that the pool is already the right size and the answer lies downstream. That is an unsatisfying incident update and a correct one — and it is why the sequence matters: check CPU, check the downstream wait, compute the required concurrency, *then* decide whether the number of workers is the thing to change.
1# Measured, not assumed:2service_time_s = 0.30 # p50 job duration, excluding queue wait3target_rate = 200 # jobs/s we need to sustain4cpu_fraction = 0.07 # per-worker CPU while "busy" -> I/O bound5 6# Little's Law: concurrency needed to sustain the rate7workers_needed = target_rate * service_time_s # 200 * 0.30 = 608 9# Now the constraint check — is 60 achievable?10if cpu_fraction > 0.7:11 # CPU-bound: ceiling is cores. More workers than cores buys12 # context switches, not throughput.13 ceiling = cores14else:15 # I/O-bound: ceiling is the shared resource being waited on.16 ceiling = db_pool_size # e.g. 2017 18if workers_needed > ceiling:19 # Raising worker count just relocates the queue.20 # Raise the ceiling, or cut service_time_s instead:21 # service_time 0.30 -> 0.10 => workers_needed 60 -> 2022 passKey points
- Worker busy-ness is identical for a CPU-bound pool and a pool blocked on a dependency — per-worker CPU is what tells them apart.
- Adding workers to an I/O-bound pool relocates the queue into a less visible place and increases pressure on the shared dependency.
- Job latency is queue wait plus service time; more workers only touches the first, while cheaper work reduces both.
- Little's Law turns pool sizing into a calculation: workers ≈ target throughput × service time, subject to a ceiling.
- The ceiling is cores for CPU-bound work and the shared downstream resource for I/O-bound work — exceeding it buys overhead, not throughput.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Queue → workers: all 20 workers report busy and 500 jobs are waiting, which the autoscaler reads as a capacity shortfall.
- 2Workers → CPU: per-worker CPU is 7%, so the workers are blocked rather than computing — the pool is not doing the work it appears to be doing.
- 3Workers → database pool: connection acquisition p50 is 340ms out of a 300ms nominal job, meaning most of each job is spent waiting for a connection.
- 4Database pool → database: DB CPU sits at 38%, so the database itself has headroom; the constraint is the pool size, not the engine.
- 5Scaling attempt → outcome: doubling workers to 40 leaves throughput nearly unchanged and raises connection wait to 780ms — the queue moved rather than shrank.
- • "All workers busy plus a deep queue means we need more workers" — true only when per-worker CPU is high.
- • "DB CPU is 38%, so the database is fine and not involved" — the database engine is fine; the connection pool in front of it is the constraint.
- • "Throughput did not improve, so we did not add enough workers" — sublinear response to added workers is the signature that the workers were never the bottleneck.
- • "Job duration went up, jobs got slower" — check the split; queue wait can grow while service time is flat, and they have different fixes.
- • "More concurrency is always more throughput" — past the ceiling it is more context switching, more contention, and a worse tail.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Per-worker CPU utilization *while* workers report busy — the single discriminator between the two failure modes.
- • Queue wait and service time as separate metrics, never summed into one "job duration" number.
- • Downstream wait time inside the job: connection-pool acquisition time, external call duration, lock wait.
- • Worker count, target throughput and measured p50 service time, so the Little's Law calculation can be done from the dashboard.
- • Saturation of the shared dependency (connection pool utilization, downstream concurrency limit) beside worker saturation.
- • Identify the actual constraint first: per-worker CPU, then downstream wait, before changing any pool size.
- • If I/O-bound on a connection pool, raise the *pool* (and verify the database can take the extra concurrency) rather than the worker count.
- • Reduce service time — batch downstream calls, remove per-job N+1 queries, cache the repeated lookup — which lowers required concurrency proportionally.
- • Size the worker pool from Little's Law against measured service time, and cap it at the ceiling implied by cores or the shared resource.
- • If CPU-bound, add cores or workers up to the core count, and treat further increases as tail-latency risk rather than throughput gain.
- • Configure the autoscaler on a signal that reflects the real constraint — queue age or downstream saturation — rather than on worker busy-ness alone.
- • After the change, confirm throughput rose approximately in proportion to the added capacity; sublinear gain means the constraint is still elsewhere.
- • Confirm queue wait fell while service time stayed flat — that is the specific signature of correctly-applied added capacity.
- • Check the downstream resource did not degrade: connection wait, dependency latency and error rate should be flat or better, not worse.
- • Re-run the Little's Law calculation with post-change numbers and confirm observed concurrency matches predicted concurrency.
- • Raising the connection pool moves concurrency onto the database, which has its own knee — this can trade a visible queue for an invisible one.
- • Reducing service time is the most durable fix and the slowest to deliver, usually requiring code changes and their attendant risk.
- • Capping worker concurrency deliberately means rejecting or delaying work under load, which is correct backpressure and still an availability trade someone must sign off on.
- • Per-worker CPU metrics add cardinality proportional to pool size, which matters on large fleets.
- • A dashboard panel pairing worker busy-ness with per-worker CPU, so the discriminator is always one glance away.
- • An autoscaling policy keyed on queue age or downstream saturation rather than worker busy-ness, reviewed when service time changes materially.
- • A load test that raises worker count and asserts throughput scales approximately linearly — a sublinear result is a failing test, not a curiosity.
- • Service time tracked as its own SLI, so a regression in job cost is visible before it shows up as a capacity request.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEPool sizes, CPU fractions and wait times are constructed to contrast the two cases. The discriminator (per-worker CPU while busy) and the Little's Law relationship generalize; the numbers do not.
- ESTIMATEDLittle's Law sizing assumes stable arrival and service rates. Bursty arrivals need headroom above the computed concurrency, which is why the result is a floor rather than a target.