The question this answers
What is a thread actually doing while it waits 500 ms for a database, and what could it be doing instead?
One HTTP handler: parse the request (0.4 ms of CPU), query PostgreSQL (500 ms of waiting), serialise the rows (1.1 ms of CPU), respond.
Across concurrent copies of this handler: the database connection pool, the query planner cache on the server, and the process's file-descriptor table. The handler's own locals are private per request.
Every accepted request eventually gets exactly one response derived from its own query result — a suspended handler must resume with its own rows, never another request's, no matter how many are in flight.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
99.7% of the request is nothing happening
Take the arithmetic seriously. 1.5 ms of CPU and 500 ms of waiting means the core is doing useful work for three-tenths of one percent of the request. On a 3 GHz core, the 500 ms of waiting is on the order of a billion and a half clock cycles spent on a thread that is not runnable. That is not a performance problem to be tuned; it is an entire machine standing still.
The word "blocked" is precise and worth keeping. When the handler calls into the database driver, the thread enters a state where the scheduler will not give it a core until the socket becomes readable. Operating Systems covers the state machine in [[process-states]] and the four I/O shapes in [[io-models]]. What matters for us is that the *core* is free the entire time, and something has to be willing to use it.
Concurrency is exactly that willingness. Whether the mechanism is another OS thread the scheduler picks up, a coroutine the runtime resumes, or a callback the event loop invokes when the socket is ready, the shape is identical: at the moment this work starts waiting, different work gets the core.
What the box actually looks like when you get this wrong
The signature of a service that needs concurrency and does not have it is unmistakable once you have seen it: latency is terrible, the queue is deep, and every resource graph is flat. CPU is nearly idle, memory is fine, disk is fine, and the network is barely moving. Something is very slow and nothing is busy.
The read-out below is the shape. Note the two numbers that matter together: 8 worker threads and 512 requests in the accept queue. Each worker is blocked in recv() on a database socket. The box has 8 cores and is using a quarter of one. Adding cores changes nothing; adding memory changes nothing; the only lever is letting each worker have more than one request in flight.
Performance owns the diagnosis of this pattern in [[cpu-bound-vs-io-bound]] and [[queueing]], and Little's Law tells you exactly how many in-flight requests you need: throughput times latency. Our job here is the design consequence — the fix is a concurrency model, not a bigger machine.
$ top -b -n1 | head -4 %Cpu(s): 3.1 us, 0.9 sy, 0.0 ni, 95.4 id, 0.5 wa MiB Mem : 32014.0 total, 24880.1 free $ curl -s localhost:9100/metrics | grep -E 'worker|queue|latency' http_worker_threads_total 8 http_worker_threads_busy 8 # all 8 blocked in recv() http_accept_queue_depth 512 http_request_duration_p50_ms 31400 # 512 queued / 8 workers * 500ms http_request_duration_p99_ms 62800 db_pool_size 20 # 12 connections idle, unused $ Diagnosis: 8 cores, 3% utilised, 512 requests waiting. Not CPU. Not memory. Not the database — it answers in 500ms as designed. The ceiling is 8 concurrent requests, because the model is one per thread.
Throughput against in-flight requests
Once each worker can hold many requests in flight, throughput climbs roughly linearly with concurrency — because you are not adding work, you are only overlapping waiting that was already happening. That is the unusual, almost free-feeling part of I/O concurrency, and it is why an event loop can serve thousands of connections on one core.
It stops being free at the first resource that is actually finite. In this service that is the connection pool at 20: past about 20 concurrent queries, additional requests queue in front of the pool instead of in front of the accept socket, and you have moved the queue rather than removed it. Past that, the database itself becomes the ceiling. See [[bounding-concurrency]] — the point of a limit is to choose where the queue lives.
The curve is modelled, not measured, and the shape is the lesson: linear while you are only overlapping waiting, flat the moment something real is saturated, and — beyond the flat part, which this curve does not show — declining, because each additional in-flight request still costs memory and scheduling.
Key points
- Concurrency exists because waiting is not work: a blocked thread holds an execution context and uses no core.
- A handler that is 1.5 ms of CPU and 500 ms of I/O uses a core for 0.3% of its lifetime.
- The fix for a waiting-bound service is more in-flight requests per execution context, not more cores.
- The signature is unmistakable: high latency, deep queue, every resource graph flat.
- Throughput climbs almost linearly with in-flight requests until the first genuinely finite resource — usually a connection pool or an upstream rate limit.
- Concurrency does not make any single request faster. It makes the other requests stop waiting for it.
- Unbounded concurrency does not remove the queue; it moves it somewhere with worse visibility. Bound it deliberately.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • The handler issues a query and calls into the driver, which performs a blocking read on a socket.
- • The thread transitions out of runnable; the scheduler removes it from the run queue and picks something else. See
[[process-states]]. - • Under a concurrency model, "something else" is another request rather than nothing: another thread from the pool, or — with async I/O — the same thread resuming a different task.
- • The kernel notes readability on the socket via the readiness mechanism the runtime registered. See
[[io-models]]and[[async-io]]. - • The suspended handler becomes runnable again with its own saved locals intact, serialises its own rows, and responds.
- • R1 issues its query and suspends; R2 runs, issues, suspends; R3 runs, issues, suspends. All three wait simultaneously and the core is free — the intended schedule.
- • R1 issues, blocks; R1's rows return while R2 is mid-serialisation; R1 stays ready until R2 yields. Readiness is not resumption, and the gap between them is queueing delay hiding inside "database latency".
- • R1 checks out pool connection 7, suspends across the await, and R2 also gets connection 7 because the pool marked it free too early — R2 receives R1's result set. The invariant "a resumed handler sees its own rows" is what pool bookkeeping actually protects.
- • Handler acquires a mutex over a shared cache, then awaits the database while holding it; every other request on the loop blocks for 500 ms. Holding a lock across an await turns a concurrency win into a global stall. See
[[lock-scope]].
- • Concurrency guarantees a waiting task does not occupy an execution context — that is the entire promise, and it is a big one.
- • It does not guarantee lower latency for any individual request. R1 still takes 500 ms; it simply stops being the reason R2 takes 1000 ms.
- • It does not guarantee your code runs on more than one core. See
[[async-is-not-parallel]]. - • It does not guarantee the downstream can absorb the extra in-flight work. Removing your own bottleneck reliably exposes somebody else's.
- • It does not guarantee ordering: with 300 requests in flight, responses complete in whatever order the database and the scheduler produce.
- • The connection pool is the first real contention point, and the wait shows up as latency attributed to the database rather than to the pool. See
[[connection-pool-saturation]]. - • The database itself contends next: 300 concurrent queries against 8 database cores is a queue on the other side of the socket.
- • The event loop contends with itself if any handler computes: 40 ms of JSON serialisation on the loop delays every ready task behind it.
- • File descriptors and ephemeral ports are hard ceilings that arrive without warning and present as connection errors, not as slowness.
- • Unbounded in-flight requests: memory grows with the queue, and the process is OOM-killed while every latency graph still looks like "slow", not "broken". See
[[unbounded-concurrency]]. - • Pool exhaustion: every request waits on a connection, so p99 becomes pool-wait time and the database looks idle throughout.
- • Thundering herd on resume: 400 suspended tasks become runnable in the same millisecond when a slow dependency recovers, and the recovery itself causes the next outage. See
[[thundering-herd]]. - • Blocking call smuggled into an async path — a synchronous DNS lookup, a
readFileSync, a compression call — which reintroduces the original problem for every task on the loop. - • Timeouts absent: a hung upstream converts "concurrency" into "unbounded accumulation of tasks that will never finish". See
[[timeouts]].
- • Any service where per-request CPU is small and per-request waiting is large: API gateways, BFFs, proxies, webhook receivers, anything that mostly calls other things.
- • Fan-out within one request: five upstream calls that do not depend on each other take the time of the slowest rather than the sum. See
[[fan-out-fan-in]]. - • Long-lived mostly-idle connections — websockets, SSE, long polling — where thread-per-connection would spend gigabytes on stacks for sockets that say nothing.
- • CPU-bound handlers: there is no waiting to overlap, so you have added suspension points and scheduling for nothing.
- • When the downstream is the bottleneck: raising your concurrency raises its queue, and the extra load can push a struggling dependency into collapse.
- • When the operation must be serialised for correctness — sequential writes to one aggregate — in which case concurrency buys you a race to solve rather than latency to save.
- • CPU time over wall time per request. Below roughly 0.1 says the request is dominated by waiting and concurrency is the lever.
- • In-flight request gauge alongside throughput. If throughput is flat while in-flight climbs, you have found the finite resource; the gauge tells you where the queue moved.
- • Pool wait time as a distinct metric from query time. Conflating them is the single most common misdiagnosis in I/O-bound services.
- • Event-loop lag or scheduler run-queue delay — the gap between a task becoming ready and actually running is invisible in application timings.
- • Little's Law as a sanity check: required in-flight requests equal target throughput times observed latency. 200 rps at 500 ms needs 100 in flight, and 8 threads cannot supply it.
- • Async colours the call graph. One
awaitdeep in a helper propagates outwards through every caller, and the retrofit is rarely small. - • Suspension points are interleaving points. Every
awaitis a place another task may mutate the shared state you read before it. See[[tasks-vs-threads]]. - • Backpressure becomes mandatory rather than optional: without a bound, the queue simply moves into your process's heap.
- • Debugging is harder because the stack no longer contains the code that started the operation, and correlating a resumed task with its origin needs deliberate context propagation.
- • Make the wait shorter. A missing index that turns 500 ms into 5 ms removes the need for any of this and is nearly always the better first move.
- • Remove the wait entirely with a cache, when the data tolerates staleness — the fastest query is the one that does not happen.
- • Batch: one query for 50 rows instead of 50 queries for one row. See
[[batch-apis]]; N+1 removal beats concurrency and adds no failure modes. - • More processes: 8 single-threaded workers behind a load balancer give 8 concurrent requests with zero shared memory. Crude, effective, and sometimes exactly right.
Three I/O calls, one thread
sequential const a = await getProfile(); const b = await getFlags(); const c = await getOrders() concurrent const [a, b, c] = await Promise.all([getProfile(), getFlags(), getOrders()]) wall clock 261 ms → 127 ms (2.06× less waiting) CPU used 21 ms → 21 ms (identical — no extra core was touched)
Why is 8 cores only 4.5×?
| workers | ideal | Amdahl only | realistic | limited by |
|---|---|---|---|---|
| 1 | 1.0× | 1.00× | 1.00× | none |
| 2 | 2.0× | 1.90× | 1.85× | serial |
| 4 | 4.0× | 3.48× | 3.19× | serial |
| 6 | 6.0× | 4.80× | 4.17× | serial |
| 8 | 8.0× | 5.93× | 4.17× | bandwidth |
| 10 | 10.0× | 6.90× | 4.17× | bandwidth |
| 12 | 12.0× | 7.74× | 4.17× | bandwidth |
| 14 | 14.0× | 8.48× | 4.17× | bandwidth |
| 16 | 16.0× | 9.14× | 4.17× | bandwidth |
Bounding concurrency with permits
| permits | goodput | mean latency | timeouts | failed of 10K |
|---|---|---|---|---|
| 1 | 25/s | 43 ms | 0.00% | 0 |
| 5 | 125/s | 43 ms | 0.00% | 0 |
| 10 | 250/s | 43 ms | 0.00% | 0 |
| 25 | 625/s | 43 ms | 0.00% | 0 |
| 50 | 1000/s | 53 ms | 0.00% | 0 |
| 100 | 1000/s | 103 ms | 0.00% | 0 |
| 200 | 1000/s | 203 ms | 0.00% | 0 |
| 350 | 1000/s | 353 ms | 0.00% | 0 |
| 500 | 0/s | 503 ms | 100.0% | 10K |
What people believe, and what is true
Concurrency makes each request faster.
Each request still takes 500 ms. What changes is that requests 2 through 500 no longer queue behind it. Concurrency buys throughput and tail latency, never single-request latency.
The database is slow, so we need a bigger database.
The database answered in 500 ms as designed. The service was capped at 8 concurrent queries and 512 requests were queued in front of it. The database was 40% idle.
More concurrency is always better for I/O work.
It is better until the first finite resource, then it only moves the queue into a place with less visibility and more memory cost.
Go deeper
Overview
A thread waiting on the network is holding a seat and eating nothing. Concurrency gives the seat to someone else until the food arrives.
Practical
Compute CPU-over-wall for the handler and in-flight-requests from Little's Law. If CPU-over-wall is tiny and your worker count is below the Little's Law figure, concurrency is the fix and the size of the fix is calculable.
Advanced
Concurrency does not remove queueing; it relocates it. Each relocation should be deliberate, bounded and instrumented, because a queue you did not choose is a queue in your heap with no metric on it.