The Thread Pool Server

The accept loop no longer spawns; it put()s the accepted socket on a work queue and returns to accept(). N workers, created at startup, loop forever: get() a connection, run the blocking handler from The Blocking Server, close, get() again. When all workers are busy, connections wait in the queue — accepted by the kernel, held by the application, served later. This is Java’s ThreadPoolExecutor, Python’s ThreadingHTTPServer with a pool, Apache’s worker MPM, and the connection-handling core of most database servers.

The queue is a Queue in the DSA sense and behaves exactly like one: FIFO, bounded or unbounded, with producer (accept loop) and consumers (workers). The synchronization underneath is a mutex plus a condition variable or a counting Semaphores and Condition Variables — the classic producer/consumer problem, which is why every OS course teaches it. In production the queue also serves as the single most useful metric the server has.

For CPU-bound handlers the answer is close to the number of cores: more threads than cores adds context switches without adding throughput (see Concurrency versus Parallelism). For I/O-bound handlers — each request waits on a database, a downstream HTTP call, a disk — a worker spends most of its life blocked, and N must be larger by roughly the ratio of total time to CPU time. A handler that computes for 2 ms and waits 40 ms keeps a core busy only 5% of the time; ~20 such threads per core keep the core busy. Formally, Little’s law: the number of in-flight requests equals arrival rate × time per request; N must cover that number or the queue grows.

Two things make the arithmetic honest. First, the downstream has its own limits: 500 workers each holding a database connection is 500 connections on the database, which is a Thread per Connection problem for the database (see the Database domain’s Scaling from One User to Millions). N is bounded above by what your dependencies tolerate, not just by your CPU. Second, blocked threads are not free: each holds a stack and a kernel entry, and hundreds of them waking on the same downstream response create a scheduler burst (the "thundering herd").

Runtimes that intercept blocking (Go, Java virtual threads) let N be "as many as are in flight" because a parked handler costs a few kB and no OS thread; the sizing question then moves to the downstream limits and to explicit semaphores. That is a runtime-specific shift, not a removal of the constraint.

When arrivals exceed what N workers can drain, the queue grows. That growth is the server telling you it is overloaded — earlier and more precisely than CPU, which may look moderate if workers are blocked on I/O. An unbounded queue hides the signal: connections are accepted and wait minutes, clients time out and retry (adding to the queue), memory grows with queued sockets and buffered requests, and latency degrades smoothly into uselessness with no error anywhere. See the OS challenge RSS climbs 200 MB an hour until the OOM killer visits.

A bounded queue forces a decision when full, and the decision is the design: (a) the accept loop blocks on put(), so new connections pile up in the kernel’s accept backlog and eventually get dropped — backpressure flows to the client as connect timeouts; (b) the server accepts and immediately responds 503/RST, so the client fails fast and can retry elsewhere; (c) the server sheds the *oldest* queued work, on the theory that its client has already given up. Option (b) is what load balancers want to see, because a fast error routes traffic to a healthier instance; (a) is what a batch system wants, because nothing is lost.

This is the same backpressure as rwnd → 0 in Flow Control: The Receive Window and as a full socket buffer in The Buffer Chain, one layer higher. A system in which every queue is bounded degrades predictably; a system with one unbounded queue fails at that queue, late.

A worker runs a blocking handler, so everything in The Blocking Server applies to it: a client that sends slowly, a client that stops reading a large response, a downstream that hangs, all hold the worker for as long as they take. With N = 32, thirty-two such requests — from one misbehaving client with 32 connections, or a downstream outage that turns every request into a 30-second wait — leave zero workers for everyone else. The pool is a blocking server with N heads.

The defences are timeouts at every blocking point: a receive timeout for the request, a send timeout for the response, a connect timeout and a read timeout on each downstream call, and a total deadline per request that is enforced regardless of which call is slow. A timeout converts an unbounded hold into a bounded one and, crucially, produces an error you can count. A pool whose timeouts are all "infinite" fails silently and completely; one with 5-second timeouts fails loudly at 5 seconds per request and recovers when the cause does.

The other defence is architectural: put an event-driven proxy in front (nginx, Envoy, a load balancer) that absorbs slow clients cheaply and hands the pool only complete requests over a fast local link. The proxy takes the slow-client problem because it is the design that solves it — see The Event-Driven Server and Forward and Reverse Proxies.