Queueing: Why Systems Get Slow Before They Get Broken

A queue exists wherever work can arrive faster than it can be served, which in a real system is nearly everywhere. Most of them are not called queues and are never instrumented: the kernel's accept backlog holding connections before your process calls accept, the runnable queue of threads waiting for a core, the pool waiter list, the disk request queue, the lock wait list, the socket send buffer. The application-level job queue that everyone thinks of is usually the *only* one with a dashboard.

This is why "the system is slow but nothing is at 100%" is such a common and confusing report. The waiting is real, but it is happening in a queue nobody exposed, so every instrumented number looks acceptable. The first move is therefore not to look for a busy resource but to look for a growing wait — depth, waiter count, or the gap between when work arrived and when it started.

Each of these queues has depth elsewhere in Engineer Atlas: the run queue and context switching in The Scheduling Problem, socket buffers in The Buffer Chain, lock waits in Locks and Deadlocks, the pool waiter list in Connection Pool Saturation: Waiting in Front of an Idle Database. What this lesson adds is the behaviour they all share once utilization gets high.

The defining property of a queue is that waiting grows non-linearly with utilization. In the simplest model — one server, random arrivals, exponentially distributed service times — the average time in system is W = S / (1 − ρ), where S is service time and ρ is utilization. That denominator is the whole story: at 50% utilization a request takes twice its service time; at 90% it takes ten times; at 99% it takes a hundred times.

The practical reading is that the last 10% of capacity costs more than the first 90%. Going from 50% to 60% utilization adds half a service time of waiting. Going from 90% to 95% doubles the total. This is why a system can absorb months of gradual traffic growth with no visible change and then degrade dramatically over a single week — the traffic did not change character, it crossed the knee.

The model's assumptions are wrong for most real systems: services have multiple workers, arrivals are bursty rather than Poisson, service times are not exponential, and queues are bounded. Multiple servers soften the curve; bursty arrivals and high service-time variance sharpen it. So treat the table below as the *shape* — non-linear, knee somewhere in the 70–90% region, catastrophic above it — rather than a lookup table for your service. The number that matters is where *your* curve bends, and only measurement gives you that (Load Testing: What Question Is This Test Answering?).

Queueing degradation is self-reinforcing, and the mechanism is worth knowing precisely because each step looks locally reasonable. Latency rises past a client timeout. The client retries — correct behaviour for a transient failure. The retry is a *new arrival*, so the arrival rate increases while the service rate has not changed. Utilization rises, the queue grows, latency rises further, more requests cross the timeout, more retries arrive. The system converges on a state where most of the work being served is work whose requester has already given up.

Two things make this worse. Requests that time out still consumed capacity — the work was done, the answer was discarded — so effective throughput falls exactly when demand is highest (Throughput: The Number That Means Nothing Without a Latency Bound). And synchronised retries arrive in waves, so the queue receives bursts rather than a smooth increase.

The interventions are all about breaking the loop rather than serving it faster: bound the queue so excess work is rejected quickly instead of accepted and delayed, add jitter so retries do not synchronise, cap retry attempts, and open a circuit when a dependency is clearly failing (Circuit Breaker, Backpressure). Shedding load feels wrong during an incident and is almost always the correct move: a fast rejection preserves capacity for the requests that can still be served in time.