CPU Saturation: When Cores Become the Queue
Throughput stops rising, latency bends upward, and the run queue grows. Confirming CPU as the constraint takes three readings; the causes range from an O(n²) loop to logging in the hot path to a lock everything spins on.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The shape: a knee, not a cliff
CPU saturation does not announce itself. It shows up as a curve: as offered load rises, latency drifts up gently, then bends sharply upward at a point that depends on how much work each request needs. Past that knee, throughput has stopped increasing — the system is serving all it can — and every extra request lands in a queue. The p50 may still look tolerable while p99 has gone somewhere unacceptable, because the tail is where queue waits accumulate (Tail Latency: Why p50 Being Fine Does Not Help).
The reason latency bends rather than steps is Queueing: Why Systems Get Slow Before They Get Broken: as utilization approaches capacity, expected wait grows non-linearly. Practically, this means the last 15% of your capacity buys almost nothing usable, which is the whole argument for Headroom: The Capacity You Deliberately Do Not Use. It also means the knee moves whenever per-request CPU cost changes — ship a slightly more expensive serializer and the knee slides left into your normal traffic range.
What confirms saturation is the *pair* of readings. Throughput flat while load rises says "no more capacity". Run queue growing says "work is waiting for a core". Either one alone has an innocent explanation; together they are conclusive, and they hold regardless of what the utilization percentage reads.
Seven things that eat CPU, and how they differ
Once CPU is confirmed as the constraint, the next question is which kind of CPU work. The candidates behave differently under load and need different fixes, and a profile plus the user/system split separates them quickly (see Self Time, Total Time, and Where the CPU Went).
The one that catches teams by surprise is logging. A debug log line in a hot path is a format call, an allocation, a serialization and often a synchronized write — and at 20k requests per second it can outweigh the business logic it was added to explain. The same applies to metrics with high-cardinality labels, where every observation builds a new series key (Cardinality: The Label That Took Down Monitoring).
Lock spinning is the deceptive one: it burns user CPU while making no progress, so the profile shows a wide, busy frame that looks like real work. The tell is that throughput does not improve when you add cores — contention scales *against* you — and that thread count above a small number makes things worse rather than better.
| Consumer | How it shows up | What actually fixes it |
|---|---|---|
| Expensive algorithm (O(n²) over a growing collection) | Cost tracks *data size*, not request rate; one wide profile frame | Better algorithm or data structure — Algorithmic Cost in a Request Handler |
| Serialization / deserialization | Wide frames in JSON or protobuf codecs; cost tracks payload size | Smaller payloads, field selection, streaming — Payload Size: 20KB, 200KB, 5MB |
| Compression | CPU rises as bandwidth falls; a deliberate trade | Tune level or threshold; skip tiny and already-compressed bodies — Compression: Cheaper Bytes, Not Fewer |
| Logging in the hot path | System + user time, allocation churn, worsens exactly under load | Sample, downgrade level, defer formatting — The Log Bill and What It Is Buying |
| Busy loop / spin-wait | A core pinned at 100% with no throughput to show for it | Block properly instead of polling; fix the wait condition |
| GC / runtime overhead | CPU in runtime frames; pauses correlate with allocation rate | Reduce allocation rate — Garbage Collection: Pause, Throughput, Footprint — Pick Two, Allocation Rate Is a Cost Even Without a Leak |
| Lock spinning / contention | High user CPU, throughput *falls* as cores or threads are added | Shrink the critical section, shard the lock — Low CPU, High Latency: Lock Contention |
Confirming it is CPU before you act
The expensive mistake is treating "CPU is high" as a diagnosis. High CPU alongside high latency is a correlation, and the causal arrow runs both ways in practice: saturated CPU causes latency, but so does a slow dependency that leaves threads spinning through retry loops while the real constraint is elsewhere (Correlation Is Not the Root Cause).
Three readings settle it. First, does throughput rise when offered load rises? If yes, you have capacity and CPU is not the wall. Second, is anything runnable-but-not-running? If the run queue is empty, requests are waiting on something other than a core. Third, does a CPU profile account for the request time? If the wall-clock time of a request is 800ms but the profile only accounts for 40ms of on-CPU work, the other 760ms was spent waiting — which is the Computing or Waiting? test, and it points at I/O, locks or a dependency.
That third check is the one that saves the most wasted work. It is entirely normal to find a service at 90% CPU where CPU is not the user-visible constraint at all — the CPU is busy retrying against a dependency that is itself the bottleneck. Optimizing the retry loop makes the chart prettier and the users no happier.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Offered load vs served throughput | 4.2k rps offered / 2.9k served | Throughput has flattened while load rises: no capacity left to give. | smoking gun |
| Run queue (8 cores) | 19 runnable | Eleven tasks waiting for a core at any instant. Work is queueing on CPU. | smoking gun |
| On-CPU time vs wall time per request | 310 ms on-CPU / 340 ms wall | Requests are executing, not waiting. Genuinely CPU-bound. | smoking gun |
| user / system split | 81% user / 6% system | Application code, not kernel overhead. Go to the profile. | suspect |
| Dependency p99 | 11 ms (unchanged) | Downstream is healthy — rules out the "busy retrying" story. | normal |
| GC CPU fraction | 4% | Not a runtime overhead problem; the cost is in application frames. | normal |
Key points
- Saturation shows as a knee: throughput flattens while offered load rises and latency bends upward, with p99 degrading long before p50.
- Two readings confirm it — flat throughput under rising load, and a growing run queue. Utilization alone cannot distinguish 1.1× from 4× overload.
- The third reading routes the fix: if on-CPU time is far below wall time per request, the CPU is busy but not the constraint.
- The causes need different fixes: algorithmic cost, serialization, compression, hot-path logging, spin loops, GC overhead, lock contention.
- Lock contention is the deceptive one — it burns user CPU while making no progress, and adding cores makes throughput worse, not better.
Progressive depth
Overview
CPU saturation means requests are waiting for a core. You see it as latency that bends upward past a certain traffic level while throughput stops growing. The percentage on the CPU chart is not the signal — waiting is.
Practical
Confirm with three readings: throughput flat under rising load, run queue above core count, and on-CPU time close to wall-clock time per request. Then split user vs system time and take a profile. The profile names the frame; the mode split says whether to look at your code or at kernel overhead.
Advanced
The knee is a queueing phenomenon, so its position depends on both service time *and* variability. High variance in per-request cost pulls the knee left even at the same mean cost — a service with a rare expensive request type saturates earlier than its average suggests. This is why capacity planning on mean CPU per request under-provisions systematically, and why Headroom: The Capacity You Deliberately Do Not Use is expressed as a target utilization well below 100%.
Internals
Underneath, the run queue is the scheduler's runnable set, and each dispatch costs a context switch: register save, possible TLB and cache disruption, and a cold start on the new core's caches. That is why a heavily oversubscribed CPU loses throughput rather than merely sharing it — the overhead is real work that displaces useful work. Under cgroup CFS quotas the kernel additionally stops all threads in the group once the period budget is spent, which produces saturation symptoms in bursts aligned to the 100ms period boundary. See Context Switching and Scheduling Simulator: FCFS, Round Robin, Priority for the mechanism.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Traffic → workers: offered load rises past the point where per-request CPU cost times arrival rate exceeds available core-seconds.
- 2Workers → run queue: threads become runnable faster than the scheduler can dispatch them; runnable count climbs above core count.
- 3Run queue → latency: each request now pays queue wait plus service time; wait grows non-linearly with utilization (Queueing: Why Systems Get Slow Before They Get Broken).
- 4Latency → tail: queue wait lands disproportionately on the unlucky, so p99 blows out while p50 drifts (Tail Latency: Why p50 Being Fine Does Not Help).
- 5Latency → timeouts → retries: clients time out and retry, adding load to a system that has none to spare — the amplification described in Retry Storms: The Load You Generated Yourself.
- • "CPU is at 100%, so we need bigger instances" — first check whether the CPU is doing useful work or spinning on a lock or a retry loop.
- • "Throughput is flat, the load generator must be broken" — flat throughput under rising load is the definition of saturation, not a test artefact.
- • "p50 is fine so users are fine" — at the knee, the damage is entirely in the tail, and the tail is where sessions and fan-outs live.
- • "The profile shows the JSON encoder, so serialization is the problem" — it is *a* problem; check the ratio of on-CPU to wall time before concluding CPU is the constraint at all.
- • "Adding threads will increase throughput" — past saturation, more threads add context switching and contention and usually reduce it (Concurrency Limits: An Unbounded Server Is a Slower Server).
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Offered load vs served throughput on one chart — the divergence point is the knee.
- • Run queue length or PSI `cpu.some` stall fraction, sampled at least every 10s.
- • Per-request on-CPU time (from a profiler) against wall-clock duration (from the trace) — the ratio is the CPU-bound test.
- • CPU split user vs system, plus GC or runtime CPU fraction as a separate series.
- • p50 and p99 latency plotted against RPS, so the knee has a location rather than a vibe.
- • Reduce per-request CPU cost at the hottest frame the profile names — this moves the knee right for every request at once ([[cpu-profiling]], [[flame-graphs]]).
- • Remove hot-path logging and high-cardinality metric emission; these are pure overhead and frequently 10–30% of request CPU in ILLUSTRATIVE profiles.
- • Cap concurrency so that excess load is rejected fast rather than queued slowly — a bounded queue with a fast 429 beats an unbounded queue with 8-second waits ([[concurrency-limits]]).
- • Scale horizontally once per-request cost is defensible; scaling first just buys the same inefficiency in more places ([[capacity-vs-efficiency]]).
- • For contention specifically, shrink the critical section or shard the lock before adding any hardware.
- • Re-run the same load ramp and show the knee moved right: the RPS at which p99 crosses your objective should be measurably higher.
- • Confirm run queue at the previous problem load is now below core count.
- • Check that p99 improved, not only p50 — CPU fixes that only move the median usually removed a constant, not the constraint.
- • Verify throughput now rises with offered load through the old plateau point.
- • Concurrency caps convert slow requests into fast rejections — better for the system, visibly worse for the individual user who gets the 429.
- • Optimizing the hot path costs engineering time and often readability; scaling out costs money but ships this afternoon.
- • Continuous profiling adds a small constant overhead (typically low single-digit percent, RUNTIME-SPECIFIC) to every process, forever.
- • A CI benchmark on the hot path with a threshold, so a change that doubles per-request CPU fails before deploy (Regression or Tuesday? Telling a Real Change from Noise).
- • An SLO-based alert on latency rather than a CPU threshold — it fires for the thing users feel, whatever the cause (Burn-Rate Alerts: How Fast Is the Budget Going?).
- • A periodic load test at the documented peak multiple, so the knee location is a tracked number rather than folklore (Load Testing: What Question Is This Test Answering?).
- • Continuous profiling in production so the next hot frame is visible before it is an incident (Always-On Profiling, and the Diff That Finds Regressions).
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe distribution, signal readings and the 10–30% hot-path logging figure are teaching examples, not measurements. Real proportions depend entirely on what the service does per request.
- WORKLOAD-SPECIFICWhere the knee sits is a function of per-request CPU cost and arrival distribution. Two services on identical hardware can have knees an order of magnitude apart.
- ENVIRONMENT-SPECIFICRun-queue thresholds assume you know the core count available to the cgroup. Under CPU limits, throttling can produce saturation symptoms at low apparent utilization.