CPU Throttling: The Latency With No Error
Over a CPU limit the container is descheduled until the next period rather than killed. Nothing errors, nothing restarts, and the tail latency gets worse for reasons nothing in the application explains.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why is my service slow when its CPU graph looks unremarkable and nothing in the logs is wrong?
CPU can be taken away and given back, so a container that exceeds its ceiling can be paused instead of terminated — which is gentler, and completely silent.
The service is slow, so it needs more CPU. Look at the CPU usage graph: it is well below the limit on average, so CPU is not the problem. Investigate the database instead.
Average CPU over a minute cannot show throttling that happens within scheduling periods that are a tiny fraction of a second long. A container can be throttled repeatedly while its averaged usage sits comfortably below the limit.
- Average CPU over a minute cannot show throttling that happens within scheduling periods that are a tiny fraction of a second long. A container can be throttled repeatedly while its averaged usage sits comfortably below the limit.
- Throttling produces no error, no exception, no restart and no log line. Every application-side signal says the service is healthy; only the latency distribution disagrees (Percentiles: Which One, and How Many Users Is That?).
- It hits multi-threaded and bursty workloads hardest. A runtime that starts a burst of parallel work consumes its quota for the period almost instantly and then waits, so short bursts of work turn into long wall-clock times.
- It hurts exactly the requests you care most about. The tail moves and the median often does not, so dashboards built on averages show nothing at all (The Average Was Fine and Users Were Not in the observability view).
- It compounds with garbage collection and with any runtime that sizes its thread pools from the machine's CPU count rather than from the cgroup quota.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A CPU limit is enforced as a quota per scheduling period. The container is allowed a fixed amount of CPU time in each period; when that is used up, its threads are descheduled until the next period starts.
- The pause is real time. The application is not slow — it is not running. From inside the process this is indistinguishable from a very unlucky scheduler, which is why profiling the application shows nothing wrong.
- A CPU request, by contrast, becomes the container's weight when several containers want CPU at once. Under contention the kernel divides time in proportion to weights, and no container is stopped (Requests and Limits).
- Parallelism multiplies quota consumption. Work spread across several threads consumes the quota several times faster in wall-clock terms, so the more parallel the workload the earlier in each period it stalls.
- The signal is a counter, not a gauge: the number of periods in which the container was throttled, and the total time it spent throttled. Both are exported per container by the node's container metrics.
- This is the compressible half of the resource story. Memory cannot be given back, so its limit kills; CPU can, so its limit stalls (OOMKilled: Over the Memory Limit).
Same ceiling, opposite consequences
The clearest way to hold this is next to its counterpart. Both are limits, both are enforced by the kernel, and they do fundamentally different things because of one property of the resource itself.
incompressible: cannot be taken back allocate -> reclaim fails -> SIGKILL container dies, exit 137 in-flight work lost restart, backoff, visible everywhere reason: OOMKilled <- unambiguous
compressible: can be taken back and returned quota used -> descheduled -> next period -> run container lives, does nothing for a while in-flight work is merely late no restart, no error, no log line reason: nothing at all <- you must go looking
The asymmetry is not a design inconsistency, it is a property of the resources. You can pause a process and resume it exactly where it was; you cannot un-allocate memory it is still using. That single difference explains why one failure announces itself and the other has to be hunted with a counter.
Why the average hides it
Throttling operates inside scheduling periods that are far shorter than any dashboard's resolution. A container can burn its allowance early in each period and sit stalled for the rest, repeatedly, while every graph you own reports moderate usage.
This sketch is a model, not a measurement: it shows the shape of what a per-minute average cannot represent.
one scheduling period, quota partly used (not throttled)
|#####.........................| runs, finishes early, idles
one scheduling period, quota exhausted (throttled)
|##############################| quota gone
|______________________________| descheduled until next period
^ the request is still open, and nothing anywhere records an error
what the per-minute dashboard shows for BOTH of the above:
"CPU usage: comfortably under the limit"
what separates them:
container_cpu_cfs_throttled_periods_total <- rises only in the second case
container_cpu_cfs_periods_total <- the denominatorWhat throttling looks like from the outside
Because there is no error, throttling is always diagnosed through something else — and that something else is usually blamed. These rows are the misattributions worth recognising.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Limit reached during a burst | p99 latency rises, p50 flat, error rate unchanged | Threads descheduled until the next period | Read the throttling counter; then decide on the limit (Requests and Limits) |
| Throttled during startup | New pods take far longer to become ready than expected | Startup is the burstiest phase most services have | Use a startup probe so the slow boot is not read as a failure (Probes: Readiness, Liveness and Startup) |
| Liveness probe times out | Restarts with no crash, no OOM and healthy logs | The probe handler could not be scheduled within its timeout | Check throttling before touching probe settings — restarting a throttled container achieves nothing |
| Autoscaler does not react | Latency climbing, replica count flat | CPU usage is capped by the limit, so the scaling signal is flat by construction | Scale on a signal that reflects the constraint, such as queue depth or concurrency (Choosing the Scaling Signal) |
| Runtime sized from the machine | A pod with a small quota starting many worker threads | Thread pools default to the node's CPU count, not the cgroup quota | Set parallelism explicitly from the limit (Build-Time and Runtime Configuration) |
| Dependency blamed | Client-side latency to a downstream service rises with no matching server-side rise | The caller is stalled, not the callee | Compare client and server timings; a gap with no server-side cause points back at the caller (Production Debugging) |
How to do it properly
Most important first.
- Check throttling counters before investigating anything else when latency has moved and errors have not. It is a cheap check and it is definitive.
- Watch the ratio of throttled periods to total periods per container, and treat any sustained non-zero value on a latency-sensitive service as a finding.
- Size CPU limits with burst in mind rather than from average usage, or decide deliberately not to set one (Requests and Limits).
- Configure runtime parallelism from the cgroup quota rather than from the node's CPU count — thread pools, worker counts and garbage-collector threads all default to the machine on most runtimes.
- Use the CPU request as the real lever for contention. If the workload is slow only on busy nodes, the request is the number that is wrong, not the limit.
- Correlate throttling with deploys. A release that adds parallelism or a background task can start throttling a workload whose limits never changed (Change Correlation).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Every replica shares the same limit, so all of them throttle under the same load — this is a service-wide latency regression, not a per-pod one. What contains it is that throttling degrades rather than fails: the service stays up and slow, so it is survivable for a long time, which is also why it goes undiagnosed for a long time.
What can go wrong
- Throttling misdiagnosed as a database or dependency problem, sending the investigation into the wrong system for hours.
- The limit raised until throttling stops, on a shared cluster, moving the contention onto neighbours who now have no protection.
- The limit removed entirely as a reflex, which is a defensible choice made for an indefensible reason and without telling anyone who shares the node.
- Autoscaling on CPU usage while the pods are throttled: usage is capped by the limit, so the scaling signal is flat exactly when the workload is starved (Choosing the Scaling Signal).
- Throttling accepted as normal because it has always been there, so a genuine regression is invisible against the background.
- A liveness probe timing out because the container was throttled, causing restarts that look like an application crash (Probes: Readiness, Liveness and Startup).
- "CPU usage is below the limit, so we are not throttled." Average usage over a minute says nothing about quota exhaustion within a period. The counter is the only answer.
- "Throttling means the node is out of CPU." The node can be idle. This is a per-container ceiling, not a machine-level shortage.
- "Throttling is like thermal throttling." Different mechanism entirely — one is a quota the platform enforces, the other is the hardware protecting itself (The First Ten Seconds Lie in the hardware view).
- "We should always set CPU limits." A defensible default in a shared cluster and a real latency cost in a single-team one. Decide it, do not inherit it.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Throttled periods and throttled time per container — near zero for latency-sensitive services, or at a level you have explicitly accepted.
- Tail latency before and after a limit change, compared against a control that did not change (Canary Analysis: Compared Against What?).
- A latency distribution whose tail moves while the median does not, alongside a flat error rate — the signature of a stall rather than a fault.
- Runtime parallelism settings that match the container's quota, verified from inside the running container rather than from the manifest.
- Changing a CPU limit replaces every pod, so a "quick tuning change" carries a full rollout's risk and timing (Apply Is Not Running).
- Reverting a raised limit reintroduces the throttling immediately and predictably, which at least makes the experiment cleanly reversible.
- Raising a limit can make the pod unschedulable if it also raises the request, leaving pods
Pendingwhile the old ones keep serving (The Scheduler, and Why a Pod Is Pending).
- Automate the signal. Throttling per container should be on the service's dashboard next to latency, so the correlation is visible without anyone knowing to look for it (Dashboards an Operator Can Act On).
- Automate the runtime configuration: deriving thread pool and GC thread counts from the cgroup quota is a platform concern that every service otherwise gets wrong individually (Service Templates).
- Keep the limit policy human, because the right answer depends on whether the cluster is shared and who bears the cost of a noisy neighbour.
- No CPU limit gives the best latency and the worst isolation. A tight limit gives predictable neighbours and a worse tail. There is no setting that gives both.
- Guaranteed QoS with request equal to limit makes behaviour predictable and reserves peak capacity continuously, which is expensive for spiky workloads (Overprovisioning).
- Watching throttling adds another per-container metric series to store and to pay for, on a dimension most teams never look at (Cardinality: The Label That Took Down Monitoring in the observability view).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICThe per-container CPU quota is Kubernetes exposing a cgroup mechanism. On a VM the equivalent ceiling is the instance's core count, which the runtime can see and size itself against, so the failure mode is ordinary CPU saturation with a visible usage graph rather than a silent stall. Burstable cloud instance types have a closer analogue — a credit balance that, once exhausted, caps the instance in a way that also does not appear as an error.
- SIMPLIFIEDDescribed without naming a period length or quota arithmetic, because the specific values depend on kernel and platform configuration and any figure quoted here would be a fabricated measurement. The mechanism — a fixed allowance per period, then a stall — is what transfers.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.