The question this answers
A service got slower with no restarts and no CPU spike. What is actually happening, and how would I prove it?
The checkout API must hold its latency budget under burst. When it cannot, the team needs to distinguish "the database is slow", "we are being killed for memory" and "we are being held at an artificial ceiling" — three problems that present very differently and are fixed in opposite directions.
Two named, measurable enforcement behaviours with distinct signatures, so a latency incident can be attributed to a resource ceiling rather than guessed at.
The memory case: loud, obvious, misattributed anyway
When a container tries to allocate past its memory limit, the kernel has no gentle option. Memory cannot be handed out in smaller portions the way CPU time can, so the cgroup OOM killer terminates a process — usually the largest one in the cgroup, which is your application. Kubernetes records OOMKilled with exit code 137, restarts the container, and the restart count increments.
Every signal here is visible, and teams still misread it, because the *symptom* users experience is a burst of errors and dropped connections, not a memory alert. The application logs typically show nothing useful — the process did not fail, it was killed mid-instruction, so there is no stack trace and no shutdown log. Engineers reach for the application logs, find nothing, and start investigating the database.
The critical detail is that the limit you set is the cause. A container OOM-killed at 512 MiB is not out of memory in any absolute sense; the node may have 40 GiB free. It exceeded a number you wrote. That reframing decides the fix: measure the real working set, raise the limit if the usage is legitimate, or find the leak if it grows without bound. Raising the limit on a leaking process buys time proportional to the increase and nothing else.
MEMORY — everything is visible, and still gets blamed on the database
$ kubectl describe pod checkout-7f2a
Last State: Terminated Reason: OOMKilled Exit Code: 137
Restart Count: 12
Limits: memory: 512Mi
Events: Warning OOMKilling Memory cgroup out of memory: Killed process 1 (node)
# Application log at that timestamp: nothing. The process was killed mid-instruction.
# Node free memory at that timestamp: 41Gi. The node was fine. Your NUMBER was too low.
CPU — nothing is visible unless you know which metric to ask for
$ kubectl describe pod checkout-7f2a
State: Running
Restart Count: 12 <-- from the OOMs above; unrelated to what follows
Limits: cpu: 500m
Events: <none> <-- there is NO event for throttling. Ever.
container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total = 0.71
^ 71% of periods hit the quota
container_cpu_usage_seconds: 0.48 cores <-- "CPU is only at 48%, so CPU is fine"
p99 latency: 95ms -> 340ms <-- the actual user-visible damageThe signature confusion, and why the CPU graph lies
Here is the sentence worth memorizing, because it is the single most common misdiagnosis in Kubernetes operations: "the pod is not restarting and CPU is not at 100%, but latency tripled." That is CPU throttling, and the reason it is so consistently missed is that the metric everyone looks at genuinely does not show it.
The mechanism explains the paradox. A CPU limit is implemented as a quota per scheduling period — typically 100 milliseconds. A limit of 500m means the container may use 50 ms of CPU per 100 ms period. If the container consumes its quota in the first 30 ms, it is stopped for the remaining 70 ms, then released when the next period begins. Average that over a minute and the CPU usage graph reports about 0.5 cores — well under the limit, apparently healthy. But a request that arrived during a stopped window waited up to 70 ms before its thread was even scheduled, and that wait is pure latency with no CPU attributed to it.
The effect is worst for exactly the workloads people cap. A multi-threaded runtime consumes quota across all its threads simultaneously, so a container with eight threads and a 500m limit exhausts its 50 ms budget in roughly 6 ms of wall-clock time and then stalls for 94 ms. Garbage collection pauses, JIT compilation and connection-pool warm-up all burn quota in bursts, which is why the symptom often appears as periodic latency spikes rather than uniform slowness.
The proof is one metric: throttled periods over total periods. If that ratio is meaningfully above zero, the container is being held at its ceiling, and no amount of investigating the database will help. The fix is to raise or remove the CPU limit — see Requests vs Limits: Two Numbers That Do Different Jobs for why that field deserves skepticism in the first place.
| Observation | OOM kill | CPU throttling | Neither — look elsewhere |
|---|---|---|---|
| Restart count climbing | Yes, every time | No, never | Maybe — check the exit code |
| Exit code | 137 (SIGKILL) | No exit at all | 1, 143, or a real crash |
| Event recorded | OOMKilling on the pod and node | None. No event exists for throttling. | Varies |
| Application logs | Silent — killed mid-instruction | Normal, just slower | Usually the actual answer |
| CPU graph | Normal | Below the limit — this is the trap | Varies |
| Memory graph | Rises to the limit, then resets | Normal | Varies |
| Latency effect | Errors and dropped connections at the kill | p99 rises far more than p50 | p50 and p99 rise together |
| The metric that proves it | OOM kill events by container | throttled periods / total periods | Dependency latency, saturation, queue depth |
The diagnostic order that works
When latency degrades, check the ceilings before the dependencies. It takes two queries, it rules out an entire class of cause, and it prevents the very common outcome where a team spends a day tuning database indexes for a problem that was a number in a manifest.
The order below is deliberate. Restarts and OOM kills first, because they are the loudest and the fastest to confirm. Throttling second, because it is invisible everywhere else and the check is one ratio. Only then the dependencies — the database, the downstream API, the queue — which is where most engineers start and where the answer genuinely is most of the time, but only after the cheap checks have been done. The Observability & Performance domain teaches saturation analysis properly; this lesson is about the two saturation signals unique to a container ceiling.
1# 1. Was anything killed? Loud, fast, unambiguous.2kubectl get pods -n production \3 -o custom-columns='NAME:.metadata.name,RESTARTS:.status.containerStatuses[*].restartCount,REASON:.status.containerStatuses[*].lastState.terminated.reason'4# REASON=OOMKilled anywhere -> the memory limit is the story. Stop here.5 6# 2. Is anything being throttled? Invisible everywhere else.7# rate(container_cpu_cfs_throttled_periods_total[5m])8# / rate(container_cpu_cfs_periods_total[5m])9# > 0.05 -> the CPU limit is materially hurting this container10# > 0.25 -> this is almost certainly your latency incident11 12# 3. Only now: dependencies. Database p99, downstream API p99, queue depth.13# Most latency incidents live here — but steps 1 and 2 cost two minutes14# and rule out the two causes that leave no trace in the application logs.15 16# Alert on both, permanently:17# OOMKilled events by container -> a limit that is too low, or a leak18# throttled ratio > 0.25 for 10 minutes -> a CPU limit doing real damageKey points
- Exceeding a memory limit kills the container:
OOMKilled, exit code 137, restart count increments, application logs are silent. - Exceeding a CPU limit throttles the container: no restart, no error, no event — and an average CPU graph that looks fine.
- The signature of throttling is "not restarting, CPU not at 100%, latency tripled". Memorize it; it is the most common misdiagnosis in the platform.
- A CPU limit is a quota per ~100 ms period; a multi-threaded process can burn its whole quota in a few milliseconds and then stall for the rest of the period.
- One metric proves it: throttled periods divided by total periods. Check it before investigating the database.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Memory limits are enforced by the cgroup memory controller; an allocation that would exceed the limit triggers the OOM killer for that cgroup.
- • The killed process receives SIGKILL, so it cannot flush logs, finish requests or shut down cleanly — hence the silent application log.
- • CPU limits are enforced as a quota per scheduling period, commonly 100 ms; the container runs until its quota is spent, then is stopped until the next period.
- • Throttling is counted, not signalled: the kernel increments a throttled-period counter and tells the process nothing.
- • Because usage is averaged over the period, a heavily throttled container reports CPU usage at or below its limit while its threads are stalled for most of each period.
- • Alerts on OOM kills and on throttled ratio, both permanently, because neither appears in any default dashboard.
- • Memory limits derived from a measured working set including peak, revisited as the application changes.
- • A deliberate policy on CPU limits: when they are set, why, and who may remove one during an incident.
- • A runbook that puts the two ceiling checks before dependency investigation.
- • Distinguishing a leak from a workload that needs more — the shape of the memory trend, not the kill itself, tells you which.
- • A memory limit slightly below the real peak: the container is killed under load, in-flight requests are dropped, and the logs show nothing.
- • A leak with a generous limit: kills every few hours, each one dropping a burst of requests, availability metrics unaffected.
- • A CPU limit half the workload's burst need: p99 latency multiplies while p50 stays fine, and every dashboard says the service is healthy.
- • A liveness probe timing out because the container was throttled: the platform restarts a perfectly healthy application, converting slowness into an outage.
- • Throttling on the ingress controller or a sidecar, which degrades every request in the cluster and is attributed to the network.
- • Throttling worsens with concurrency: more threads consume the same quota faster, so the stall fraction grows as traffic grows.
- • Horizontal scaling can mask throttling — more replicas each throttled less — while leaving the per-pod ceiling in place and the cost multiplied.
- • Memory pressure tends to grow with connection and request concurrency, so a limit that held at 1,000 requests per second may not at 3,000.
- • Memory limits are the control that stops one container's exhaustion bug from evicting every other pod on the node.
- • Without them, a workload that can be driven to allocate — by a large upload or a crafted request — is a node-level denial-of-service vector.
- • Throttling is a fairness mechanism between tenants, and removing a CPU limit is a deliberate decision to allow one workload to consume idle node capacity.
- • Kill and throttle events are useful anomaly signals: a container suddenly hitting ceilings it never used to hit is worth investigating, not just resizing.
- • Throttling is invisible cost: capacity you are paying for on the node exists and the limit forbids using it.
- • Over-raising memory limits to stop kills inflates the reservation on every replica, which is one of the more expensive reflexes in a cluster.
- • Scaling out to work around throttling multiplies spend to solve a problem that a single manifest field caused.
- • OOM kill events by container — the definitive memory signal, and one that no default dashboard shows.
- • Throttled periods over total periods, per container — the definitive CPU-limit signal, and the one nobody looks at.
- • Memory working set trend over days, which separates a leak from a workload that simply needs more headroom.
- • p99 latency correlated against the throttled ratio, which is what turns a suspicion into a proof.
- • The signal that lies: average CPU utilization. It is *designed* to look normal while the container is stalled for most of every scheduling period.
- • Remove the CPU limit and keep the request — this is the fix for most throttling incidents, not a workaround.
- • A compute model without per-container ceilings: a VM sized for the workload has neither failure mode in this form.
- • Serverless, where memory is the only dial and CPU is allocated proportionally, removing throttling from your vocabulary.
- • Vertical autoscaling to set memory from observed usage, when the workload tolerates being restarted to resize.
- • Buys isolation between tenants on a shared node; costs two invisible failure modes that neither logs nor default dashboards reveal.
- • Buys leak containment through memory limits; costs restarts whenever the number is even slightly too low.
- • Buys fairness through CPU limits; costs burst capacity that was free, and latency damage that is almost never attributed correctly.
What people believe, and what is true
OOMKilled means the node ran out of memory.
Usually the node was fine. The container exceeded the limit you set, and the OOM killer acted on that cgroup only.
If CPU usage is below the limit, the container is not CPU-constrained.
Averaging hides throttling. A container stalled for 70 ms of every 100 ms period reports usage below its limit by construction.
Throttling shows up as an event or an error.
There is no event and no error. It is a counter the kernel increments and never tells the process about.