K8s RuntimeKUBERNETES-SPECIFICSIMPLIFIED

OOMKilled: Over the Memory Limit

Memory cannot be taken back, so the only enforcement available is termination. Over the limit, the kernel kills the process — it does not slow it down or warn it.

What actually happens How to do it properly Evidence and rollback

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

What exactly happens when a container reaches its memory limit, and why does it die rather than degrade?

The problem

A container that keeps allocating will eventually take memory that other workloads on the machine need, and memory in use cannot be reclaimed from a process that is still using it.

What teams do first

The container was killed, so the limit was too small. Raise it, redeploy, and the problem is solved.

How it breaks

If the workload has a leak, a bigger limit changes the interval between kills and nothing else — and it makes each kill more disruptive because more work is in flight (Memory Leaks: Growth That Does Not Come Back in the backend view).

How it breaks in production

If the workload has a leak, a bigger limit changes the interval between kills and nothing else — and it makes each kill more disruptive because more work is in flight (Memory Leaks: Growth That Does Not Come Back in the backend view).
The kill takes in-flight requests with it. There is no signal handler, no drain, no graceful shutdown: SIGKILL cannot be caught (Draining: Stopping Without Dropping).
Every replica of the service has the same limit and the same code, so traffic that pushes one over pushes them all over within a short window. Raising the limit raises it everywhere and delays a synchronised failure rather than preventing one.
Raising limits without raising requests changes the QoS class and the node's real oversubscription, so the next failure may be node-level eviction hitting a different workload entirely (Requests and Limits).
A managed runtime that sizes its heap from the machine rather than the cgroup will keep growing towards a ceiling that does not exist for it, and a larger container limit simply lets it grow further before it dies.

Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

The memory limit becomes a hard ceiling on the container's cgroup. When an allocation would exceed it, the kernel first tries to reclaim within that cgroup — dropping page cache, writing back dirty pages.
If reclaim cannot free enough, the kernel invokes the OOM killer scoped to that cgroup and terminates a process inside it, usually the largest. The container exits with code 137, which is 128 plus signal 9.
Kubernetes reports this as container state Terminated with reason OOMKilled. The pod is not deleted; the kubelet restarts the container according to the restart policy, with exponential backoff — the CrashLoopBackOff you see is the backoff, not a separate failure.
This is different from node-level memory pressure. When the node itself runs short, the kubelet evicts whole pods in QoS order — BestEffort first, then Burstable by how far over its request it is, then Guaranteed. The victim of a node-level eviction is frequently not the workload that caused the pressure.
Memory is incompressible. CPU can be handed back and forth at microsecond granularity, so the kernel can throttle. There is no equivalent for memory a process is actively using, which is why the two limits behave so differently (CPU Throttling: The Latency With No Error).
The container is killed without notice. There is no grace period, no preStop hook, and no opportunity to finish work — unlike every other way a pod goes away.

From allocation to exit code 137

The sequence matters because it explains why nothing in the application gets a chance to react, and why the pod object survives while the container does not.

What happens at the ceiling

1
Allocation requested
The process asks for memory that would take the cgroup over memory.max.
fails by Nothing yet — this is normal behaviour under growth.
evidence Working-set memory approaching the limit on the container's own metrics.
2
Reclaim attempted
The kernel tries to free memory within this cgroup: page cache, writeback of dirty pages.
fails by Anonymous memory in active use cannot be reclaimed, and there is normally no swap on a node.
evidence A brief rise in I/O and a stall in the process just before the kill.
3
OOM kill
The kernel terminates a process inside the cgroup with SIGKILL.
fails by Not catchable, not deferrable — in-flight work is lost with no shutdown path (Graceful Shutdown).
evidence Container exit code 137.
4
Kubelet records it
Sets container last state Terminated, reason OOMKilled.
fails by Lost if you only look at the current state after a restart — use the previous state.
evidence kubectl describe pod shows the reason under Last State.
5
Restart with backoff
The container is restarted per the pod's restart policy, with exponential backoff.
fails by Repeated quickly enough, this shows as CrashLoopBackOff and hides the original reason behind a generic-sounding state.
evidence Restart count climbing; previous-container logs available with --previous.
6
Capacity effect
While the container is down or backing off, its share of traffic moves to the remaining replicas.
fails by Those replicas now do more work, reach the ceiling sooner, and follow it down.
evidence Error rate and latency rising as ready replica count falls (Headroom).

Plateau or climb — the only question that matters first

SIMPLIFIEDDrawn as clean shapes. Real curves are noisier, and a runtime with a large garbage-collected heap can look like a climb while it is only deferring collection — which is why the window has to be long enough to include at least one full collection cycle.

Two completely different problems produce identical OOMKilled events, and the memory curve separates them in seconds. Undersizing plateaus above the limit; a leak climbs without settling. Everything you do next depends on which one you have.

Same event, opposite fixes

Undersized limit

memory
  |        ______________  <- steady plateau, above the limit
  |       /
  |  ____/
  +---------------------- time
  restarts track PEAK TRAFFIC
  same shape every day

  fix: raise the limit to the plateau plus margin

Leak

memory
  |          /|   /|   /|  <- climbs, killed, climbs again
  |        /  | /  | /  |
  |      /    |/   |/   |
  +---------------------- time
  restarts track UPTIME and WORK DONE
  interval shrinks as traffic grows

  fix: find the retention; a bigger limit only lengthens the sawtooth

Raising the limit on the left is correct and permanent. Raising it on the right is a mitigation that makes each eventual kill more expensive, because more in-flight work is lost per event and the failures stay synchronised across replicas. The curve tells you which one you are in before you have read a single line of application code.

Container OOM versus node pressure

These are two different mechanisms with two different victims, and confusing them sends the investigation to the wrong team. One is your workload hitting its own ceiling; the other is the node running short and the kubelet choosing who to sacrifice.

Which memory failure am I looking at?

Trigger	Symptom	Cause	Response
Container exceeds `limits.memory`	Last state `OOMKilled`, exit code 137, pod stays on its node	The cgroup ceiling for this container was reached	Look at this workload's memory curve; the node is probably fine
Node runs short of memory	Pods `Evicted` with a node-pressure message, rescheduled elsewhere	Total usage on the node exceeded what the kubelet will tolerate	Find which workload grew; the evicted pods are usually not it (Requests and Limits)
A container with no memory limit grows	Other pods evicted; the culprit keeps running	Nothing constrains it, and eviction picks by QoS rather than by fault	Set the limit on the unbounded workload; policy should have prevented this (Policy as Code)
Runtime heap sized from the machine	OOM kills well below what the app "should" use	The runtime never saw the cgroup limit and grew towards the node's memory	Set the runtime's ceiling explicitly from the container limit
A large request or upload buffered whole	Kills correlated with specific requests, not with uptime	Per-request peak memory far above the steady state	Stream instead of buffering, and bound the request size at the edge
Sidecars in the same pod	The wrong container is named in the kill	Limits are per container; the pod's total is the sum	Read which container was terminated before assuming it was the application

How to do it properly

Most important first.

Before changing any number, establish the shape: does memory usage plateau under steady traffic, or does it climb monotonically? A plateau above the limit is undersizing; a climb is a leak, and they need opposite responses.
Set memory request equal to memory limit so the workload is Guaranteed and is last in the eviction queue. Bursting memory buys nothing except a later, less predictable kill.
Configure the runtime's own memory ceiling from the container limit, not from the machine — most managed runtimes have an explicit setting or a container-aware mode for exactly this.
Alert on OOMKilled container terminations, not on memory usage percentage. The kill is unambiguous; the percentage is noise until it is not.
Keep the memory profile of a request path in mind for anything that buffers whole payloads. Large uploads, unbounded result sets and in-memory sorts are the usual sources of a spike that dwarfs the steady state (Pagination That Survives a Large Table in the backend view).
Treat a rise in the OOM rate as a release-correlated question first: what changed, and when did the rate change (Change Correlation).

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongEveryone

One testEveryone

What contains it

Nothing meaningful, when the cause is traffic-driven: every replica carries the same limit and the same code, so they reach the ceiling within a short window of each other and capacity falls while load is at its highest. A rolling restart is not containment here, because the trigger is load rather than a change. Containment only exists if the memory growth is release-correlated, in which case the canary sees it first (Canary Analysis: Compared Against What?).

What can go wrong

Failure modes, including of the mitigation

Limit raised, leak untouched: the same failure at a longer interval, now with more lost work per event.
Limit raised beyond what the node can honour for all its pods, converting container-level kills into node-level evictions that hit innocent workloads.
Restart backoff masking the problem: the container restarts, serves for a while, dies again, and the service looks merely "a bit flaky".
Capacity collapse under load: replicas die at peak, the survivors take more traffic, reach the ceiling sooner, and die faster (Cascading Failure: When the Response to Failure Causes More Failure in the backend view).
Alerting on restarts only, so an OOM kill on a workload with a long interval between kills is never investigated.
A liveness probe blamed for restarts that were actually OOM kills — the two look identical in a restart count and completely different in the last terminated state (Probes: Readiness, Liveness and Startup).

Misreads this invites

"OOMKilled means the node ran out of memory." Almost always it means this container reached its own limit, on a node with memory to spare. Node-level exhaustion produces evictions with different reasons.
"The application handled it badly." The application had no opportunity to handle anything. SIGKILL is not deliverable to the process.
"CrashLoopBackOff is the problem." That is the restart backoff. The problem is in the previous container's termination reason.
"Memory limits cause outages, so remove them." Removing the limit moves the failure from one container to the whole node, and the kubelet then chooses a victim by QoS rather than by fault.

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked

Container last-state reason is OOMKilled with exit code 137 — the definitive signal, and the only one that distinguishes this from every other restart cause.
Working-set memory per container against its limit, plotted over a window long enough to show whether it plateaus.
Restart count per container correlated with traffic, not with deploys — a leak tracks work done, an undersized limit tracks peak.
After a fix: the same peak traffic with no terminations, and a memory curve that flattens rather than one that has simply not reached the new ceiling yet.

How you get back

You cannot roll back a kill. The process is gone and its in-flight work is lost, which is why anything that must not be lost needs to be recoverable outside the process (Idempotency in Backends in the backend view).
If the OOM started with a release, rolling back the release is the fastest mitigation and the correct one — memory regressions are a normal release defect (Rollback: Only Useful If It Is Actually Safe).
If it started with traffic rather than a release, reducing load or adding replicas restores service while you investigate. Raising the limit is a mitigation too, as long as it is recorded as one rather than filed as a fix (Stop the Harm Before You Understand It).

What to automate, and what stays human

Automate detection: an alert keyed on OOMKilled terminations per workload, routed to the owning team, catches this before it becomes a capacity event (Alert on Symptoms, Not on Causes).
Automate the guardrail that every workload has a memory limit at all, so an unbounded container cannot quietly become the node's problem (Policy as Code).
Do not automate raising the limit in response to kills. That is an automated way to convert a leak into a node outage, and it is a decision that needs someone to look at the memory curve (The Automation Trap).

What this costs

A tight limit turns a leak into an early, contained, obvious failure. A generous limit gives the workload room and lets a leak grow until it can damage the node — you are choosing between a loud small failure and a quiet large one.
Guaranteed QoS reserves memory the workload usually is not using, which lowers cluster density and raises cost.
Restarting on OOM is genuinely useful for workloads with slow leaks and no fix in sight. It is also a way to run indefinitely without fixing anything (Production Anti-Patterns).

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

KUBERNETES-SPECIFICPer-container limits and the OOMKilled reason are Kubernetes surfacing a cgroup behaviour. On a VM there is no per-process ceiling by default: the machine fills up and the kernel's global OOM killer picks a victim by heuristic, which is often the largest process rather than the guilty one — so the same underlying bug takes out the whole instance instead of one container. A PaaS typically kills and restarts the instance with a memory-exceeded event. Serverless enforces a per-invocation ceiling and returns an error to the caller.
SIMPLIFIEDThe reclaim path is described as one step. In practice the kernel does several kinds of reclaim, and whether swap exists changes the behaviour — Kubernetes nodes conventionally run without swap, which is what makes the ceiling so abrupt.

Where the depth lives

This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.

Backendmemory-leaks resource-limits

Cloudoom-and-cpu-throttling

Observability & Performancememory-leaks leak-vs-cache-growth allocation-profiling

OS & Networkingmemory-pressure allocate-memory signals containers-and-the-os

Computer Architectureworking-set

This lesson

Reliability impact5

Invisibility3

Prevalence5

Automatability3

Context dependence2

One of the few Kubernetes failures with an unambiguous signal — OOMKilled says exactly what happened. It stays common because the obvious fix, raising the limit, resolves the symptom often enough to be believed.

Blast radius if this is wrongEveryone

One testEveryone

What contains it

Kubernetes Runtime