The question this answers
What happens at each boundary of the resources a workload has been allocated, and why do the boundaries behave so differently?
A service must stay within whatever CPU and memory it was allocated, on every instance, under peak load — and must fail in a way its operators can diagnose when it does not.
A predictable, enforced allocation: the workload is guaranteed what it reserved, prevented from exceeding what it was capped at, and isolated from neighbours that try to exceed theirs.
Four walls, four completely different enforcement mechanisms
The useful mental model for compute is a box. CPU, memory, storage and network are its walls, and the workload lives inside. What makes this more than a metaphor is that the walls are enforced by different mechanisms with different behaviour on contact, and knowing which is which turns a confusing incident into an obvious one.
CPU is *elastic and shared*. Exceeding your CPU allocation does not fail; it queues. Work slows down, latency rises, and — where a quota is enforced — the scheduler simply stops running you for part of each period. Nothing errors. The application sees latency and reports it as a downstream problem.
Memory is *hard and fatal*. There is no queueing for memory. Exceeding the allocation gets the process killed by the kernel's out-of-memory handling, with no chance to log, no exception, no stack trace. To the application's own telemetry the process simply ceases; to the platform it is an exit code and an event. This asymmetry — CPU degrades, memory kills — is the single most useful thing to internalize about resource limits, and it is why memory sizing deserves more care than CPU sizing.
Storage is *hard and non-fatal*: writes fail, the process usually survives, and the application returns errors while CPU and memory look healthy. Network is *elastic with a ceiling*: throughput plateaus, latency rises, nothing errors, and the symptom is indistinguishable from a slow dependency.
- CPU degrades. Over-subscription queues; a quota throttles. Latency rises, nothing errors, utilization can look moderate throughout.
- Memory kills. No graceful degradation exists. The process is terminated and the application cannot report it.
- Storage errors. The write fails, the process survives, and the failure surfaces as application errors with healthy CPU and memory.
- Network plateaus. Throughput stops rising, latency climbs, and it is indistinguishable from a slow downstream service without instance-level network metrics.
- Two of the four walls produce *no error at all*, which is why they are diagnosed last.
Reading the evidence for each wall
Because two walls are silent and one is fatal-without-a-message, every one of them requires evidence from outside the application. The application cannot tell you it was throttled, cannot tell you it was killed, and cannot distinguish its own network ceiling from a slow dependency.
The specific incident worth remembering: a service was restarting roughly every forty minutes under load. Application logs showed a clean startup and then nothing — no exception, no shutdown message, just a new startup line. Every application metric looked healthy right up to each gap. The cause was a memory limit set months earlier against a working set that had since grown, and the only evidence anywhere in the system was a kernel OOM line and a container exit code. The team spent two days in the application before anyone looked at the platform events, because the application's own telemetry is structurally incapable of recording its own death.
The counterpart on the CPU side is quieter and lasts longer: a container consuming 55% of its CPU quota on average while being throttled during every burst. The dashboard says there is headroom. The throttled-periods counter says the workload was denied CPU in 30% of enforcement periods. Both are true; only one explains the latency.
MEMORY WALL — evidence is outside the application kernel: Out of memory: Killed process 4711 (node) total-vm:2.1GB platform: exit code 137, reason OOMKilled, restarts: 34 application log: <clean startup> ... <nothing> ... <clean startup> -> the app cannot log its own kill. stop reading app logs. CPU WALL — utilization and denial are different counters cpu.usage : 55% of quota <- "we have headroom" cpu.throttled_pct: 30% of periods <- "we were denied CPU" p99 latency : 210ms -> 1.4s during throttled periods -> both numbers are honest. only the second explains latency. STORAGE WALL — the only one that errors properly ENOSPC: no space left on device disk usage 100%, CPU 12%, memory 40% -> healthy-looking instance, failing application NETWORK WALL — looks exactly like a slow dependency instance network out: flat at the instance ceiling for 40 min downstream service p99: unchanged -> your egress is capped; the dependency is innocent ILLUSTRATIVE: representative output, not a real capture.
Reservation, limit, and the gap between them
Most platforms let you express two numbers per dimension: what the workload is *guaranteed* (a reservation or request, used for placement) and what it *may not exceed* (a limit). The relationship between them is a genuine design decision with three defensible answers and one common mistake.
Setting reservation equal to limit gives predictable, isolated behaviour and wastes the difference between typical and peak usage on every instance. Setting a limit far above the reservation allows bursting and oversubscribes the host, which works until several workloads burst together and the host itself is exhausted. Setting no limit at all means one workload can starve every neighbour — and on memory, can trigger a system-level OOM that kills a process the platform did not intend to kill.
The common mistake is a *memory* limit set from an early measurement and never revisited, combined with no CPU limit. That combination is precisely backwards: the dimension that kills you is pinned to a stale number, and the dimension that merely slows down is unbounded. Reverse it — leave memory generous relative to the observed peak, and constrain CPU if you need predictability. See Requests vs Limits: Two Numbers That Do Different Jobs for how this looks under an orchestrator.
resources:
requests:
cpu: 100m
memory: 256Mi # measured once, 8 months ago
limits:
memory: 256Mi # the fatal wall, set from a stale number
# no cpu limit # the survivable wall, left wide open
# result:
# working set grew to ~400Mi -> OOMKilled every ~40 min
# no application error, no stack trace, restarts look "normal"
# meanwhile one hot instance can starve the whole node of CPUresources:
requests:
cpu: 500m # what it needs at typical load -> drives placement
memory: 512Mi # observed p99 working set
limits:
memory: 768Mi # ~1.5x observed peak: room to grow, still bounded
cpu: 2 # burst allowed, but one instance cannot take a node
# plus, because two walls are silent:
# alert on container_oom_kills > 0 (memory wall)
# alert on cpu_throttled_periods_pct > 10 (cpu wall)
# alert on disk_free_pct < 15 (storage wall)
# and review the numbers when the working set changes, not once.The dimension that terminates your process should carry headroom and an alert; the dimension that merely slows it down is the one worth capping for predictability. Getting this backwards produces the most confusing failure in the domain — a process that vanishes cleanly, on schedule, with nothing in its own logs.
Key points
- Compute is a bounded envelope with four walls, and each wall is enforced by a different mechanism with different behaviour on contact.
- CPU degrades and throttles, memory kills, storage errors, network plateaus. Two of the four produce no error at all.
- A process cannot log its own OOM kill, so memory failures must be diagnosed from platform or kernel events rather than application telemetry.
- CPU utilization and CPU throttling are different counters; a workload can look comfortable and be denied CPU in a third of enforcement periods.
- Give headroom and alerts to the fatal wall (memory) and shape the survivable one (CPU) — the reverse is the most common and most confusing misconfiguration.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The platform enforces CPU with a scheduler quota per period: exceed it and the workload is simply not scheduled for the remainder of that period.
- • Memory is enforced by refusing allocation and invoking out-of-memory handling, which selects and terminates a process — a kernel action, not an application one.
- • Storage limits are filesystem-level: the write returns an error and the process decides what to do with it, which is why this wall is the only well-behaved one.
- • Network limits are shaped at the virtual interface, so excess traffic queues and then drops rather than erroring at the socket layer.
- • Reservations drive placement decisions; limits drive enforcement. They are answering different questions and should rarely be the same number.
- • Setting and periodically revisiting both numbers per dimension — a limit set from a single early measurement ages into an outage.
- • Alerting on the silent walls specifically: OOM kill count, throttled-period percentage, disk free percentage. Nothing else surfaces them.
- • Understanding what your runtime does inside the envelope: a JVM or Node heap limit that does not know about the container limit will happily grow into a kill.
- • Reviewing limits after any change to working-set size — a new cache, a bigger page size, a library upgrade all move the memory wall.
- • Periodic silent restarts with clean application logs: an OOM kill against a stale memory limit, visible only in platform events.
- • Latency spikes correlated with load while CPU utilization looks moderate: throttling against a CPU quota.
- • Application errors with a completely healthy-looking instance: a full disk, usually from unrotated logs or accumulated temporary files.
- • A workload with no limits starving its neighbours on a shared host, converting one team's bug into everyone's incident.
- • A runtime heap sized independently of the container limit, so the garbage collector believes it has room that the platform will kill it for using.
- • Adding instances does not move a per-process wall: an OOM at 512 MB is an OOM at 512 MB on twenty instances as well as on two.
- • CPU scales out well because the work is divisible; memory frequently does not, because a working set is a property of the process rather than of the traffic.
- • Under an orchestrator, reservations decide how many workloads fit on a node, so over-reserving reduces density and under-reserving oversubscribes the host — see Scheduling: How a Pod Chooses a Node.
- • The envelope is a containment boundary as well as a sizing one: an unbounded workload can deny service to every co-tenant on the host.
- • Memory limits bound the damage from a leak or an unbounded input, which makes them a resilience control and not only a cost control.
- • Local scratch space inside the envelope persists for the life of the instance and is readable by anything else running in the same boundary.
- • Reservations are what you effectively pay for under an orchestrator, because they reserve capacity on a node whether or not the workload uses it.
- • The gap between reservation and actual usage across a fleet is the single largest source of recoverable waste in most container platforms — see Right-Sizing Without Causing an Outage.
- • Over-generous memory limits cost nothing directly if the reservation is honest; over-generous *reservations* cost real capacity on every node.
- • Working set against the memory limit, as a ratio rather than an absolute, so the alert survives a resize.
- • Throttled-period percentage alongside CPU utilization; neither is interpretable without the other.
- • OOM kill events and container restart reasons, which are the only record of the fatal wall.
- • The signal that lies: memory usage reported by the runtime. A garbage-collected runtime reports heap, not resident set, and the kernel kills on resident set.
- • A platform that sizes for you — a function platform where you pick one memory number and CPU scales with it removes three of the four decisions, at the cost of control.
- • No limits at all, on a dedicated single-tenant instance. If one workload owns the machine, limits add failure modes without adding isolation.
- • Vertical scaling of the whole instance instead of per-workload limits, when there is exactly one workload — simpler, and the envelope is the machine.
- • Fixing the working set rather than raising the wall: an unbounded in-memory cache or an unpaginated query is a code fix, and raising the limit only postpones it.
- • Tight limits buy predictable neighbours and density and charge occasional kills and throttling when the workload deviates from its measurement.
- • Generous limits buy resilience to growth and charge oversubscription risk on the host, which converts one workload's bad day into a shared one.
- • Reservation equal to limit buys the most predictable behaviour available and wastes the difference between typical and peak on every instance, permanently.
What people believe, and what is true
Hitting a resource limit makes the application return an error.
Only the storage wall does that. CPU throttles silently, memory kills the process without an error, and the network plateaus. Two of the four are entirely invisible to the application.
If CPU utilization is under 60% there is CPU headroom.
Utilization measures what you consumed, not what you were denied. A workload can sit at 55% average and be throttled during every burst — the throttled-periods counter is the one that matters.
Set limits equal to requests to be safe.
It is predictable and it permanently wastes the difference between typical and peak on every instance. It is a legitimate choice for latency-critical workloads and a poor default for everything else.