How Resource Settings Go Wrong
Four failure shapes come from two numbers being wrong in two directions each — and each shape has a distinct symptom that tells you which one you are looking at.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
My workload is unhealthy and the code did not change. Which resource number is wrong, and in which direction?
Requests and limits are usually chosen once, from a guess, by someone who no longer works on the service — and both being wrong produces symptoms that look like application bugs.
Copy the resource block from a service that works, double it for safety, and move on. Nobody has time to profile every workload, and generous numbers cannot hurt.
Generous numbers do hurt, in a way that shows up on the invoice rather than in the logs: requests are reserved whether or not they are used, so an inflated request buys idle capacity forever (Idle Capacity).
- Generous numbers do hurt, in a way that shows up on the invoice rather than in the logs: requests are reserved whether or not they are used, so an inflated request buys idle capacity forever (Idle Capacity).
- The copied service has a different shape. A request/response API and a batch worker have opposite profiles, and a number that is right for one is wrong for the other in both directions.
- A too-low memory limit does not degrade — it kills, under load, which is precisely when you needed the replica (OOMKilled: Over the Memory Limit).
- A too-low CPU limit produces latency with no error, no restart and no log line, which is the hardest symptom to attribute in this entire domain (CPU Throttling: The Latency With No Error).
- A too-low request means the workload is packed onto a busy node and loses every contention fight, so it is slow only when the neighbours are busy — an intermittent problem that never reproduces.
- Nobody revisits the numbers. The workload gets a new dependency, a new cache, a new runtime version, and the manifest still carries a guess made two years ago (Configuration Drift).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- There are two numbers and each can be too low or too high, so there are four shapes. Because request and limit are read by different systems, the four shapes have genuinely different symptoms rather than degrees of the same one (Requests and Limits).
- Request too low: the scheduler thinks the pod is small, packs it tightly, and under contention the pod gets its (small) share. The workload is fine on a quiet node and slow on a busy one.
- Request too high: the pod reserves capacity it never uses. The cluster reports scheduling pressure while its nodes sit idle, and you pay for both.
- Limit too low: memory gets a kill, CPU gets a throttle. Same cause, entirely different presentation.
- Limit too high or absent: nothing constrains the container, so a leak or a regression expands until it destabilises the node and the kubelet starts evicting by QoS class — often something other than the culprit.
- The right numbers come from observed behaviour under realistic load, over a window long enough to include the peaks and the periodic work. A single reading tells you almost nothing about a workload whose memory grows between garbage collections.
Four shapes, four symptoms
This table is the diagnosis. Start from the symptom column, because that is what you have, and read backwards to the number that is wrong.
| Setting | Direction | What you observe | What it actually costs |
|---|---|---|---|
| Request | Too low | Slow only when the node is busy; irreproducible in isolation | Loses contention fights; packed onto crowded nodes |
| Request | Too high | Pods Pending while node CPU graphs look idle | Reserved capacity nobody uses, billed continuously |
| Memory limit | Too low | OOMKilled, restarts, CrashLoopBackOff under load | Loses in-flight requests every time; capacity drops at peak |
| Memory limit | Too high or unset | Node memory pressure; unrelated pods evicted | A leak grows until it damages neighbours instead of one container |
| CPU limit | Too low | Tail latency with no errors, no restarts, no log lines | The hardest symptom in the module to attribute |
| CPU limit | Too high or unset | Neighbours slow down when this workload is busy | Isolation traded away; attribution becomes guesswork |
The fixes that make it worse
Each of these is a reasonable first response that has a specific way of backfiring. The pattern is the same throughout: the number that produced the symptom is not always the number that is wrong.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
Container OOMKilled | Restart loop at peak traffic | Limit below real peak, or a leak | Raise the limit only after checking whether usage plateaus. If it grows without bound, a bigger limit buys a longer fuse (OOMKilled: Over the Memory Limit) |
| Tail latency with no errors | p99 up, p50 flat, CPU graph unremarkable | CPU limit reached within scheduling periods | Check throttling counters before touching anything else (CPU Throttling: The Latency With No Error) |
| Slow only sometimes | Same version fast on one node, slow on another | CPU request too low, so it loses the share fight on contended nodes | Raise the request, not the limit — the limit was never reached |
Pods Pending, nodes idle | Scheduling failures with low utilisation | Requests inflated across the fleet | Lower requests towards observed usage; adding nodes buys the same idle capacity again (The Scheduler, and Why a Pod Is Pending) |
| Unrelated pods evicted | Node under memory pressure; victims are BestEffort | An unlimited container expanded into the node | Set the limit on the culprit; the evicted pods were collateral, not the problem |
| Automated resizing applied | Fleet-wide pod replacement, unexplained restarts | A recommender wrote new values, which replaces pods | Constrain it to windows and disruption budgets, or make it advisory (The Automation Trap) |
Where the numbers should come from
There are four honest sources for a resource number and they differ mainly in how much you should trust the result. The mistake is not using a weak source — it is using a weak source and then never revisiting it.
Where does the number come from?
when The service already runs and you have per-container metrics covering a peak.
cost Requires the metrics to exist and a window long enough to include periodic work — the best answer and the slowest to get.
when Pre-launch, and you can reproduce the traffic mix well enough to trust it.
cost Confidently wrong if the mix is wrong; test data is usually kinder than production data (Why Local Success Predicts So Little).
when A new service of a familiar shape on an established platform.
cost Right for the median service and wrong for the outliers, who will not find out until traffic arrives (Golden Paths).
when Genuinely unknown workload, and you have written down when you will measure it.
cost Only honest if the revisit actually happens; without the date it is the guess that lives forever.
How to do it properly
Most important first.
- Diagnose before tuning. Identify which of the four shapes you have from its symptom, because the fixes point in opposite directions and a wrong guess makes it worse.
- Set memory request equal to memory limit, at observed peak plus a margin you can state and defend.
- Set the CPU request at a realistic working level rather than at peak, since it is a share as well as a reservation.
- Change one number at a time and watch the specific signal it should move. Changing both at once means you learn nothing from the result (Change Size: Why Small Changes Are Safer, and When They Are Not).
- Put the numbers where they can be reviewed with the code, and treat a resource change as a deploy — because it is one (A Config Change Is a Production Change).
- Re-derive them after any change to runtime, dependencies or traffic shape, and at least whenever the service gets a significant release.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
One workload's numbers are contained to that workload, and every replica shares them — so all replicas fail together under the traffic that exposes the mistake. It becomes cluster-wide when the values live in a shared service template, or when an unconstrained container destabilises its node and the kubelet begins evicting by QoS class, which hits workloads that did nothing wrong.
What can go wrong
- Tuning the limit when the request was the problem, so the workload is still packed onto a contended node and is now also less constrained.
- Raising a memory limit to stop OOM kills, which converts a fast repeated failure into a slow leak that eventually takes a node (OOMKilled: Over the Memory Limit).
- Removing a CPU limit to stop throttling in a shared cluster, and moving the problem to whoever is unlucky enough to share a node.
- Applying an automated recommendation cluster-wide, which replaces every pod at once and is a fleet-wide rollout nobody scheduled.
- Sizing from a load test that does not reproduce the production traffic mix, and getting a number that is confidently wrong.
- "OOM kills mean we need a bigger limit." Sometimes. They equally often mean a leak, and raising the limit only moves the failure further away from its cause (OOMKilled: Over the Memory Limit).
- "The pod is using 40% of its limit, so it is fine." Peak matters, not average, and the limit is not the number the scheduler used to place it.
- "High utilisation means we are efficient." It also means there is nothing left for a node failure or a rolling update (Headroom).
- "Resource settings are ops tuning." They are part of the workload's definition, change its behaviour under load, and belong in review with the code.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Usage against request and limit per container, at peak, over a window that includes your worst hour.
- Throttled time or throttled periods, per container, trending at or near zero for latency-sensitive services.
- Zero
OOMKilledterminations outside deliberate tests. - Sum of requests versus sum of usage per node — the gap between them is the money the guess is costing (Cost Drivers).
- The tail latency of the service on busy nodes versus quiet nodes. A gap there is a request problem, not a limit problem.
- Restoring the previous values is another full rollout, so a resource change carries the same risk and the same timing as any deploy (Apply Is Not Running).
- Reverting a lowered memory limit does not un-kill the pods that were already terminated; the recovery is complete only once the replacement pods are ready.
- Reverting a raised request can leave pods unschedulable if the cluster shrank while the higher value was in place — check capacity before rolling back, not after (The Scheduler, and Why a Pod Is Pending).
- Automate observation and recommendation: usage percentiles against configured values, surfaced per workload, is the single highest-value piece of tooling here.
- Automate rejection of the obviously wrong: no memory limit, or a limit far above anything the workload has ever used, is a policy check (Policy as Code).
- Keep the application of recommendations human, or at least scheduled and disruption-aware. Automatic resizing that ignores rollout windows turns a cost optimisation into an availability event (The Automation Trap).
- Accurate numbers require measurement, and measurement requires per-container resource metrics that someone has to run and pay for.
- Tight sizing raises utilisation and reduces the slack that absorbs traffic spikes and node failures — you are converting money into risk, deliberately.
- Uniform defaults across a platform are easy to reason about and wrong for most individual services; per-service tuning is right and does not scale without tooling (Golden Paths).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICThe four shapes come from Kubernetes having two independent numbers. On a VM there is one number — the instance size — so the failure modes collapse to "too small" (the machine runs out and everything on it suffers) and "too large" (you overpay). A PaaS plan tier behaves the same way. The compensation is that a VM cannot be wrong in two directions at once; the cost is that you cannot pack workloads at all.
- SCALE-SPECIFICBelow a handful of services, uniform generous defaults are cheaper than the tooling needed to size properly. Past that, the aggregate waste and the aggregate incident rate both justify per-workload measurement.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.