K8s RuntimeKUBERNETES-SPECIFICSIMPLIFIED
Reading a Broken Workload
One decision tree covers most Kubernetes failures: is it running, is it ready, is it routed — and each "no" points at a different, small set of causes.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The production question
The service is not serving. Where do I look first, and what does each state actually mean?
The problem
Kubernetes reports its state across many objects and several components, so the same user-visible symptom can originate anywhere from admission to routing — and the pod list alone rarely tells you which.
What teams do first
Look at the application logs. That is where errors are, so that is where the answer is.
How it breaks
A pod that never scheduled has no logs, because no container ever ran. kubectl logs returns nothing and the actual explanation is sitting in an event (The Scheduler, and Why a Pod Is Pending).
How it breaks in production
- A pod that never scheduled has no logs, because no container ever ran.
kubectl logsreturns nothing and the actual explanation is sitting in an event (The Scheduler, and Why a Pod Is Pending). - A container that was killed has its explanation in the previous container's state and logs, both of which the default commands hide behind a flag.
- A pod that is
Runningand notReadyhas perfectly clean logs and is receiving no traffic, because readiness — not the log stream — controls routing (Probes: Readiness, Liveness and Startup). - A workload that is entirely healthy can still serve nothing if the service selector matches no pods, in which case every pod-level signal is green and the failure is in routing (Service Discovery in Operation).
- Starting in the application means starting in the one place that is fine in roughly half of these cases, and the cost of that is measured in incident minutes.
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The decision tree has two branches and covers most of the space. Not serving → is it running? No means scheduling or resources; crashing means logs, config or runtime. Yes means ask the second question. Is it ready? No means readiness or a dependency. Yes means the problem is in the service, the network or the routing layer.
- Each Kubernetes state is a precise claim about which stage was reached, and the states are ordered:
Pendingnever scheduled,Waitingscheduled but not started,Runningstarted,Readypassing readiness, in the endpoint list means routed (Apply Is Not Running). - Events carry the diagnosis for everything before the container starts — scheduling, image pull, volume attachment, admission. Logs carry it for everything after. Neither can answer the other's questions.
- The kubelet keeps the previous container's termination reason and logs after a restart. Reading them is what separates an OOM kill from a liveness kill from an application crash, all three of which look identical in a restart count.
- The last hop — service, endpoints, dataplane — has its own state that no pod-level command shows. An empty endpoint list with healthy pods means the selector does not match, and that is a manifest problem rather than a runtime one.
The tree
Three questions, asked in this order, in every case. The order matters more than the commands: each answer eliminates a whole region of the system, and asking them out of order is how investigations end up in the application by default.
UserLLMAgentToolDataDecisionHumanGuardrail
Every state, and what it actually claims
Read the cause column as "what this state is telling you", and the response column as the next command rather than the fix. Several of these states are routinely read as the problem when they are the consequence.
Workload states and their real meanings
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
Pending | No node assigned; no logs exist at all | No node passed the scheduler's filters | Read the FailedScheduling event — it names the reason per node (The Scheduler, and Why a Pod Is Pending) |
ImagePullBackOff / ErrImagePull | Scheduled, container never starts | Tag does not exist, no pull secret, registry unreachable | Check the exact image reference and registry access (Artifact Registries) |
CreateContainerConfigError | Scheduled, container never starts | A referenced ConfigMap or Secret key does not exist | Check the reference, not the application (ConfigMaps and Secrets) |
CrashLoopBackOff | Repeated restarts with growing delay | The backoff state — the cause is the previous termination | Read previous logs and the previous termination reason before anything else |
Previous reason OOMKilled | Restarts correlated with load, exit code 137 | The container reached its memory limit | Check whether memory plateaus or climbs (OOMKilled: Over the Memory Limit) |
Previous reason Error, non-zero exit | Dies immediately after starting | The application exited — usually missing config, a failed startup check, or a bad command | Previous logs, then startup validation (Validate at Startup, Fail Clearly) |
Running but not Ready | Clean logs, no traffic, rollout stalled | The readiness probe is failing, often on a dependency | Read the probe failure event; do not restart it (Probes: Readiness, Liveness and Startup) |
| Restarts with no crash and no OOM | Healthy logs, periodic kills | Liveness probe failing — possibly because the container is throttled | Check throttling before probe settings (CPU Throttling: The Latency With No Error) |
Pod Evicted | Pod gone, rescheduled elsewhere | Node-level resource pressure; the kubelet chose by QoS class | Find which workload grew — often not the evicted one (Requests and Limits) |
Node NotReady | Many pods across services affected at once | Kubelet, network or node health problem | Stop debugging the workload; this is a node or infrastructure question |
Stuck Terminating | Pod will not go away | A finalizer waiting, or a grace period still running | Check finalizers and the grace period before forcing (Draining: Stopping Without Dropping) |
| Healthy pods, empty endpoints | Everything green, nothing served | The service selector matches no pods, or none are ready | Compare selector to pod labels — a manifest problem, not a runtime one (Service Discovery in Operation) |
The commands, in the order the tree asks for them
TOOL-SPECIFIC
kubectl against a Kubernetes API server. The equivalent first pass on a VM fleet is instance status, the target group health, the launch and user-data logs, and the last deployment record — same three questions, four different tools.Read-only, cheap, and in an order that follows the questions rather than habit. The two flags that matter most are the ones people forget under pressure: sorting events by time, and asking for the previous container.
A first pass that answers all three questions
1# 1. Is it running? States, restarts, and which node.2kubectl get pods -l app=checkout -o wide3 4# Everything before the container starts lives in events, not logs.5kubectl describe pod checkout-7d9f8-abcde6kubectl get events --sort-by=.lastTimestamp | tail -307 8# 2. After a restart, the explanation is in the PREVIOUS container.9kubectl logs checkout-7d9f8-abcde --previous10kubectl get pod checkout-7d9f8-abcde \11 -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'12 13# 3. Ready, and routed? Healthy pods with an empty endpoint list14# means the selector does not match.15kubectl get pod checkout-7d9f8-abcde \16 -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'17kubectl get endpointslices -l kubernetes.io/service-name=checkout18 19# And what changed, which is usually the fastest question of all.20kubectl rollout history deployment/checkoutEvery command here is read-only. Nothing in a first pass should delete a pod, drain a node or edit a live object — those destroy the evidence you are about to need, and reconciliation will recreate the same failure anyway.
How to do it properly
Most important first.
- Ask the three questions in order — running, ready, routed — before running any command. The order is what stops you from starting in the application by default.
- Read events before logs for any pod that is not
Running. Sort them by time; the useful one is rarely the most recent. - For any restart, read the previous container's termination reason first.
OOMKilled,Errorand a liveness-triggered kill need three different investigations (OOMKilled: Over the Memory Limit). - When pods look healthy and nothing is served, check the endpoint list for the service. It is a five-second check that resolves an entire failure class.
- Correlate with what changed before diagnosing from first principles — a workload that was fine an hour ago and is broken now had something happen to it (Change Correlation).
- Write down the tree. This is the single most runbook-shaped piece of knowledge in the module, and it works precisely because nobody has to be clever at 3am (Runbooks).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Blast radius if this is wrongOne tenant
One testEveryone
What contains it
Debugging itself changes nothing, so the blast radius is the actions taken during it — and those are real. Deleting pods, draining nodes, editing live objects and scaling to force a change all affect the workload being investigated, and a drain or a node-level action reaches every workload on that node. The containment is doing read-only steps first and writing down what you changed (Manual Production Changes).
What can go wrong
Failure modes, including of the mitigation
- Debugging the application while the pod never started, because the log command returned nothing and that was read as "no errors".
- Reading the current container's logs after a restart, which are the logs of the healthy new process rather than the one that died.
- Deleting the pod to "reset" it, which destroys the previous state and the evidence, and reconciliation recreates it in the same failed condition anyway (Reconciliation: The Loop Under Everything).
- Treating
CrashLoopBackOffas a cause. It is a backoff state; the cause is in the previous termination. - Escalating a scheduling problem to the application team, or an application problem to the platform team, because the state was not read carefully.
- Assuming that healthy pods mean a healthy service, when the last hop is where the failure is.
Misreads this invites
- "
CrashLoopBackOffis an error." It is the backoff between restarts. The error is the reason the previous container terminated. - "No logs means nothing happened." It usually means no container ran, which is itself the finding.
- "The pod is
Running, so the service works."Runningmeans a process started. Ready means it will be sent traffic; routed means the traffic arrives. - "Restarting fixes it." Sometimes, and it also destroys the evidence and does nothing about a cause that is outside the pod (Production Anti-Patterns).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
How you know it worked
- The pod's phase, its container states, and the reason field on the previous termination — read explicitly rather than inferred from a restart count.
- Events for the pod and its owning ReplicaSet, in time order. Scheduling and image failures live here and nowhere else.
- The service's endpoint list containing the pods you expect, which is the only proof that routing agrees with the pod status.
- A request that reaches the pod, made from inside the cluster and from outside it — the two failing differently localises the problem to the edge (Operating the Edge).
How you get back
- If the workload was working before a change, restoring the previous desired state is faster than diagnosis and is usually the right first move (Rollback: Only Useful If It Is Actually Safe).
- Rollback does not help when the cause is outside the workload — a full cluster, an unavailable registry, a broken dependency. Recognising that early is worth the thirty seconds it takes to check.
- Preserve evidence before you reset anything. Once the pod is deleted, its events and its previous container logs go with it, and you will want them for the postmortem (Reconstructing What Actually Happened).
What to automate, and what stays human
- Automate the collection, not the conclusion: a bundle that captures pod status, previous logs, events and endpoints in one command removes the part of debugging that is typing.
- Automate the alert that distinguishes causes —
OOMKilledterminations,FailedSchedulingevents and probe failures should each be their own signal rather than a generic restart alert (Alert on Symptoms, Not on Causes). - Keep the diagnosis human. This tree narrows the search; deciding what a finding means about your system is judgement (The Automation Trap).
What this costs
- The tree is deliberately shallow, so it will not resolve subtle problems — mesh configuration, kernel-level networking, storage performance. It is designed to resolve the common ninety percent quickly and to tell you honestly when you have left its territory.
- Evidence collection during an incident costs a minute or two before mitigation. That is usually worth it, and it is not worth it when users are down and the rollback is obvious.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICThe states and commands are Kubernetes. The tree itself is not: on a VM fleet the same three questions are "did the instance launch", "did it pass the target group health check" and "is it registered", and on a PaaS they collapse into a build log, a deploy log and a routing state. What is unusual about Kubernetes is that all three questions have separate, readable, per-object answers — more visibility and more places to look.
- SIMPLIFIEDCovers workload-level failures. It does not cover control-plane failures, CNI and service-mesh problems, storage performance, or node-level kernel issues, all of which present as workload symptoms and need a different investigation.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
Domains that do not exist yet
- — Testing & Reliability Engineering — why a decision tree written down beats accumulated intuition when the person on call is not the person who built it.