The question this answers
What are the moving parts of a Kubernetes cluster, and which one is responsible when something does not run?
An engineer on call needs to answer, at speed, where a deployment got stuck: was it rejected on submission, never scheduled, scheduled but unable to start, or started and failing its probe? Those are four different components and four different fixes.
A directional model of the cluster in which every failure has an address — so "it is not running" becomes a question with four possible answers instead of one shrug.
Five boxes and one direction
The whole system flows one way. You write down desired state. It is accepted, validated and stored by the control plane. Controllers notice the gap and create the workloads needed to close it. The scheduler decides which machine each workload goes to. The node agent on that machine starts it, and what actually runs is a pod. Status flows back up the same path.
Almost every Kubernetes question a beginner asks is really "which box is this?". Was the manifest rejected — that is the API server. Is the workload created but marked pending — nothing has scheduled it, so that is the scheduler and, usually, capacity. Is it scheduled but not starting — that is the node agent, and usually an image pull or a mount. Is it running but not receiving traffic — that is readiness, and therefore the service layer.
Learn the direction before the vocabulary. Engineers who memorize component names without the flow can recite what etcd is and still cannot say why their pod is Pending.
Pending with an events list naming exactly which constraint eliminated every node.ContainerCreating or ImagePullBackOff; the node goes NotReady and its pods are evicted.The control plane in one picture
The control plane is not one process. The API server is the front door and the only component anything else talks to — controllers, the scheduler, node agents and you all go through it, which is why it is also the single place authorization and auditing happen. The datastore behind it holds every object; it is a consensus-replicated key-value store, and it is the thing whose backup you must actually be able to restore. The scheduler and the controller manager are just clients of the API server with special jobs.
On every worker node there are two things worth knowing about: the node agent, which owns the pods on that machine, and the network proxy or dataplane, which implements service routing. Everything else — the runtime, the CNI plugin, the CSI driver — is a pluggable implementation detail that becomes very relevant on exactly one bad day.
The property that surprises people: nodes do not talk to each other through the control plane. Pod-to-pod traffic goes directly across the cluster network. So a control-plane outage leaves your application serving traffic normally, which is exactly why teams discover it hours late, when a deploy hangs.
Reading a stuck deployment with the model
The model earns its keep during an incident. Someone says "the deploy is stuck". Instead of guessing, you walk the flow downward and stop at the first box that has not done its job. The state field on the object tells you where you are, and the events attached to it usually tell you why.
Notice that every row in the table below has a *different owner and a different fix*. Adding memory does not fix an image pull. Restarting the pod does not fix a scheduling constraint. This is the whole reason to hold the model in your head rather than reaching for the same three commands.
| What you see | Which box owns it | Usual cause | What does not help |
|---|---|---|---|
| Manifest rejected on apply | API server | Schema error, admission policy, missing RBAC permission | Retrying. It will be rejected identically. |
| Object exists, replicas stay 0 | Controller | Controller unhealthy, or a selector that matches nothing | Scaling up. Nothing is acting on the count. |
Pod Pending | Scheduler | No node with enough free CPU/memory; a taint or affinity rule excludes every node | Deleting the pod. The replacement is equally unplaceable. |
Pod ContainerCreating / ImagePullBackOff | Node agent | Registry credentials, wrong tag, volume that will not mount | More replicas. Every one of them fails the same way. |
Pod Running, no traffic | Service and readiness | Readiness probe failing, or a label selector that does not match the pod | Restarting. The probe will fail again in ten seconds. |
Pod CrashLoopBackOff | Pod / application | Bad config, missing secret, unmigrated database, real bug | Anything in the cluster. This one is your code. |
Key points
- The flow is one-directional: desired state → API server and datastore → controllers → scheduler → node agent → pod, with status returning along the same path.
- Every component talks only to the API server, which makes it the single authorization, audit and failure chokepoint.
- Pod-to-pod traffic bypasses the control plane entirely, which is why a control-plane outage is quiet: traffic flows, changes do not.
- A pod's state field names the box that is stuck, and each box has a different fix —
Pendingis capacity,ImagePullBackOffis the registry,CrashLoopBackOffis your application. - Learn the direction of flow before the component names; the names are only useful once you know where they sit.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • A manifest is submitted to the API server, which authenticates the caller, authorizes the action, validates the object and writes it to the datastore.
- • Controllers watch the datastore through the API server and create the lower-level objects needed to satisfy the declaration.
- • The scheduler watches for pods with no node assigned, filters nodes by constraints, scores the survivors and binds the pod to the winner.
- • The node agent on that machine sees a pod assigned to it, pulls images, mounts volumes, starts containers and begins running probes.
- • Status is written back through the API server, closing the loop and making the gap visible to whichever controller cares about it.
- • Control-plane availability and version upgrades, including the order in which control plane and nodes may be upgraded.
- • Datastore backups, and a restore you have actually performed — see Restore Testing, because a cluster backup you cannot restore is a cluster you cannot rebuild.
- • Node lifecycle: patching, replacing and draining machines without dropping the workloads on them.
- • The pluggable pieces you chose — network plugin, storage driver, ingress controller — each of which is a component with its own failure modes and upgrade cadence.
- • API server saturated or down: kubectl hangs, deploys stall, autoscaling and healing stop, and user traffic is completely unaffected until something dies.
- • Datastore quorum loss: the cluster becomes read-only or unavailable, and the definition of desired state is at risk.
- • A node going
NotReady: after a grace period its pods are marked for eviction and rescheduled, causing a burst of load on the remaining nodes. - • A misbehaving controller or webhook: admission hangs and every apply in the cluster times out, including the one that would fix it.
- • Datastore write throughput is the classic control-plane ceiling; it is driven by object count and churn, not by application traffic.
- • Watch connections scale with the number of controllers, agents and operators; a cluster full of custom operators pressures the API server more than one full of pods.
- • Node count has practical limits per cluster, and past them the answer is more clusters — which is an organizational cost, not a technical one.
- • The API server is the authorization boundary for the entire cluster; RBAC there decides who can read secrets, exec into pods and change what runs.
- • The datastore contains every secret object in plaintext unless encryption at rest is explicitly enabled — see ConfigMap vs Secret — and the Honest Limit of a Secret and Key Management and Encryption at Rest.
- • Node agents hold credentials that can be escalated if a workload escapes its container, which is why node-level isolation matters for untrusted workloads.
- • Control-plane endpoints should not be publicly reachable without strong justification; a public API server with weak authentication is the cluster equivalent of a public database — compare Public Exposure, Read With Context.
- • Managed control planes are a fixed per-cluster charge, so cluster count — not workload count — is what drives that line item.
- • Self-managed control planes trade that charge for three or more machines you size, patch and keep highly available yourself.
- • Worker nodes dominate the bill; control-plane cost is small in money and large in attention.
- • API server request latency and error rate by verb — the earliest signal that the cluster is becoming hard to change.
- • Datastore leader elections and disk latency; consensus stores are unusually sensitive to slow disks.
- • Node readiness and the count of pods in each phase, which together give you the shape of a cluster-wide problem in one glance.
- • The signal that lies: application dashboards. They stay perfectly green through a total control-plane outage, because traffic never traverses it.
- • A managed container service with no exposed control plane, if what you need is "run these containers" rather than "program a cluster".
- • A managed Kubernetes offering rather than a self-built one — the control plane is the part with the worst effort-to-value ratio to operate yourself.
- • A single-binary lightweight distribution for edge or small-scale use, which collapses these components into one process.
- • For one service, no cluster at all: an instance group with health checks reproduces the useful half of this model — see One Big VM or Several Small Ones.
- • Buys one uniform place to describe and observe every workload; costs a five-component distributed system between you and your process.
- • Buys pluggability for networking, storage and ingress; costs the obligation to choose, upgrade and debug each plugin.
- • Buys a clean separation between control and data planes; costs the confusion of an outage where control is dead and data is fine.
What people believe, and what is true
If the control plane is down, the site is down.
Running pods keep serving and pod-to-pod traffic never touches the control plane. What you lose is deploys, scaling and healing — dangerous, but invisible at first.
etcd is an implementation detail I can ignore.
It holds the definition of desired state and every secret object. Its backup, its disk latency and its quorum are directly your problem on self-managed clusters.
Pod Pending means the cluster is broken.
It almost always means the scheduler could not find a node that satisfies the pod's requests and constraints. The events on the pod say exactly which constraint eliminated which nodes.