The Problems Kubernetes Answers
Five operational problems appear the moment you have many containers on many machines. Every Kubernetes object is an answer to one of them, and is only worth learning as such.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What problems appear once you have many containers on many machines, and which object answers each?
Containers solve environment reproducibility for one process. They say nothing about where that process should run, what happens when it dies, how callers find it, how it gets replaced, or how you get more of it.
Learn the object types as vocabulary. Read the list — Pod, Deployment, ReplicaSet, Service, Ingress — memorise what each field does, and assemble manifests by pattern-matching against examples.
Vocabulary without the problem produces manifests that work and cannot be debugged. When a rollout stalls, you need to know which controller is waiting on what, not which field goes where.
- Vocabulary without the problem produces manifests that work and cannot be debugged. When a rollout stalls, you need to know which controller is waiting on what, not which field goes where.
- It leaves you unable to evaluate anything. Every question of the form "should this be a Deployment or a StatefulSet" is a question about the problem, and vocabulary gives no way in.
- Copied manifests carry copied mistakes. The most common are missing resource requests and a liveness probe that checks a downstream dependency — both invisible until load or an outage arrives.
- The abstraction stops being transferable. Every orchestrator answers these five problems; only Kubernetes uses these five nouns.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Start from the situation and the problems fall out. You have N containers to run and M machines to run them on, machines and containers both fail, and the set of running containers changes continuously.
- Placement: something has to decide which machine each container runs on, subject to available CPU and memory. That is the scheduler (The Scheduler, and Why a Pod Is Pending), and it decides using declared requests (Requests and Limits).
- Recovery: when a container or a machine dies, something has to notice and act. That is a controller running a reconciliation loop, comparing observed state against declared state (Reconciliation: The Loop Under Everything).
- Discovery: replacement containers have new addresses, so callers cannot hold onto one. Something must provide a stable name that resolves to the current healthy set. That is a Service (Services: A Stable Address Over Moving Pods).
- Rollout: replacing every container with a new version without dropping traffic requires ordering, health gating and a way to stop. That is what a Deployment controller does (Deployments: Declaring What Should Be Running).
- Scaling: the right number of containers changes with load, so the replica count needs to be a controlled variable rather than a constant (Horizontal Pod Autoscaling).
Five problems, five answers
Read this table as the shape of the whole module. Each row is a problem you would have to solve by hand on a fleet of machines, and the object that solves it for you.
The third column is the part worth remembering: what actually goes wrong when the answer is missing or misconfigured. That is what turns the row from vocabulary into a debugging tool.
| Problem | Answer | What it looks like when it is wrong |
|---|---|---|
| Where should this container run? | Scheduler, against declared requests | Pods stuck Pending, or nodes oversubscribed and everything slow (The Scheduler, and Why a Pod Is Pending) |
| What happens when it dies? | Controllers reconciling toward desired state | A dead workload nobody replaced, or a crash loop nobody noticed (Reconciliation: The Loop Under Everything) |
| How do callers find it? | Service — a stable name over a changing set | Callers holding dead addresses; errors that clear after a retry (Services: A Stable Address Over Moving Pods) |
| How do we replace all of them safely? | Deployment — gradual, health-gated rollout | Both versions serving incompatibly, or a rollout stalled at half (Version Coexistence: N and N+1, in Both Directions) |
| How do we get more of them? | Replica count as a controlled variable | Manual scaling during an incident, or scaling on the wrong signal (Choosing the Scaling Signal) |
The same story without an orchestrator
It is worth walking the problems in order on a plain fleet of VMs, because that is where the objects come from. Nothing here is Kubernetes-specific; it is what you build if you do not have it.
Every step below is a real thing teams write. The point is not that writing them is wrong — for two services it is entirely reasonable — but that the fifth time you write them, you have built an orchestrator with no documentation.
- 1Decide placement
Pick which host each container runs on, tracking free CPU and memory somewhere.
fails by The tracking is a spreadsheet or a person, so it drifts and hosts get oversubscribed.
evidence A host inventory that matches reality after a week of changes.
- 2Supervise processes
Restart containers that exit, on each host.
fails by Handles a dead container; does nothing about a dead host.
evidence A killed container comes back without a human.
- 3Register addresses
Write each instance's address somewhere callers can read.
fails by The registry and reality disagree during restarts, so callers get dead addresses (Service Discovery in Operation).
evidence Registry contents match running instances during a rolling restart.
- 4Roll out a version
Replace instances in batches, waiting for health between batches.
fails by The script has no notion of "healthy", so it proceeds through a broken version (A Successful Deploy Is Not Evidence of a Healthy System).
evidence A deliberately broken build stops the rollout instead of completing it.
- 5Change the count
Add or remove instances as load changes.
fails by Adding instances also requires placement and registration, so the three steps must stay consistent.
evidence Scaling up during load produces instances that receive traffic.
Kubernetes is one implementation of these five steps with a consistent state model behind them. That model — declare desired, reconcile continuously — is the actual product (Reconciliation: The Loop Under Everything).
How the answers fit together
The objects are not a flat list; they nest. A Deployment owns ReplicaSets, a ReplicaSet owns Pods, a Pod holds containers, and a Service selects Pods by label rather than by ownership — which is why a Service can front pods from two different ReplicaSets during a rollout.
That last detail is the source of a great deal of confusion and of one important behaviour: during a rolling update, both the old and the new version are behind the same Service address at the same time (Version Coexistence: N and N+1, in Both Directions).
How to do it properly
Most important first.
- Learn each object by the question it answers, and be able to state that question before writing any of its YAML.
- When something misbehaves, identify which of the five problems is failing first. That picks the layer to look at, which is most of the work in cluster debugging (Reading a Broken Workload).
- Notice which problems your system does not have. A single-workload system has a rollout problem and no placement problem, which is why simpler platforms fit it.
- Carry the five questions to any other orchestrator you meet. They transfer; the nouns do not.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Misunderstanding the model shows up as a misconfigured workload rather than a cluster-wide event; the containment is that most such mistakes affect one service until a shared resource is involved.
What can go wrong
- Objects created because a template had them, not because a problem needed them — an Ingress in front of an internal-only service, a Service with no consumers.
- The wrong object for the problem: a Deployment used for a workload that needs stable identity and its own storage (StatefulSets: Identity, Storage and Order).
- Solving a problem twice — an application-level service registry running inside a cluster that already provides discovery, with two sources of truth that disagree during rollouts.
- Assuming an object solves more than it does. A Service gives a stable address; it does not retry, it does not know your dependency is unhealthy in a way probes do not catch, and it does not balance by anything you would call intelligence.
- "Kubernetes is a container runtime." It is not; it schedules and supervises containers that a runtime such as containerd actually runs, using the same kernel primitives any container uses.
- "These problems only exist at scale." They exist as soon as you have two machines and care about a machine dying. Scale changes whether automating them is worth it.
- "If I know the objects, I know Kubernetes." The objects are the interface. The behaviour under failure — eventual consistency, probe semantics, resource pressure — is the substance (Apply Is Not Running).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For each object in your manifests, someone can name the failure that would occur if it were deleted.
- A new engineer can predict what happens when a node is drained, without reading documentation, because they know which controller is watching what.
- Debugging starts at the right layer: rollout problems get looked at through the Deployment and ReplicaSet, traffic problems through the Service and endpoints.
- This is a framing lesson rather than a change, but its practical form is: manifests derived from problems can be deleted safely when the problem goes away, and copied manifests cannot, because nobody knows what they were for.
- Automate manifest generation from a small set of inputs once the pattern is stable — a service template that asks for the problem-level facts rather than the YAML (Service Templates).
- Do not automate away the understanding. A generator that emits objects nobody can explain reproduces the vocabulary problem at scale.
- Deriving objects from problems is slower to learn than memorising a manifest, and much faster to debug from.
- The five-problem model is deliberately simplified: it omits jobs, network policy, RBAC, custom resources and operators. Those are answers to further problems, and are worth meeting the same way.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICThe object names are Kubernetes'. The five problems are not: ECS answers them with task definitions, services and target groups; Nomad with jobs, groups and its own service registry; a PaaS answers all five invisibly and gives you almost no control over the answers.
- SIMPLIFIEDFive problems is a teaching frame. It deliberately leaves out batch execution, network policy, authorisation and extension via custom controllers, each of which adds objects that are not derivable from these five.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.