Reconciliation: The Loop Under Everything
You write desired state; a controller observes actual state; the difference is the instruction. That loop never stops running, which is the whole idea.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What is actually happening between the moment you apply a manifest and the moment the cluster matches it — and why does it keep happening afterwards?
An imperative instruction executes once. A production platform has to keep the system in the shape you asked for through node failures, evictions, crashes, restarts and other people's changes — long after whoever typed the command has gone home.
kubectl apply is a remote command. It says "run five replicas", the cluster runs five replicas, and the operation is complete when the command returns. Applying is deploying.
The command returns as soon as the desired state is stored. Nothing has run yet. A pipeline that treats a successful apply as a successful deploy is verifying that the API server accepted a document (Apply Is Not Running).
- The command returns as soon as the desired state is stored. Nothing has run yet. A pipeline that treats a successful apply as a successful deploy is verifying that the API server accepted a document (Apply Is Not Running).
- A one-shot command has no answer for what happens when a node dies at 3am. Something has to notice the gap and act on it without a human, and that something is a loop, not a command.
- It makes hand edits look harmless. Anything you change by hand that contradicts stored desired state gets reverted by the controller that owns it, usually within seconds and without explanation.
- It hides the most common Kubernetes failure shape: the desired state is perfectly valid and the cluster cannot reach it. Nothing is "broken" in the sense the command model expects — the loop is simply not converging.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The core loop is five steps and it is the same for every object type: desired state (what you declared, stored in the API server) → controller (a process watching that object type) → observed state (what is actually true right now) → difference → action (an API call that reduces the difference).
- The API server is a store with a watch interface, not an executor. It validates, persists, and notifies. Everything that changes the world is a controller reacting to what it stored.
- The loop is level-triggered, not edge-triggered. A controller does not process a queue of events describing what changed; it repeatedly compares the current desired state to the current observed state. A missed notification therefore self-corrects on the next pass, and replaying an event twice is harmless.
- Controllers chain. A Deployment controller does not create pods — it creates and scales a ReplicaSet. The ReplicaSet controller creates pods. The scheduler binds pods to nodes (The Scheduler, and Why a Pod Is Pending). The kubelet on that node starts containers and reports status back. Each link is its own loop with its own desired and observed state (Deployments: Declaring What Should Be Running, ReplicaSets: The Layer You Should Not Manage).
- This is why self-healing is not a feature bolted on top. Deleting a pod does not remove it in any lasting sense; it changes observed state, the ReplicaSet controller sees a difference, and it makes a new one.
Desired, observed, difference, action
kubectl apply.This is the whole model. You never tell Kubernetes to do something; you tell it what should be true. A controller is a process whose entire job is to notice that it is not true yet, and to take one step towards making it true — then look again.
Because it looks again, the loop is not a deployment mechanism that happens to run at deploy time. It is running right now, on everything, and it will run again when a node dies. That is the difference between orchestration and a deploy script.
Level-triggered is why it survives
An edge-triggered system reacts to change notifications: something happened, do the corresponding thing. That works until a notification is lost, delivered twice, or arrives out of order — at which point the system is permanently wrong and nothing will correct it.
A level-triggered system reacts to the current gap. It does not care how many notifications it missed, because the next comparison sees the same gap and takes the same action. Lost messages become latency rather than corruption, and duplicated actions become no-ops.
This is also why "the cluster undid my change" is not a bug report. The comparison does not know your edit was deliberate; it knows observed state differs from desired state.
on node_failure(node): count = pods_lost(node) start(count) more pods # missed the event -> short forever # handled it twice -> now running seven # script crashed mid-way -> unknown state
every pass: desired = spec.replicas # 5 observed = count(healthy pods) # 3 if observed < desired: create(desired - observed) if observed > desired: delete(observed - desired) # missed a pass -> next pass fixes it # ran twice -> second pass is a no-op # crashed mid-way -> next pass sees the real gap
The second one has no memory to corrupt. Its correctness depends only on what is true now, which is the only thing you can actually observe after a failure. Every retry, restart and duplicate becomes harmless, and that is what makes the loop safe to run continuously and unattended.
When the loop does not converge
A converging loop is boring. A non-converging one is the most common Kubernetes incident, and it has a specific signature: the desired state is accepted, no component reports an error, and nothing changes. Read these as "which controller is stuck and on what".
The three questions that resolve nearly all of them, in order: has the controller seen my change (observedGeneration), what does the controller say in its conditions, and what does it say in its events.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Apply returns success, nothing changes | status.observedGeneration behind metadata.generation | The responsible controller is down, throttled, or not watching this namespace | Check the controller's own pods before anything else — a controller is a workload too |
| Deployment stuck part-way | Progressing becomes False, reason ProgressDeadlineExceeded | New pods never became ready — usually a readiness or dependency problem (Probes: Readiness, Liveness and Startup) | Read the new ReplicaSet's pods, not the Deployment |
| Pods created, never scheduled | Pods Pending with FailedScheduling events | No node satisfies the pod's requests or constraints (The Scheduler, and Why a Pod Is Pending) | Read the scheduler's event message — it names the filter that excluded each node |
| Your edit reverts within seconds | The object flaps between two values | Another controller owns this field and is reconciling it back (Drift) | Change desired state at its source, not the live object |
| Writes rejected across a namespace | Apply fails with a webhook timeout | An admission webhook is unavailable and configured to fail closed | Fix or bypass the webhook; desired state cannot be updated until then |
| Field change refused | field is immutable on apply | Some fields are set at creation and cannot be reconciled | Replace the object deliberately — and check what that does to traffic first |
| Everything green, users see the old version | Converged Deployment, stale behaviour | The image tag was reused, so the digest never changed (Tags Versus Digests) | Deploy by digest; the loop compared two identical specs and correctly did nothing |
How to do it properly
Most important first.
- Read every symptom as a question about the loop: what is desired, what is observed, and which controller is responsible for closing the gap. That question resolves more Kubernetes incidents than any command does.
- Compare
metadata.generationwithstatus.observedGenerationbefore debugging behaviour. If they differ, the controller has not yet acted on your change and you are looking at the old spec running. - Treat the stored desired state as the source of truth, and make sure it lives in version control rather than in someone's shell history (Infrastructure as Code).
- Expect hand edits to be reverted, and treat that as correct behaviour rather than a bug. If you need a temporary change to survive, change desired state (Manual Production Changes).
- When you need to know whether a rollout finished, ask the controller's own status conditions rather than inferring it from the apply exit code (A Successful Deploy Is Not Evidence of a Healthy System).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing at the mechanism level — a controller applies whatever desired state it is given, to every object matching it, as fast as it can. The only real limits are namespace and RBAC scoping, and the workload's own rollout strategy, which is why maxUnavailable is a safety setting rather than a tuning one.
What can go wrong
- The loop converges perfectly on the wrong thing. Reconciliation guarantees the cluster matches what you wrote; it has no opinion about whether what you wrote is correct.
- Two controllers with overlapping ownership fight, and objects flap between two states forever. A GitOps controller reverting a
kubectl editevery thirty seconds is the common version. - A validating or mutating admission webhook is down, so writes to the affected object types are rejected and desired state cannot be updated at all — reconciliation has nothing new to converge on.
- The controller is healthy, the desired state is valid, and it cannot be reached: no node has room, an image cannot be pulled, a quota is exhausted. Everything reports "working"; nothing progresses.
- An immutable field is changed. The apply is rejected rather than silently reconciled, which is safer but surprises people who expect the loop to handle anything.
- "Reconciliation means the cluster fixes problems." It means the cluster removes differences from your declared state. If your declared state is the problem, it will maintain the problem with great persistence.
- "Applied means deployed." Applied means stored. The distance between the two is where most Kubernetes surprises live.
- "This is unique to Kubernetes." The pattern is not — an autoscaling group reconciles instance count, and an IaC tool reconciles infrastructure when you run it. What is unusual is that it runs continuously and covers most object types (State).
- "Self-healing means high availability." A restarted pod is not a recovered service if it restarts into the same broken dependency, and restarts hide the underlying fault (Probes: Readiness, Liveness and Startup).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
status.observedGenerationequalsmetadata.generation— the controller has seen your change.- The Deployment's
Progressingcondition isTruewith reasonNewReplicaSetAvailable, andAvailableisTrue. Not "the apply succeeded". kubectl rollout statusreturns rather than hanging, and the replica counts in.statusmatch.spec.- Events on the object show the controller taking actions, and stop once converged. A stream of repeated identical events means the loop is running and failing.
- Rollback is not a special mechanism here — it is another desired state. Apply the previous manifest and the same loop converges back.
- That only holds for things the loop owns. Anything with side effects outside the cluster — a migration a job ran, a message published, a row deleted — is not reversed by restoring desired state (Destructive Migrations).
kubectl rollout undoworks because the previous ReplicaSet is still stored, not because the platform remembers what you did.
- Automate the delivery of desired state: render it, review it, apply it from a pipeline, and let the loop do the rest. This is the one place where automation adds almost no new failure surface, because the mechanism was already a loop.
- Keep the decision about what the desired state should be human. A controller that changes replica counts on a signal is fine; a controller that decides a risky change is acceptable is not (The Automation Trap).
- You give up the ability to know when something has finished by watching a command exit. Every operation becomes asynchronous, and verification becomes a separate step you have to build.
- Debugging moves from "read the script" to "work out which of several chained controllers stopped", which is harder for newcomers and needs the mental model this lesson is about.
- Level-triggered convergence makes the system robust and makes it stubborn. Anything you do out of band is undone, including the emergency fix you meant to keep.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICKubernetes runs this loop continuously for nearly every object type. A VM autoscaling group reconciles exactly one dimension — instance count against desired capacity — and nothing else; a PaaS deploy is a one-shot imperative action with a process supervisor restarting crashes; an IaC tool reconciles only during a run a human started, which is why drift accumulates between runs (Drift).
- SIMPLIFIEDShown as one loop. In reality a single
kubectl applyengages a chain of loops — Deployment, ReplicaSet, scheduler, kubelet, endpoint controller, and often an ingress or service-mesh controller — each with its own desired and observed state, each able to stall independently.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — convergence, level-triggered replication and why an eventually consistent control plane is easier to make correct than a transactional one.