Reconciliation
The single idea that makes Kubernetes behaviour predictable: you do not issue commands, you declare a desired state, and a controller loop repeatedly compares it against observed state and acts to close the gap. Everything confusing about the platform is easier once you stop reading manifests as instructions.
This is how Kubernetes works, not what production means. A VM autoscaling group achieves a comparable replacement guarantee through a health check and a launch template, and a PaaS does it invisibly. What the reconciler buys you is that the loop is continuous and the desired state is declared; what it costs you is that nothing is a one-off action, including your manual fixes, which will be reverted.
Desired versus observed
Delete a pod, then run the loop. Notice that the deletion changed nothing on its own.
- — Observed state matches desired state. The controller has nothing to do — which is what the steady state of a reconciler looks like.
- — Deleting a pod does not remove it, it schedules a replacement. To actually reduce the count you change the desired state.
- — A manual change to a managed object is reverted on the next loop, which is the same mechanism working correctly.
- — Convergence is eventual. An apply that returns success means the desired state was recorded, not that anything is serving yet.
The pod is not serving traffic
A decision tree for the other half of the platform: when the loop has done its job and the pod still is not taking traffic.
Two questions, in order, resolve almost all of it: is the container running, and is the pod Ready. Not running splits into never scheduled and starts-then-dies, which have completely different causes. Running-but-not-serving splits into readiness failing and routing not reaching it. Work top-down; the state field tells you which half you are in before you read a single log line.