The question this answers
How do I go from "I want three copies of this image running" to three actual pods, and what happens when I change the image?
The API needs three replicas so that losing one machine does not take the service down, and it must be able to move from v7 to v8 without a window in which zero replicas are serving.
A declared replica count that is continuously restored, plus a controlled, reversible transition when the declaration changes — new pods are created and proved healthy before old ones are removed.
Two controllers, one chain
Nobody should create pods directly. A bare pod has no controller behind it, so when its node dies it is simply gone — nothing has an opinion about restoring it. What you create instead is a Deployment: a declaration of a replica count and a pod template.
The Deployment does not create pods either. It creates a replica set: a lower-level controller whose entire job is "keep exactly N pods matching this template alive". The Deployment's job is one level up — it manages *replica sets over time*, which is precisely what a rollout is. Change the image and the Deployment creates a second replica set with the new template, then shifts the desired counts between them: old from 3 toward 0, new from 0 toward 3.
That two-level split is not bureaucracy; it is what makes rollback trivial. The old replica set still exists with a count of zero. Rolling back is not a rebuild — it is setting the old one's count back up and the new one's back down. This is also why "how many replica sets should I have" has a natural answer: one active, plus however many old ones you keep for rollback history.
What a rollout actually does, step by step
The default strategy replaces pods gradually, governed by two numbers: how many pods may be unavailable below the desired count, and how many extra may exist above it. Those two numbers decide whether your rollout needs spare capacity, whether it can ever dip below your redundancy target, and how long it takes.
The step that people skip is the readiness gate. A new pod is only counted toward the desired total once its readiness probe passes — which means a wrong readiness probe silently defeats the entire mechanism. If the probe returns healthy the instant the process starts, the rollout will happily replace all three replicas with three processes that are running and cannot serve, and the Service will route to all of them. Liveness vs Readiness is not a side topic; it is the thing that makes a rollout safe.
A rollout that stalls does not roll back on its own by default. It stops, half-complete, with some replicas on each version, and waits — which is the correct conservative behaviour and also a terrible surprise if you assumed otherwise. The progress deadline exists to turn that silent stall into a visible failed condition; set it, and wire it to your deployment pipeline. See Rolling Deployment and the Compatibility It Demands for the strategy in general and Four Ways to Replace Running Code for when a different one is warranted.
- 1Template changed
The Deployment spec now names
api:v8. A new replica set is created with desired 0.A mutable tag such as
latestmeans the template did not change and no rollout happens at all. - 2Surge one new podseconds
New replica set goes to 1. Cluster now needs capacity for 4 pods, not 3.
No spare capacity means the surge pod is Pending and the rollout stalls before anything has moved.
- 3Wait for readiness10–60s
The new pod must pass its readiness probe before it counts.
A probe that passes too early admits a pod that cannot serve; a probe that is too strict stalls the rollout.
- 4Retire one old podgrace period
Old replica set goes to 2. The retiring pod leaves endpoints and receives the termination signal.
No graceful shutdown handling means in-flight requests are dropped at this exact moment.
- 5Repeat per replica
Surge, gate, retire — once per replica until the old set is at 0.
Version skew: both versions serve simultaneously, so the database schema must satisfy both.
- 6Complete
New replica set at 3, old at 0 but retained for rollback.
If it stalls instead, it stays half-migrated indefinitely unless a progress deadline turns it into a failure.
The declaration, and the fields that decide everything
Here is the same Deployment with the fields that actually matter annotated. Most of a Deployment is boilerplate; four decisions carry the weight. The replica count sets your redundancy floor. The image reference decides whether a rollout is reproducible. The strategy numbers decide whether you need spare capacity and whether you can dip below redundancy. The readiness probe decides whether any of it is safe.
Two traps worth naming explicitly. First, replicas: 3 on a two-node cluster does not give you three-machine redundancy — without an anti-affinity or topology spread rule, all three can land on one node and a single machine loss takes the service down. Second, a mutable tag makes rollback meaningless: rolling back to api:latest rolls back to whatever latest points at now, which is the version you were trying to escape. Pin by digest — see The Container Registry.
1apiVersion: apps/v12kind: Deployment3metadata: { name: api }4spec:5 replicas: 3 # DECISION 1: redundancy floor. Spread across nodes is a separate rule.6 revisionHistoryLimit: 5 # how many old replica sets stay available for rollback7 progressDeadlineSeconds: 600 # turns a silent stall into a visible Failed condition8 strategy:9 type: RollingUpdate10 rollingUpdate:11 maxUnavailable: 0 # DECISION 3: never dip below 3 serving replicas...12 maxSurge: 1 # ...which means the cluster needs room for a 4th13 selector:14 matchLabels: { app: api }15 template:16 metadata:17 labels: { app: api }18 spec:19 topologySpreadConstraints: # without this, all 3 replicas may land on one node20 - maxSkew: 121 topologyKey: kubernetes.io/hostname22 whenUnsatisfiable: ScheduleAnyway23 labelSelector: { matchLabels: { app: api } }24 containers:25 - name: api26 image: registry.example/api@sha256:9f2c... # DECISION 2: digest, not a mutable tag27 readinessProbe: # DECISION 4: the gate the rollout waits on28 httpGet: { path: /readyz, port: 8080 }29 initialDelaySeconds: 530 periodSeconds: 531 terminationGracePeriodSeconds: 30 # your app must actually use this windowKey points
- A Deployment manages replica sets over time; a replica set keeps exactly N pods matching one template alive. The split is what makes rollback cheap.
- Rolling back is not a rebuild — the old replica set still exists at zero, and rollback just moves the counts back.
- A rollout advances only as new pods pass readiness, which means a wrong readiness probe defeats the entire safety mechanism.
- A stalled rollout stops half-migrated and waits; without a progress deadline that failure is silent.
replicas: 3is not three-machine redundancy unless a spread or anti-affinity rule says so, and a mutable image tag makes rollback meaningless.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • You declare a replica count and a pod template on a Deployment object.
- • The Deployment controller creates a replica set whose selector matches that template, and sets its desired count.
- • The replica-set controller creates pods until the count is met, and recreates any pod that disappears.
- • When the template changes, the Deployment creates a second replica set and shifts desired counts between old and new, bounded by maxUnavailable and maxSurge.
- • Each new pod counts toward the target only after its readiness probe passes; each retired pod is removed from endpoints and then signalled.
- • The replica count, which is a redundancy decision and needs a spread rule to actually deliver machine-level redundancy.
- • Probe correctness — the readiness endpoint should reflect whether the process can serve, including reachable dependencies.
- • Graceful shutdown in the application, since the rollout will terminate pods under live traffic.
- • Schema and API compatibility across versions, because a rolling update guarantees a window where both versions run at once.
- • Rollback rehearsal: a rollback path you have never exercised is not a rollback path.
- • A rollout stalls at one new replica because the image tag is wrong or the registry credential expired; the old version keeps serving and nobody notices for hours.
- • Insufficient capacity with
maxSurge: 1: the surge pod is Pending, the rollout cannot proceed and cannot complete. - • A readiness probe that always passes: all replicas are replaced with processes that cannot serve, and the Service routes to every one of them.
- • Version skew breaking the database:
v8writes a columnv7does not know about, and the survivingv7replicas start erroring mid-rollout. - • All replicas on one node: a single machine loss takes down a service that was declared with three replicas.
- • Replica count scales throughput linearly only while the shared dependency behind it — usually the database connection pool — has room.
- • Rollout duration is roughly replicas × (startup + readiness delay + grace period), which is why large fleets take a long time to release.
- • Scaling up and rolling out at the same time compete for the same cluster capacity, and the rollout is the one that stalls.
- • The Deployment spec is where the image reference, the service account and the security context are decided — it is a security artifact, not just a runtime one.
- • Pinning by digest is a supply-chain control: it makes the running artifact identical to the reviewed one — see The Infrastructure Supply Chain.
- • Permission to update a Deployment is permission to run arbitrary images with that workload's identity; treat it as production access.
- • A pod template that mounts a broadly scoped service-account token gives every replica that privilege — see Least Privilege in Infrastructure.
- • Cost is replicas × per-pod requests, so replica count is a direct, linear cost lever and one of the easiest places to overpay.
- •
maxSurgerequires headroom that exists only to make rollouts possible — real capacity, billed continuously, used for minutes a week. - • Retained replica-set history is nearly free in money and valuable in recovery time; keeping a few revisions is the cheap side of a trade.
- • Desired, updated, ready and available replica counts — four numbers that together describe exactly where a rollout is.
- • Rollout duration and stall conditions, alerted on the progress deadline rather than watched by a human during a deploy.
- • Per-pod restart counts immediately after a rollout, which is where a bad release announces itself first.
- • The signal that lies: "the deploy succeeded". The command returned; it does not mean the new pods are serving correctly, only that the API accepted the change.
- • A DaemonSet when you need one pod per node rather than N pods total — a node agent is not a replica count problem.
- • A Job or CronJob for work that ends; a Deployment will restart a completed process forever, which is a common and confusing misuse.
- • An autoscaling instance group with a rolling update policy, if you are not otherwise in a cluster — the same guarantees at machine granularity.
- • A PaaS release, which gives you rolling updates and rollback with no strategy fields to get wrong.
- • Buys a declared replica count and a reversible rollout; costs a template that must be correct, because it is applied to every replica identically.
- • Buys gradual replacement with no downtime; costs a mandatory window where two versions serve simultaneously, which your data layer must tolerate.
- • Buys cheap rollback via retained replica sets; costs nothing much — this is one of the genuinely good trades in the platform.
What people believe, and what is true
A Deployment creates pods.
It creates a replica set, which creates pods. That indirection is what makes rollback a count change rather than a rebuild.
A failed rollout rolls back automatically.
By default it stalls half-migrated and waits. Automatic rollback is something your pipeline does after the progress deadline marks it failed.
Three replicas means surviving a machine failure.
Only if they are on three machines. Without a spread or anti-affinity rule the scheduler is free to place all three on one node.