Rolling Deployments
Replacing instances a few at a time keeps the service up, and guarantees that two versions of your code run against one database at the same time.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What must be true about my code for it to be safe to replace instances gradually?
Deploy the new version without a maintenance window and without a capacity dip, and be able to stop halfway if it looks wrong.
The platform replaces old instances with new ones a few at a time. As long as the new version passes tests, the rollout is a deployment detail with no application consequences.
The new version renames a column in a migration that runs before the rollout, and every remaining old instance starts throwing on a column that no longer exists — a full outage lasting until the rollout completes (Expand and Contract Migrations).
- The new version renames a column in a migration that runs before the rollout, and every remaining old instance starts throwing on a column that no longer exists — a full outage lasting until the rollout completes (Expand and Contract Migrations).
- The new version writes a job payload with a new field shape; old workers consume it and crash or silently drop data. The queue is a version boundary too (Job Queues).
- A user's two requests land on different versions and the UI shows contradictory state, because the API response shape changed mid-rollout.
- Instances are replaced faster than the new ones can warm up, so the surviving instances absorb full traffic on a cold pool and the p99 spikes for the duration of every deploy.
- Old instances are terminated without drain, so the rollout itself is a source of 502s regardless of whether the new code is correct (Graceful Shutdown).
What is actually happening
- A rolling deploy replaces instances in batches. At any moment during the rollout the fleet contains both versions, both receiving live traffic, both connected to the same database, the same cache and the same queues.
- Two parameters control it: how many extra instances may exist above the desired count (surge), and how many may be missing below it (unavailable). Surge costs capacity; unavailable costs headroom.
- The platform decides an instance is "done" when its readiness probe passes. If readiness is shallow — a route returning 200 without checking anything — the rollout proceeds through broken instances at full speed (Health Checks: Startup, Readiness, Liveness).
- Rollback is another rolling deploy in the opposite direction. It takes the same amount of time, which is why "we can just roll back" is a slower safety net than people expect.
- The shared substrate is the real constraint. Code deploys gradually; the database schema, the cache key format, the queue message format and any shared file layout do not — they change instantaneously for everybody.
- Compatibility must therefore hold in both directions: new code must work with data written by old code, and old code must work with data written by new code, for the entire rollout and for as long as rollback remains possible.
The two-version window is the actual subject
The mechanics of a rolling deploy are simple and mostly handled for you. The engineering content is entirely in one consequence: for the length of the rollout, your service is two programs sharing one database, one cache and one set of queues.
Everything in this module — expand-contract, canary, blue-green — is a different answer to that same fact. Rolling says "make the versions compatible". Blue-green says "do not overlap them, and accept the cost". Canary says "overlap them deliberately, with a small blast radius".
Batching, surge and the shape of the rollout
Two knobs decide what a rollout costs. Surge allows temporary extra instances so capacity never dips; unavailable allows temporary missing instances so no extra capacity is needed. Setting both to non-zero is the common default and the one that quietly reduces capacity during every deploy.
The third, unofficial knob is time. A rollout that pauses between batches gives your health gate — and your on-call engineer — a chance to observe the new version under real traffic before it is the only version left.
What are you optimising the rollout for?
when Capacity is tight and a dip would cause queueing.
cost Needs room for extra instances; rollout is slower.
when Low-risk change, strong pre-production confidence, need speed (a security fix).
cost The two-version window is short but you learn nothing before it closes.
when Behaviour change, new query patterns, anything touching the database.
cost Long rollout, long overlap; needs per-version metrics to be worth it.
when High-risk change on a critical path, or a service with few instances.
cost Human in the loop; does not scale to many services.
when You want real-traffic evidence before committing the fleet.
cost Requires per-version routing and metrics (Canary Deployments).
The compatibility checklist nobody writes down
Before a change ships, one question answers most of the risk: if the old version is still running, does this break it? And the reverse: if this version is rolled back, does the data it wrote break the old one?
The table is the checklist. It is worth reading against every change that touches a shared substrate, which is more changes than it feels like.
| Change | Old version sees | Safe in one release? | Do this instead |
|---|---|---|---|
| Add a nullable column | Ignores it | Yes | — |
| Add a NOT NULL column with no default | Insert fails | No | Add nullable, backfill, add the constraint later (Expand and Contract Migrations). |
| Rename a column | Column missing: every query throws | No | Add the new one, dual-write, migrate reads, drop later. |
| Drop a column | Query throws if still selected | No | Stop reading it in one release, drop it in a later one. |
| Add an API response field | Not returned by old instances | Yes, if clients tolerate absence | Never make a client depend on a field mid-rollout. |
| Change a response field's type | Client sees both types | No | Add a new field; deprecate the old (Backward Compatibility: The Real Rules in API Design). |
| Change a queue payload shape | Old worker crashes or drops data | No | Version the payload; consumers tolerate unknown fields first. |
| Change a cache key format | Cold cache; both formats coexist | Yes, but expect a stampede | Change the key prefix so old entries are ignored, not misread (Cache Stampede). |
| Change a cache *value* shape at the same key | Old version deserializes new data | No | New prefix. Never reuse a key across value shapes. |
How to build it
Most important first.
- Make every change backward and forward compatible for one release. Add before you read, read before you write, remove only after nothing writes (Expand and Contract Migrations).
- Never deploy a schema change and the code that requires it in the same step. Separate them into distinct releases with a completed rollout between.
- Version your queue message payloads and tolerate unknown fields in consumers, so a producer that ships first does not break a consumer that ships second.
- Make readiness mean something: dependencies reachable, config validated, pool built, migrations at the expected version. A rollout is only as safe as its readiness check (Validate at Startup, Fail Loudly).
- Set surge and unavailable so capacity never dips below what current traffic needs.
maxUnavailable: 0with a surge of one batch is the safe default when capacity is tight. - Bound the rollout with a health gate: automatic pause or abort if error rate or latency crosses a threshold during the deploy.
- Keep the rollout slow enough that a problem is visible before the last old instance is gone. A rollout that completes in 20 seconds gives you no decision point.
- Decouple release from deploy for risky behaviour: ship the code dark behind a flag and turn it on separately, so the rollout and the behaviour change are two independently reversible events (Feature Flags: Rollout, Kill Switches and Debt).
What can go wrong
- A migration in the container entrypoint, so every instance races to run it simultaneously at startup (Schema Migrations from the Application Side).
- Rollback blocked because the new version already wrote data the old version cannot read — the deploy became one-way without anyone deciding that.
- Health gate configured on a metric that lags by minutes, so the rollout finishes before the signal arrives.
- The rollout stalls because new instances never pass readiness, leaving a partially-deployed fleet indefinitely. Without a timeout and automatic abort, this is a silent half-outage.
- Cache entries written in the new format read by old instances. The cache is shared mutable state across versions, and it is the most commonly forgotten one (Cache Invalidation).
- Both versions serve concurrently for the entire rollout: any read of data the other version wrote is a cross-version compatibility test running in production.
- Two instances starting simultaneously can both attempt a migration. Migrations need an advisory lock or a separate one-shot job, never a startup hook (Schema Migrations from the Application Side).
- A request handled by the new version can be retried by the client onto an old version, so retry paths must be compatible in both directions (Idempotency in Backends).
- A job enqueued by the new version can be consumed by an old worker, and vice versa, for as long as the queue has depth — which can outlast the rollout (Queue Backlog).
- A rollout is a window where an old version with a known vulnerability is still serving. For security fixes, rollout speed is part of the fix.
- Both versions must enforce the same authorization rules. Shipping a permission tightening gradually means the loose version is live for the whole rollout — acceptable, but it should be a decision.
- Config and secret changes propagate on instance replacement, so a rotated credential must be accepted by both versions during the overlap (Secrets Are Not Configuration).
- "Rolling means zero downtime." It means no *capacity* downtime. If the two versions are incompatible, it means a partial outage that lasts exactly as long as the rollout.
- "The tests passed, so the deploy is safe." Tests run one version against one schema. A rolling deploy runs two versions against one schema, and almost nobody tests that configuration.
- "We can roll back instantly." A rollback is a full rolling deploy, and it is impossible at all once new data exists that the old code cannot read.
- "Only the database has this problem." Caches, queues, object-storage layouts and search indexes are all shared across versions and all change instantaneously.
Operating it
- Label every metric, log and span with the version or image digest. Without it, "the new version is slower" is unprovable while the rollout is happening (The Metrics a Backend Must Emit).
- Plot error rate and latency split by version during the rollout. A per-version panel turns a deploy decision from an argument into a reading.
- Emit deploy markers to the dashboards so the correlation between a change and a graph is visible rather than inferred (Deploys Are the First Suspect).
- Alert on rollout duration. A stalled rollout produces no errors and no page, and can sit half-finished for hours.
- More instances means a longer rollout at the same batch percentage, which means a longer two-version window and more time for a compatibility defect to matter.
- At high instance counts, per-instance startup cost — image pull, warmup, pool construction — dominates rollout time (Containerizing a Backend).
- With many services deploying independently, the two-version problem becomes a cross-service problem: your caller may be one version behind for hours, so API compatibility is the same discipline at the network boundary (Backward Compatibility: The Real Rules in API Design).
- Rolling deploys give you no capacity spike and no separate environment, and pay for it by making every change a compatibility exercise. Blue-green trades the opposite way (Blue-Green Deployments).
- A slow rollout is safer and lengthens the window in which two versions coexist. Both directions have a real cost; the balance depends on how confident your health gate is.
- Expand-contract discipline means two or three releases where one would have done, and a period where the schema carries both shapes.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALAny gradual replacement strategy — orchestrator rollout, instance-group update, manual instance-by-instance restart — creates the same two-version window.
- CLOUD-SPECIFICSurge/unavailable semantics, health-gate behaviour and automatic rollback differ by platform: some abort and revert on a failed health gate, others simply stop and leave the fleet split. Know which yours does before you need it.
- SCALE-SPECIFICWith two instances the overlap is seconds and compatibility bugs may go unnoticed; with two hundred it is many minutes and every incompatibility becomes a visible incident.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.