StrategiesGENERALKUBERNETES-SPECIFIC

Rolling: Two Versions, One Database

Replacing instances in batches keeps the service up — at the price of a window where old and new code run simultaneously against exactly the same state.

What actually happens How to do it properly Evidence and rollback

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

While a rolling deploy is in progress, two versions of my code are live at once — what does that actually oblige me to guarantee?

The problem

To avoid downtime you must have the old version serving while the new one starts. That means, for the duration of the rollout, your system is running two different programs against one database, one cache, one queue and one set of clients.

What teams do first

Rolling is the safe default: instances are replaced gradually, health-checked at each step, and the service never goes down. Set it and stop thinking about it.

How it breaks

The service does not go down, but it does become internally inconsistent. A user's first request goes to v2, their second to v1, and if the two disagree about the shape of a session, a cookie or a cached object, the user sees the disagreement.

How it breaks in production

The service does not go down, but it does become internally inconsistent. A user's first request goes to v2, their second to v1, and if the two disagree about the shape of a session, a cookie or a cached object, the user sees the disagreement.
The mixed window is not brief on a large fleet. A rollout that respects readiness gates and drain time on hundreds of instances can run for a long stretch, and everything must hold for all of it.
The database is not rolled. A migration that v2 requires and v1 cannot tolerate breaks v1 the moment it lands, for however long v1 is still serving — which is most of the rollout (A Migration and a Deploy Are One Event).
Rollback re-enters the same mixed window from the other direction, and now the shared state contains data written by v2 that v1 has never seen.

Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

The platform replaces instances in batches. Two settings govern the shape: how much extra capacity may exist during the rollout, and how much capacity may be missing. Surge-first keeps you safe; unavailable-first makes a failed rollout into an outage (Recreate: Stop Everything, Then Start the New Thing).
Each new instance must pass readiness before traffic reaches it and before the next batch begins. Readiness is therefore the gate that makes rolling safe — and a readiness check that returns healthy before the process can actually serve removes that safety silently (Probes: Readiness, Liveness and Startup).
The crucial consequence: for the duration, every shared surface has two writers and two readers with different code. Database rows, cache entries, queue messages, session data, files, and the API contract as seen by clients mid-session.
This is why backward-compatible migration is not a nicety attached to rolling deploys — it is the precondition that makes them possible at all. If v1 cannot run against the schema v2 needs, then during the rollout your fleet is partly broken by construction (Version Coexistence: N and N+1, in Both Directions).
The obligation runs both ways. v1 must tolerate what v2 writes, because v1 is still serving. v2 must tolerate what v1 writes, because v1 is still writing. And for rollback to be possible, v1 must still tolerate v2's data after the rollout finished.

The mixed window, and what is shared inside it

During a rolling deploy the fleet is not "mostly v1" or "mostly v2". It is both, in front of exactly one copy of everything that holds state. This diagram is the entire lesson: the split at the top is the strategy, the join at the bottom is the problem.

One rollout, halfway through

UserLLMAgentToolDataDecisionHumanGuardrail

Why the migration has to be backward compatible

DATABASE-SPECIFICThe lock behaviour of each statement differs sharply by engine and version — adding a nullable column is cheap on modern PostgreSQL and MySQL but adding one with a default was not always so, and the drop at the end takes a lock whose duration depends on the engine. The four-phase shape is general; check what each statement actually locks on your engine before running it against a live table.

This is the specific consequence people are taught as a rule and rarely shown as a mechanism. The schema is not versioned per instance. It changes once, for everyone, at the moment the migration runs — while most of the fleet is still v1.

So the constraint is not "migrate carefully". It is: the schema must be simultaneously valid for the version being removed and the version being added, for the entire rollout, and for the entire rollback if you need one.

The same rename, done twice

1-- Unsafe during a rolling deploy: one statement, two broken versions.
2-- v1 is still running and still selects old_name.
3ALTER TABLE users RENAME COLUMN old_name TO new_name;
4-- every v1 instance now errors on every read, for the rest of the rollout
5 
6-- Safe: additive first, destructive much later, in separate deploys.
7-- Deploy 1 (schema only) — v1 is unaffected, it does not know the column exists.
8ALTER TABLE users ADD COLUMN new_name text;
9 
10-- Deploy 2 (code) — v2 writes both columns, reads old_name.
11--   v1 still works: old_name is authoritative and still written.
12--   Rollback to v1 works: nothing v1 needs has been removed.
13 
14-- Deploy 3 (backfill, then code) — copy history, then v2 reads new_name.
15--   Rollback still works: old_name is still there and still current.
16 
17-- Deploy 4 (contract) — only once no running or rollback-target version
18-- reads old_name at all:
19ALTER TABLE users DROP COLUMN old_name;
20-- from this moment, rolling back to v1 is no longer a rollback.

The thing to notice is the last comment, not the DDL. The contract step is the point at which your rollback target stops being valid — so it must be a deliberate, separate decision made after you have stopped needing to roll back (Expand, Migrate, Contract).

A rollout, minute by minute, with the compatibility questions attached

Read this as a checklist rather than a story. Every row is a moment where a specific assumption about coexistence is being tested, and the rollback line at the end is where most teams discover they only designed for one direction.

Rolling out v2, then rolling back

T+0changeMigration runs. From here the schema serves both versions — or it does not, and the rest of the timeline is an incident.
T+1mchangeFirst batch replaced. Fleet is now 10% v2, 90% v1, both writing the same tables.
T+2msignalA user's session hits v2 then v1. Anything v2 stored in a new shape must be readable by v1 right now.
T+4msignalv2 enqueues a message with a new field. A v1 consumer picks it up: does it ignore the field, or reject the message (Dead Letter Queues Are an Operation)?
T+9mactionFleet is 60% v2. Error rate is compared against baseline per version, not in aggregate (Canary Analysis: Compared Against What?).
T+14mrecoveryRollout completes. The mixed window closes — except for long-lived connections still held by drained instances.
T+21msignalA defect appears that only shows under full traffic. Decision: roll back.
T+22mactionRollback begins. The mixed window reopens, in the other direction, for another full rollout duration.
T+23msignalv1 instances now read rows that v2 wrote. Everything v2 created during the last 20 minutes must be readable by v1 — this is the guarantee nobody tested.
T+36mrecoveryRollback complete. Total exposure was two rollout durations plus the time to decide, not one.

Durations are illustrative. The structural points are that rollback costs a second full rollout, and that the compatibility obligation is symmetric — v1 must read v2's data, not merely the reverse.

changesignalactionrecovery

How to do it properly

Most important first.

Design every change for a fleet running both versions. The question to ask in review is: what happens if this request hits the old code and the next one hits the new code?
Make schema changes additive and multi-step. Add a column, write both, backfill, read new, and only then remove the old — across separate deploys (Expand, Migrate, Contract).
Version anything serialized into shared state: cache values, queue messages, session payloads. An unversioned cache entry written by v2 and read by v1 is a deserialization error in production (Operating a Cache).
Surge before you remove, so a bad rollout stalls at full capacity instead of draining you to zero.
Keep readiness honest — it must mean "this process can serve a real request", including its dependencies (Probes: Readiness, Liveness and Startup).
Know your rollout duration. It is the length of time your compatibility guarantees have to hold, and it is also roughly your rollback time.

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongOne percent

One testEveryone

What contains it

Contained early by batch size — only the instances already replaced can serve bad code — but the containment evaporates for any change that touches shared state, because a schema or cache change reaches every instance the moment it lands regardless of how few are running new code.

What can go wrong

Failure modes, including of the mitigation

A migration that v1 cannot tolerate lands at the start of the rollout, so the majority of the fleet is broken for the majority of the deploy.
Session or cookie format change: users bounce between versions and are logged out, or see alternating behaviour, until the rollout completes.
Shared cache poisoning — one version writes a value the other cannot deserialize, and the errors persist after the rollout ends until the entries expire.
Readiness that passes before dependencies are connected, so each batch takes traffic it cannot serve and the rollout looks healthy while errors climb.
A rollout that stalls halfway and is left there, so the fleet sits in the mixed state indefinitely — the state everyone assumed was transient.
Long-running requests or streams held open by old instances well past their batch, extending the mixed window past the rollout (Draining: Stopping Without Dropping).

Misreads this invites

"Rolling deploys have no downtime, so they are low risk." They have no downtime and a long window of mixed behaviour, which is a different risk, not an absent one.
"The mixed window is a few seconds." It is the whole rollout, plus any long-lived connections the old instances are still serving.
"Health checks make it safe." Health checks catch instances that fail to start. They do not catch two versions disagreeing about a data format — both versions are perfectly healthy.
"Rollback is instant." Rollback is another rolling deploy. Plan for it to take as long as the deploy did.

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked

Error rate stays flat *during* the rollout, not just after it. A spike that resolves when the last batch lands is the signature of a coexistence problem, and it will happen again on rollback.
Both versions are visible separately in telemetry, by version label, so you can tell which one the errors are coming from (Deploys on the Same Timeline as the Symptom).
Requests are not dropped at batch boundaries: in-flight count reaches zero on an instance before it terminates.

How you get back

Roll back by deploying the previous artifact — which is another rolling deploy, of roughly the same duration, through the same mixed window.
Rollback is therefore not fast. If you need reversal in seconds rather than in the length of a rollout, you need blue/green or a flag, not rolling (Blue/Green: Paying for the Fastest Rollback There Is).
Rollback only works if the old version can still run against the current state. Once a contracting migration has removed what v1 read, rolling back the code does not roll back the schema, and you are rolling forward (Roll Forward: When Going Back Is the Harder Option).

What to automate, and what stays human

Automate the rollout, the readiness gate between batches, and the automatic halt on failing batches. A human watching a progress bar is not a safety mechanism.
Automate the compatibility check where you can: contract tests between versions, and a CI check that a migration is additive.
Keep the decision to resume a stalled rollout human. A stall is information, and continuing past it is a judgement about why it stalled.

What this costs

Zero downtime is bought with permanent design constraints: every change must be coexistence-safe, forever, including the ones where that is inconvenient.
Multi-step migrations mean a schema change takes several deploys and several days instead of one statement, and the intermediate states must be maintained (Zero-Downtime Migrations).
Slow, well-gated rollouts are safer per batch and longer overall, and a long rollout is a long window in which something else can go wrong.

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

GENERALThe coexistence obligation applies wherever more than one instance is replaced gradually, on any platform. What varies is only how the batching is expressed.
KUBERNETES-SPECIFICKubernetes expresses this as a Deployment with strategy.type: RollingUpdate and maxSurge / maxUnavailable, gated by the readiness probe. A VM autoscaling group calls it an instance refresh with a minimum healthy percentage; a managed platform usually does it silently and gives you no settings. The obligation on your code is identical in all three.