The question this answers
What must be true about my code and my schema before replacing instances a batch at a time is actually safe?
The service must be upgraded without a maintenance window and without provisioning a second fleet, on a workload where requests arrive continuously and are answered by whichever instance the load balancer picked.
A continuous transition at roughly constant capacity: instances are replaced in batches, each batch only receiving traffic once it reports ready, with the old version serving throughout.
The progression, and the state that matters
Mechanically it is simple. Take a batch out of the load balancer's rotation, drain its in-flight requests, replace it with the new version, wait for readiness, put it back, repeat. Capacity dips by at most one batch, or not at all if the platform starts the new instance before removing the old one — which is what a surge setting does, at the cost of briefly running above target capacity.
The state that matters is the middle. For the entire duration of the rollout — which on a large fleet with conservative batch sizes is tens of minutes — both versions are serving production traffic, against the same database, the same cache, the same queues, the same object store. A user's first request may be answered by V2 and their second by V1. A message written by V2 may be consumed by V1. A cache entry written by V2 will be read by V1.
Everything difficult about rolling deployment lives in that sentence. The rollout configuration does not check any of it; it checks readiness probes. Compatibility is a property of your change, established at review time, and nothing in the infrastructure will catch you getting it wrong.
t+0 [V1][V1][V1][V1] 4/4 serving. Baseline. t+30s [--][V1][V1][V1] instance 0 drained: removed from the LB, finishing in-flight requests. t+50s [V2][V1][V1][V1] MIXED. 25% of requests answered by V2. Both write to one database. t+80s [V2][--][V1][V1] capacity dips to 3/4 unless a surge instance was started first. t+100s [V2][V2][V1][V1] MIXED. A cache entry written by V2 may now be read by V1. t+150s [V2][V2][V2][V1] MIXED. A queue message enqueued by V2 may be consumed by V1. t+200s [V2][V2][V2][V2] done. The mixed window lasted ~150s. On 200 instances it lasts an hour. # Rollback from t+150s is another rolling deploy in reverse — minutes, not seconds. # Anything V2 already wrote in a new format is still there when V1 comes back.
The compatibility rules, stated concretely
The general rule is that release N+1 must tolerate everything release N produces, *and* release N must tolerate everything release N+1 produces. The second half is the one people forget, because it is not what "backward compatible" usually means. During a rollout the old version is a live consumer of the new version's output.
That turns almost every interesting change into a two-release sequence, usually called expand and contract. Add the new column and write to both; deploy; backfill; switch reads to the new column; deploy; stop writing the old one; deploy; drop it. Three or four releases to rename a column, which sounds absurd until the first time a single-release rename takes checkout down for eleven minutes because half the fleet is writing to a column the other half is reading from.
The same discipline applies beyond the database: an enum value the old version cannot parse, a queue message with a new required field, a cache entry with a changed serialisation format, an API response field the old client-side code assumes is present. Each of these has the same shape and the same fix — add before you require, require before you remove.
1-- Release 1: EXPAND. Add the new column, nullable. Old code ignores it.2ALTER TABLE orders ADD COLUMN customer_ref text;3-- app v2 writes BOTH customer_id and customer_ref; reads customer_id.4-- Safe in a mixed fleet: v1 does not know customer_ref exists.5 6-- Between releases: BACKFILL, in batches, off the hot path.7UPDATE orders SET customer_ref = customer_id::text8 WHERE customer_ref IS NULL AND id BETWEEN $1 AND $2; -- chunked; never one statement9 10-- Release 2: MIGRATE READS. app v3 reads customer_ref, still writes both.11-- Safe in a mixed fleet: v2 wrote both, so v3 always finds a value.12 13-- Release 3: STOP WRITING THE OLD ONE. app v4 writes and reads customer_ref only.14-- Safe: no live version reads customer_id any more.15 16-- Release 4: CONTRACT. Only now, and only once no rollback target needs it.17ALTER TABLE orders DROP COLUMN customer_id;18 19-- The one-release version of this:20-- ALTER TABLE orders RENAME COLUMN customer_id TO customer_ref;21-- ...during which every instance still running v1 throws 'column does not exist'22-- on every write, for as long as the rollout takes.Draining, readiness and the requests nobody counted
A rolling deploy that is compatibility-safe can still drop requests, and the cause is almost always at the two ends of an instance's life. On the way in, a readiness probe that returns healthy before the application can actually serve — before the connection pool is warm, before configuration is loaded — means the load balancer sends traffic to an instance that immediately fails. On the way out, an instance that exits on SIGTERM immediately abandons every in-flight request, and each one becomes a 502.
Both are boring and both are extremely common, which is why they get their own lessons: Liveness vs Readiness for the entry side and Graceful Shutdown: The 502 Spike Nobody Investigates for the exit side. The characteristic signature is a small, sharp spike of 5xx that starts exactly when a deploy starts and stops when it finishes, and that nobody investigates because "it is just the deploy".
The other subtlety is batch size. Small batches mean a long rollout and a long mixed window; large batches mean a short window and a real capacity dip if readiness is slow. The right size is a function of how long an instance takes to become ready — which is why slow-starting applications make every deployment strategy worse. See Startup Time & Cold Start.
| Shared thing | Rule | Symptom when violated |
|---|---|---|
| Database schema | Add before requiring; drop only after no live version reads it | Errors on roughly half of writes, varying by which instance answered. |
| Queue messages | New fields optional; old consumers must skip unknown fields | Messages land in a dead-letter queue, or are silently dropped by old consumers. |
| Cache entries | Version the key or keep the serialisation compatible | Deserialisation errors, or worse, a silently wrong value. |
| API responses | Additive only during the window; no removed or retyped fields | Client errors that depend on which instance served the request. |
| Session and token format | Both versions must accept both formats | Users randomly logged out, proportional to the rollout progress. |
| Feature flag semantics | A flag's meaning must not change between versions | The same flag produces different behaviour depending on the instance. |
Key points
- Rolling replaces instances in batches at roughly constant capacity and with no maintenance window.
- For the whole rollout both versions serve traffic against one database, one cache and one set of queues.
- N+1 must tolerate N's data and N must tolerate N+1's — the second half is the one teams forget.
- Almost every schema change becomes expand, backfill, migrate reads, contract: three or four releases instead of one.
- Compatibility-safe rollouts still drop requests when readiness probes lie or shutdown is not graceful.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The controller marks a batch of instances for replacement and removes them from the load balancer's target set.
- • Existing connections drain for a configured grace period while new ones go elsewhere.
- • New instances start; the readiness probe gates when each becomes eligible for traffic.
- • Once ready, the new instances join the target set and the next batch begins.
- • With a surge setting the new instance is created before the old one is removed, keeping capacity flat at the cost of briefly exceeding it.
- • You own the compatibility review on every change that touches schema, message formats, cache entries or public responses.
- • You own batch size and grace period, tuned against how long your application takes to become ready.
- • You own the migration sequencing, including backfills that must be chunked so they do not lock a hot table. See Managed Databases.
- • You own the rollback plan, which for rolling is another rolling deploy — and which does not undo data already written in the new format.
- • You own the decision to stop mid-rollout, which is the fastest containment available and should be one command.
- • A single-release column rename: half the fleet errors on every write until the rollout completes or is reverted.
- • A new required field in a queue message that old consumers reject, filling a dead-letter queue during the mixed window.
- • A readiness probe returning healthy before the connection pool is established, so each new batch serves errors briefly.
- • A hard exit on SIGTERM dropping in-flight requests at every batch boundary — a 502 spike that correlates exactly with deploys.
- • A rollback that restores the old code but not the data: V2 already wrote records in a format V1 cannot read.
- • A stuck rollout: the new version fails readiness, the controller waits, and the fleet sits half-upgraded until someone notices.
- • Rollout duration scales with fleet size divided by batch size, so large fleets have long mixed windows and need stricter compatibility discipline.
- • Readiness time is the multiplier on all of it; a slow-starting application makes every rollout longer and every rollback slower.
- • Connection churn scales with batch size: replacing many instances at once forces many clients and pools to re-establish connections at the same moment.
- • Database connection count can spike mid-rollout when surge instances coexist with the instances they replace. See Serverless and Database Connections for the same arithmetic in a different setting.
- • A security fix is only fully in effect once every instance carries it; during the rollout the vulnerable version is still serving a shrinking fraction of traffic.
- • Mixed-version windows can mean mixed authorisation behaviour, which makes an authorisation change a poor candidate for a slow rollout.
- • Draining matters for correctness and for security: an abandoned in-flight request may have completed a side effect without returning a result, which is exactly the ambiguity idempotency keys exist to resolve.
- • Essentially free in capacity: one extra batch briefly, or none at all without a surge setting.
- • The real cost is engineering time — the two-release discipline on schema changes is a permanent tax on every migration.
- • A long rollout on a large fleet has an opportunity cost: the mixed window is a period of elevated risk, and it scales with fleet size.
- • Error rate and latency split by version — without that split a mixed-window failure looks like a diffuse partial outage.
- • Rollout progress and readiness failures per batch, so a stuck rollout is noticed by an alarm rather than by a person.
- • 5xx count during the deploy window specifically, compared to the preceding baseline. A recurring small spike is the graceful-shutdown bug.
- • Dead-letter queue depth during and after a deploy, which is where message-format incompatibility surfaces.
- • The signal that lies: aggregate error rate early in the rollout. At one instance in twenty, a completely broken version raises the aggregate by five percent, which looks like noise.
- • Recreate with a short window, when the change is not coexistence-safe and the two-release dance is not worth it. Often the honest answer for internal tools and batch workloads.
- • Blue/green, when rollback speed matters more than capacity cost — although the shared database means the compatibility requirement does not go away. See Blue/Green: Two Environments, One Switch.
- • Canary, when you want the same mechanism with a metrics gate and a smaller initial blast radius.
- • Deploy dark behind a feature flag, so the risky behaviour change is decoupled from the rollout entirely and the rolling deploy carries no semantic change at all.
- • Buys zero downtime at flat capacity; costs a mixed-version window whose length grows with the fleet.
- • Buys simplicity — it is the platform default and needs no extra routing layer; costs a compatibility discipline that nothing enforces automatically.
- • Smaller batches buy a smaller capacity dip and cost a longer window of elevated risk.
What people believe, and what is true
Rolling deployment is zero-downtime.
It is zero-*window*. Requests are still dropped if readiness lies or shutdown is abrupt, and a mixed-version incompatibility produces errors for the whole rollout.
Backward compatible means the new version reads old data.
It also means the old version tolerates new data, because during the rollout the old version is a live consumer of the new version's writes.
Rolling back a rolling deploy undoes the release.
It restores the code. Data written in the new format stays written, which is why contract steps must wait until no rollback target needs the old shape.