Version Coexistence: N and N+1, in Both Directions
Any deploy without downtime runs two versions of your code against one set of state — and rollback runs them in the other order, which is the direction nobody tests.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
For how long, and under what obligations, do two versions of my code have to work at the same time?
A deploy with no downtime means the old version is still serving while the new one starts. The database, cache, queue and clients are shared and unversioned, so for the length of the rollout your system is two programs sharing one state.
The rollout takes a couple of minutes. Anything incompatible during that window will resolve itself as soon as the last instance is replaced.
The window is not brief. It is the whole rollout — longer on a large fleet, longer still with a canary that holds at each step, and unbounded if the rollout stalls or is deliberately paused.
- The window is not brief. It is the whole rollout — longer on a large fleet, longer still with a canary that holds at each step, and unbounded if the rollout stalls or is deliberately paused.
- Some effects outlive the window. A cache entry written by v2 in a shape v1 cannot read keeps failing until it expires. A message v2 published sits in a queue until a consumer takes it. Neither is fixed by the rollout finishing.
- Rollback runs the same mixed state backwards, and now v1 has to read everything v2 wrote for the whole time it was live. That direction is almost never tested, and it is the direction you need during an incident.
- The schema is not part of the rollout at all. It changes once, for every instance simultaneously, usually before any new code is running (A Migration and a Deploy Are One Event).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Two versions coexist behind one address. Every request goes to one of them arbitrarily, so any observable difference between them is a difference the same user can see between two consecutive requests.
- The obligation is symmetric and has two directions people conflate. Backward compatible: the new version must work with data and messages the old version produced. Forward compatible: the old version must tolerate data and messages the new version produces. A no-downtime deploy needs both.
- The forward direction is the surprising one, and it is what makes rollback possible. If v1 cannot read what v2 wrote, then the moment v2 has written anything, rolling back is no longer a rollback — it is a new, untested state transition (Rollback: Only Useful If It Is Actually Safe).
- Compatibility is owed on every shared surface, not just the database: schema, cache serialization, queue message format, session and cookie payloads, files on shared volumes, and the API contract as seen by a client mid-session (Schema Leakage is the application-side view of why these leak).
- The mechanism that makes this tractable is separating the additive change from the destructive one, across separate deploys, with the destructive step taken only once nothing you might roll back to still needs the old shape (Expand, Migrate, Contract).
Every shared surface, and what each version owes the other
Coexistence is usually taught as a database topic, which is why teams get the database right and are surprised by the cache. The obligation applies to everything that outlives a single request and is touched by both versions.
| Shared surface | What v2 must tolerate from v1 | What v1 must tolerate from v2 | How long it lasts |
|---|---|---|---|
| Database rows | Rows without the new column; old enum values; nulls where v2 expects data | New columns it does not know about; new values in existing columns | Until backfill completes, and until the rollback window closes |
| Database schema | The pre-migration shape, if the migration has not run yet | The post-migration shape, immediately and for everyone | Permanent — the schema is not versioned per instance |
| Cache entries | Values serialized by the old code | Values serialized by the new code, possibly in a new shape | Until every entry expires, which outlives the rollout |
| Queue messages | Messages produced before the deploy, still unconsumed | Messages with new or reordered fields | Until the queue drains — potentially long after the rollout (Operating Queues and Scheduled Work) |
| Sessions and cookies | Sessions created by the old version | Sessions created or upgraded by the new version | Session lifetime, which is usually much longer than a deploy |
| Files on shared storage | Files written in the old format | Files written in the new format | Until rewritten — often never |
| API responses to clients | Requests from clients that expect the old contract | Nothing — but a client mid-session sees both versions' responses | Rollout duration for web clients; release cycles for mobile (Deprecation as a Process, Not a Label) |
The same change, staged and unstaged
The staged version is longer, slower and more code. What it buys is that at no point in the sequence is there a state the previous version cannot run against — which means at every point, rollback is still a rollback.
migration: RENAME old_name TO new_name
+ code: read/write new_name
-> deployed together
-> every v1 instance errors immediately
-> and stays broken for the whole rollout
-> rollback restores code, not schema: still brokendeploy 1: ADD COLUMN new_name (nullable) -> v1 unaffected; it does not know it exists deploy 2: code writes both, reads old_name -> rollback to v1 works: old_name still authoritative deploy 3: backfill, then code reads new_name -> rollback still works: old_name still written deploy 4: DROP COLUMN old_name -> only once no rollback target reads it
The right-hand sequence never creates a state that an in-flight or rollback-target version cannot serve. The cost is four deploys, a dual-write period and a cleanup nobody enjoys — paid to keep the rollback option alive at every intermediate step, which is exactly the option you want during the deploy that goes wrong.
The direction nobody tests
Forward compatibility — the old version reading the new version's data — has no natural moment when anyone exercises it. The rollout tests backward compatibility for free, because the new code is reading old data all the way through. The reverse only happens during a rollback, under pressure, for the first time.
- T+0changeAdditive migration applied. Both versions can serve; nothing has been removed.
- T+3mchangeRollout at 40%. v2 begins writing the new column and a new-shaped cache value. Backward compatibility is being exercised continuously and passing.
- T+12mrecoveryRollout complete. Every instance is v2. The mixed window appears closed.
- T+18msignalA queue message written by v2 with a new field is still unconsumed; a v1 consumer would reject it. Nobody is looking, because the rollout is done.
- T+55msignalA defect surfaces under a traffic pattern that only occurs on the hour. Decision: roll back.
- T+56mactionRollback starts. v1 instances begin reading rows and cache entries written during the last hour — the forward direction, for the first time ever.
- T+57msignalv1 deserialization errors on cache values in the new shape. The rollback is now producing its own error rate, on top of the one it was meant to stop.
- T+59mactionCache flushed to clear the incompatible entries. Downstream read load spikes as every request misses (Operating a Cache).
- T+1h04mrecoveryErrors resolve. The rollback took four times as long as the rollout, because half of it was unplanned.
Durations are illustrative of ordering, not measurements. The structural point is the T+57m row: forward compatibility was never exercised until the moment it was load-bearing.
How to do it properly
Most important first.
- Ask the review question explicitly for every change: what happens if this request hits the old code and the next one hits the new code — and what happens if we go back?
- Make schema changes additive and staged. Add, dual-write, backfill, switch reads, and only then remove — each in its own deploy (Expand, Migrate, Contract).
- Version everything you serialize into shared state. A version tag on a cache value or a queue message turns an unreadable payload into a handled case (Operating a Cache).
- Write tolerant readers: ignore unknown fields, do not require fields that were added recently, and never let an unexpected value be fatal (Backward Compatibility: The Real Rules in the API domain is the full set of rules).
- Define how long the old version remains a rollback target, and treat that period as the period during which forward compatibility must hold. Contract only after it expires.
- Test the mixed state deliberately: run old and new against the same database in a lower environment and send traffic to both (Contract Tests Between Services give you the API half of this cheaply).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Only weakly contained. The code half is bounded by how many instances have been replaced, but the state half is not routed at all — a schema or serialization change applies to every instance the instant it lands, which is why a canary at one percent does not contain an incompatible migration.
What can go wrong
- A destructive migration applied at the start of a rollout, so most of the fleet is broken for most of the deploy.
- A cache poisoned with a new serialization format: errors persist past the rollout until entries expire, and reappear on rollback.
- Queue messages produced by v2 that v1 consumers reject, landing in the dead-letter queue and needing reprocessing later (Dead Letter Queues Are an Operation).
- A session or cookie format change that logs users out or alternates behaviour until the rollout completes.
- A rollout paused indefinitely at a canary step, which quietly makes the transient mixed state into the permanent one.
- Forward compatibility assumed rather than tested, discovered during the rollback that was supposed to be the safe option.
- Long-lived connections and streams held by drained instances, keeping the old version live long after the rollout reported completion (Draining: Stopping Without Dropping).
- "Backward compatible means the new version reads old data." That is half of it. The other half — old code reading new data — is what makes rollback work, and it is the half that gets skipped.
- "The mixed window is short." It is the rollout, plus drained connections, plus cache lifetimes, plus anything sitting in a queue. On a paused canary it is indefinite.
- "We tested the migration." Testing that the migration applies is not testing that the previous version still runs against the result.
- "We can always roll back." Only until the contract step. After that, the previous artifact is not a valid program for the current state (Destructive Migrations).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Error rate is flat *during* the rollout, per version, not only after it. A spike that ends when the last batch lands is a coexistence defect that will recur on rollback.
- A deliberate mixed-state exercise has been run: both versions serving from the same database, with traffic, and nothing failed.
- The rollback direction has been demonstrated — the old version reading rows and messages the new one wrote — rather than assumed.
- Rollback is the reason forward compatibility matters, so it is not a separate topic: if the old version cannot read the new version's data, you have no rollback.
- Rolling back code does not roll back the schema. Once the contract step has run, going back requires a new migration, which makes it a roll-forward wearing a rollback's clothes (Roll Forward: When Going Back Is the Harder Option).
- Data written during the mixed window persists in whichever shape it was written. Plan for the old version to encounter it indefinitely, not just during the rollback.
- Automate detection of destructive schema changes in CI — a check that flags a drop, a rename or a narrowing type change is one of the highest-value gates in this domain.
- Automate contract testing between the previous released version and the candidate, in both directions.
- Keep the decision about when it is safe to run the contract step human. It depends on how far back you might need to roll, which is a judgement about risk rather than a property of the code.
- A one-statement change becomes four deploys spread over days, and the intermediate states — dual writes, two columns, tolerant readers — must be maintained and eventually cleaned up.
- Tolerant readers hide genuine errors: code that ignores unknown fields also ignores fields it should have understood.
- Keeping the old shape available for a long rollback window means carrying redundant data and code for that window.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe two-direction obligation holds anywhere versions are replaced gradually — containers, VMs, serverless aliases, or a mobile app where the old version lives on users' devices for months and forward compatibility is measured in release cycles rather than minutes.
- DATABASE-SPECIFICWhich schema operations are cheap and which take a disruptive lock varies sharply by engine and version: adding a nullable column is generally cheap on current PostgreSQL and MySQL, while type changes, adding constraints and dropping columns differ in both lock scope and duration. The staging shape is universal; the cost of each statement must be checked against your engine (PostgreSQL in Production: Connections, VACUUM, Partitioning, Replication for one engine's specifics).
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.