MigrationGENERALSCALE-SPECIFICCONTESTED
Designing the Migration
Every change to a running system has an initial state, a transition state, a final state and a way back. The transition state is the one people skip, and it is live in production longest.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
The question
I have designed what the system should look like afterwards. What else does a change to a live system need before it is a design?
The requirement
Split the users table's authentication fields into a separate credentials store, so that authentication can be rate-limited and audited independently.
The obvious build
Design the end state, write the migration script, deploy the new code and run the script. The transition is whatever happens between the two commands, and it takes a few seconds.
Why it breaks
The transition does not take seconds. A rolling deploy means old and new code are both live for minutes; a backfill of ten million rows takes hours; the nightly batch job holds a forty-minute view of a schema that is changing underneath it.
How it breaks as requirements change
- The transition does not take seconds. A rolling deploy means old and new code are both live for minutes; a backfill of ten million rows takes hours; the nightly batch job holds a forty-minute view of a schema that is changing underneath it.
- Old code meets new data. The instance that has not been replaced yet reads a row written in the new shape and does something unspecified — usually a null-pointer error, occasionally a silent wrong answer.
- The rollback plan is "revert the deploy", which does not revert the data. After the first credential is written to the new store, the old code no longer has the complete picture (Reversible and Irreversible Decisions).
- As requirements arrive mid-migration, they have to be implemented against a schema that is neither the old one nor the new one, and nobody wrote down what that intermediate shape is or how long it lasts.
Requirement→Constraints→Invariants→Responsibilities→Boundaries→Interfaces→State→Dependencies→Failure→Implementation→Tests→Feedback→Evolution
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
Constraints
- The system serves traffic continuously; there is no window.
- Deploys are rolling, so for several minutes old and new code run simultaneously against the same data (Version Coexistence: N and N+1, in Both Directions in DevOps applies directly).
- Two other services read the
userstable, one of them a batch job that runs nightly and holds a connection for forty minutes. - Rollback must be possible for two weeks, which means the old code must keep working against whatever the new code writes.
Invariants
- Authentication must succeed for every existing user at every moment of the migration, including the minutes when both code versions are live.
- No credential may exist in only one store during a period where either store might be authoritative.
- The system must be releasable and revertible at the end of every step, not only at the end of the sequence.
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
Responsibilities
- The design owns four artefacts, not one: the target, the sequence of intermediate states, the compatibility rules that hold during each, and the recovery path from each.
- Each intermediate state owns a written invariant — what is true while the system is in it — because that is what tells you whether a bug is expected or not.
- Someone owns the duration of the transition state as a number. "As short as possible" is not a duration and does not survive contact with a backfill (Data Migration).
- Someone owns the abort: the specific step at which rolling back stops being cheap, and what happens after it.
Boundaries
- The step boundary is the deploy: each step must be independently deployable, independently revertible, and safe when only half the fleet has it.
- The compatibility boundary is wherever old and new meet — old code with new data, new code with old data, old clients with new servers. Each pairing needs an answer (Backward Compatibility as a Constraint).
- The point of no return is a boundary too, and it should be a deliberate, named step rather than a moment discovered afterwards.
Four states, and the one that gets skipped
Draw the change as a state machine over the system rather than over an entity, and the omission becomes obvious. Most designs contain exactly two of these nodes — the initial state, implicitly, and the final state, in detail — with an unlabelled arrow between them.
The guards matter more than the states. Each one is a condition that must hold before the system is allowed to move on, and most migration incidents are a transition taken without its guard.
Splitting credentials out of the users table
InitialExpandedBackfilledSwitchedContracted ·
| From | On | To | Guard | Effect |
|---|---|---|---|---|
| Initial | deploy dual-write | Expanded | new store schema deployed and old code tolerates its existence | every new or changed credential written twice |
| Expanded | backfill completes | Backfilled | row counts and sampled values match between stores | both stores hold every credential |
| Backfilled | deploy read-switch | Switched | shadow reads from the new store matched the old for a full business cycle, including the nightly job | authoritative reads move; rollback is still a config change |
| Switched | rollback | Expanded | dual-write still active, so users is still complete | reads return to users; no data repair needed |
| Switched | retirement date reached | Contracted | two weeks at Switched with no rollback and no consumer still reading the old columns | dual-write and columns removed; this is the point of no return |
must be impossible
- Initial → Switched — Reading from a store that has not been backfilled means every pre-existing user fails authentication. This is the "deploy the new code and run the script" plan, and the gap between the two commands is an outage for everyone who logs in during it.
- Contracted → Switched — The credential columns are gone; there is nothing to roll back to. Contraction is deliberately terminal, which is why its guard is a rollback window that expired unused rather than a date on a plan.
- Expanded → Contracted — Dropping the old columns while they are still authoritative for reads destroys the data the system is actively serving from. Every path to Contracted must pass through Switched.
Three of these five states are transition states, and the system will spend more calendar time in them than in either end state. That ratio is the argument for designing them (Explicit State).
What actually happens in the unlabelled arrow
The failures below all live in the gap between "deploy the new code" and "run the script". None of them is exotic; each is a direct consequence of two versions of something being live at once.
The common shape is a pairing nobody tested: old code with new data, new code with old data, or a consumer outside the repository with either.
The transition state, and how it bites
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Rolling deploy, half the fleet updated | Intermittent 500s for ten minutes, then they stop | Old instances read a row written in the new shape and had no branch for it | Deploy the reader before the writer, always: tolerate the new shape in one release, produce it in the next (Backward Compatibility as a Constraint) |
| Backfill running over ten million rows | Transition state designed for an hour is still live after nine days | Throughput estimated from a sample; production has hot partitions and a nightly lock window | Rehearse the backfill on a full-size copy and quote its real duration as the compatibility window (Planning a Backfill in Data Engineering) |
| Nightly job holds a 40-minute transaction | Job fails or reads a mixture of old and new shapes | The slowest consumer, not the deploy, sets how long compatibility must hold | Inventory long-running consumers before sequencing; the compatibility window is the max over all of them |
| Rollback after the read-switch | Reverting the deploy does not restore correct behaviour | New code wrote data the old code cannot interpret; the revert only moved code | Keep dual-write active across the read-switch so a revert needs no data repair — this is the whole reason the sequence has five steps |
| Two migrations overlapping in the same table | A bug that reproduces only in one combination of two flags | The transition states multiply rather than add | Serialise migrations in the same area; finish one before starting the next (Incremental Migration) |
Pricing the sequence against the shortcut
The comparison people avoid making is that the safe sequence is more work, and the shortcut sometimes works. Making the trade explicit is more persuasive than insisting on process, and it is also honest about which system you are in.
The variable that decides it is the length of the transition window, and that is dominated by the backfill and the slowest consumer — neither of which is visible from the code being changed.
Splitting credentials out of `users`, two ways
The change
Move authentication fields to their own store so authentication can be rate-limited and audited independently.
One deploy: new code plus a migration script
UserRepositoryAuthServicemigration_014.sql
testsauth_testuser_repository_test
3 modules · 2 test files
One release, half a day of work, and a window of several minutes during a rolling deploy in which old instances read rows they do not understand — plus a rollback that no longer restores correct behaviour once the script has run.
Five states, four deploys, dual-write across the read-switch
UserRepositoryCredentialStoreDualWriteCredentialsAuthServicebackfill_jobshadow_read_report
testsauth_testcredential_store_testold_code_vs_new_data_testbackfill_validationrollback_rehearsal
6 modules · 5 test files
Four releases over about three weeks, each individually safe and revertible, with a named guard before each transition and a rollback that is a config change up to the final step.
what it cost Six times the calendar, dual-write code that is genuinely unpleasant to read, and a period where the codebase has two credential representations and a reader must know which is authoritative. It also constrains the end state: the new store must be shaped so the old code can keep working against
users throughout, which rules out some designs that would be better in isolation. That constraint is the real price of revertibility, and it is not always worth paying — for a system with a usable maintenance window and one consumer, the first column is the right answer (When Design Does Not Pay).How to build it
Most important first.
- Write the four states down before writing code: initial, transition (usually several), final, and recovery. If the transition is one line and the final state is a page, the design is not finished.
- Make every step compatible in both directions: old code must tolerate new data, and new code must tolerate old data, for the whole time both can exist (Expand and Contract).
- Estimate how long each transition state lasts, including backfills and the slowest consumer. That number, not the deploy time, is how long the compatibility rules must hold.
- Design the recovery path per step, and be honest about which step makes rollback stop being a revert and start being a data repair.
- Keep the number of concurrently-live transition states small. Two overlapping migrations in the same area is a combinatorial state space nobody can reason about (Incremental Migration).
- Delete the transition scaffolding as a scheduled step. Dual-write code, compatibility shims and flags that outlive the migration are the residue this design produces if nobody owns removing them (What Technical Debt Actually Is).
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
Cost of the next change
- A change designed with all four states costs more up front — typically three or four deploys instead of one, plus the compatibility code. That is the price and it should be quoted.
- The next change in the same area is cheaper, because the team now has the pattern, the verification and the habit of asking what happens when old meets new.
- A change designed only for its end state costs one deploy and, with some probability, an incident whose cost is the outage plus a data repair. The expected cost of the shortcut is not obviously lower — it is just concentrated differently (The Cost of Change).
- What stays expensive regardless: anything that changes the meaning of stored data rather than its shape. No sequencing makes a semantic change safe; it only makes it observable (Data Migration).
What the recommended approach costs
- Four deploys instead of one is slower, more coordination, and more chances for a step to be forgotten in the middle.
- Bidirectional compatibility code is genuinely ugly — a period where the codebase reads two shapes and writes two shapes — and it will be read by people who do not know why.
- Designing for rollback constrains the target: some end states cannot be reached in a revertible sequence at all, and choosing a slightly worse end state to keep the path safe is a real trade someone has to make deliberately.
What can go wrong
Failure modes
- The transition state is left undocumented, so when something breaks at 3am nobody can say whether the observed behaviour is the migration working or the migration failing.
- A step is deployed that is safe in isolation but not safe when half the fleet is running it — the classic rolling-deploy failure, invisible in any single-version test (Partial Failure).
- The backfill is slower than expected and the transition state, designed to last an hour, lasts three weeks. Everything built assuming "briefly" is now a permanent property of the system.
- Rollback is planned but never rehearsed, and its first execution is during an incident.
- The mitigation fails on its own terms: a compatibility shim written for the transition becomes load-bearing, and removing it later is its own migration.
Dependencies, and their direction
- The transition depends on every consumer of the data, including the batch job and the analyst's query, whether or not they are in your repository (Do We Need a Package for This?).
- It depends on the deploy mechanism: rolling, blue-green and recreate produce genuinely different compatibility requirements, and a design that assumed one breaks under another.
- The recovery path depends on data still being readable by the old code, which is a constraint on what the new code is allowed to write.
Misreads
- "This is just having a migration script." The script is one step in the transition. The design is the sequence, the compatibility rules and the recovery path, and the script is the least interesting of them.
- "Rolling back the deploy is the rollback plan." It is the rollback plan for code. Data written during the transition does not roll back, and the moment the new code writes something the old cannot read, the deploy revert stops being sufficient.
- "The transition is short so it does not need designing." Backfills, slow consumers and long-running jobs routinely stretch a "few seconds" transition into weeks. Estimate it rather than assuming it.
- "We can do the whole thing in one deploy if we take a maintenance window." Sometimes true and worth considering honestly — a window is a legitimate engineering choice with a known cost, not a moral failure. It stops being available the moment the system is expected to be continuously up (Zero-Downtime Migrations in DevOps).
Testing it, and how it ages
What to test, and at which boundary
- Test the pairings explicitly: old code against new data, new code against old data. In practice this means running the previous release's test suite against the new schema, which almost nobody does and which catches most rolling-deploy failures.
- Rehearse the rollback per step, on production-sized data, before the step ships.
- Assert the transition-state invariant in code — a check that fails loudly if a row exists in a state the design says is impossible (Enforcing Invariants).
- Verify the backfill by comparing counts and sampled rows between old and new representations, not by trusting the script's exit code (Validating a Backfill Before You Publish in Data Engineering).
How this design ages
- Transition states are supposed to be temporary and frequently are not. Give each one an expiry date and an owner, and treat an expired one as an incident rather than a nuisance (Revisit Triggers).
- As the system acquires more consumers, the compatibility window lengthens — the slowest consumer sets it — so the same migration gets more expensive over time, not less.
- Teams that do this a few times develop a house sequence and stop redesigning it per migration, which is the point at which migration stops being a project and becomes a routine (Expand and Contract).
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- GENERALThat a live system passes through intermediate states, and that those states have their own invariants, follows from the system not stopping — true of a database schema, a message format, a config key and a mobile app rollout alike.
- SCALE-SPECIFICOn a single instance with a maintenance window, the transition state genuinely is seconds and most of this is overhead. On a rolling fleet with a ten-million-row backfill and a nightly consumer, the transition lasts weeks and is the dominant design problem. The advice is not scale-free and pretending otherwise makes small teams do ceremony for nothing.
- CONTESTEDThe strongest opposing view: for many systems a two-minute maintenance window at 4am is cheaper, simpler and safer than a four-step zero-downtime sequence with bidirectional compatibility code, and the industry's reflexive rejection of windows has produced enormous accidental complexity in systems whose users are asleep at 4am anyway. That is right more often than zero-downtime advocacy admits; it stops being right when the system has global users, external SLAs, or when the window turns out to be too short for the backfill and you discover that at 4:02am.
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
DevOpszero-downtime-migrationsdatabase-migration-safetyrolling-compatibilityrollbackmigration-and-deploy-coupling
Domains that do not exist yet
- — Testing & Reliability Engineering — rehearsing a rollback and measuring blast radius are reliability practices; this lesson only insists that the sequence be designed so that a rollback exists to rehearse.