Expand, Migrate, Contract
The pattern that makes schema change safe: add the new shape, move to it, and only remove the old shape in a later deploy.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How do I change a schema when two versions of my code are running against it at the same time?
A schema change and a code change cannot be applied simultaneously across a fleet. Whichever goes first, there is a window in which the schema and the running code disagree — and any single-step change guarantees that window contains something broken.
Rename the column and update the code in the same pull request. The migration runs, the deploy follows, and it is one atomic change from the team's point of view.
It is atomic in the repository and not in production. Between the migration and the last replaced instance, old code is querying a column that no longer exists — every request touching that table fails.
- It is atomic in the repository and not in production. Between the migration and the last replaced instance, old code is querying a column that no longer exists — every request touching that table fails.
- Rolling back the deploy does not help: the schema is already renamed, so the old artifact fails the same way it did during the rollout.
- Rolling back the schema does not help either, if the new code has already written rows under the new name.
- The window is not brief. A rolling deploy across a real fleet takes as long as it takes, and a stuck rollout extends it indefinitely (Rolling: Two Versions, One Database).
- Deploying the code first inverts the problem rather than solving it: new code queries a column that does not exist yet, and the canary fails immediately.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The pattern replaces one incompatible change with a sequence of individually compatible ones. Each step leaves a schema that both the currently deployed code and the next version can operate against.
- Expand. Add the new shape alongside the old. Nothing reads it yet. This step is additive, so no running code is affected and its rollback is "leave it there".
- Deploy compatible code. Ship code that writes both shapes and still reads the old one. After this deploy, every write is present in both places; historical rows are not.
- Backfill. Copy the historical rows into the new shape, batched and throttled, while dual writes keep new rows current (Backfills).
- Switch reads. Move reads to the new shape, ideally behind a flag so the switch is reversible without a deploy (Feature Flags: Deploy Is Not Release).
- Verify. Confirm the new shape is complete and that the switch changed nothing observable.
- Contract. Stop writing the old shape, then remove it — in a later deploy, never during the rollout that introduced the new one.
- The property that makes it work is that at no point does the currently running code depend on something the schema does not have, and at no point does the schema forbid something the running code does.
Six steps, and the two that must not share a deploy
The steps are unremarkable individually. The discipline is entirely in the boundaries between them: which ones may share a release and which must not.
One rule carries most of the safety: contract happens in a later deploy than the switch, never in the same rollout. Everything else is bookkeeping.
- 11. Expand
Add
new_name, nullable, no default. Nothing reads or writes it.fails by Adding it
NOT NULLor with a volatile default, turning an additive change into a rewrite.evidence Column exists; no change in error rate or latency.
- 22. Deploy dual-write code
Every write path writes both columns; all reads still use
old_name.fails by A write path nobody remembered — an admin tool, an importer, a consumer of a queue.
evidence Count of rows with
new_namepopulated rises with write volume, from every path. - 33. Backfill
Copies historical rows into
new_namein batches, throttled against replication lag.fails by One statement over the whole table; no resume point after a failure (Backfills).
evidence Zero rows remain where
old_nameis set andnew_nameis not. - 44. Switch reads
Reads move to
new_name, behind a flag so the switch is reversible in seconds.fails by Shipping the switch as a deploy, so reverting it needs a rollout during an incident.
evidence Endpoint error rate and latency unchanged against baseline; the flag has been flipped back and forth once, deliberately.
- 55. Verify
Confirms nothing still reads or depends on
old_name, across every deployable and every scheduled job.fails by Verifying by reading code instead of by measuring the running system.
evidence A counter on the old read path has been zero for longer than the lifetime of your longest-lived job (Cron Jobs in Production).
- 66. Contract
Stop writing
old_name, then drop it — in a separate, later release.fails by Shipping it with step 4. This is the failure the whole pattern exists to prevent.
evidence Column gone; error rate flat; and you already know nothing referenced it, because step 5 proved it (Destructive Migrations).
Steps 1–2 may ship together only if the migration runs strictly before the new code starts. Steps 4 and 6 must not, under any circumstances, share a release.
The same rename, as production experiences it
Written as a sequence of releases, the pattern looks slow. Written against a clock, it becomes obvious why: the expensive resource being managed is the time during which two code versions coexist, and that time is not yours to compress.
- Release 1changeMigration adds nullable
new_name. No code changes. Additive, inert, reversible by ignoring it. - Release 1 + rolloutsignalNothing observable happens. This is the correct outcome of an expand step.
- Release 2changeCode deploys that writes both columns and reads
old_name. During the rollout, some instances write one column and some write two — both states are valid. - Release 2 + rolloutsignalDual-write coverage metric climbs to 100% of writes as the last old instance is replaced.
- Backfill windowactionBatched backfill runs against live traffic, throttled on replication lag. It can be paused, resumed, and re-run; new writes are already covered by dual writes.
- VerificationsignalCompleteness query returns zero remaining rows. A sampled consistency check finds no divergence between the two columns.
- Release 3changeReads switch to
new_namevia a flag, ramped rather than flipped.old_nameis still written and still correct, so the flag is a real revert path. - SoaksignalThe old read path counter goes to zero and stays there, across at least one full cycle of every scheduled job.
- Release 4changeDual writes removed.
old_nameis now stale but still present — deliberately, for one more release. - Release 5recovery
DROP COLUMN old_name. Irreversible, and by now boring.
The gap between release 3 and release 5 exists so that a rollback to release 2 remains possible for as long as anyone might want one. Contracting early does not make the change finish sooner; it makes the rollback path disappear while you still need it.
What people actually ship instead
Almost every migration incident is one of two collapses of this sequence: doing it in one step, or doing steps 4 and 6 together. Both look like reasonable simplifications in review, because a diff does not show deploy boundaries.
migration: ALTER TABLE users RENAME old_name TO new_name; code: reads and writes new_name deploy: t0 migration applies t0 every running instance is old code -> queries old_name -> ERROR t1 first new instance starts serving correctly t2 ... rollout continues, error rate proportional to remaining old instances t3 rollout completes, errors stop rollback? old artifact fails against the new schema too.
release 1 add new_name (nullable) -> nothing reads it release 2 write both, read old_name -> both states valid backfill batched, throttled, resumable -> no schema change release 3 read new_name behind a flag -> revert = flip flag release 4 stop writing old_name -> still present release 5 drop old_name -> irreversible, and safe
Not because it is tidier — because in the second version there is no moment at which the deployed code and the schema disagree. The error window in the first version is exactly the duration of the rollout, and its rollback path is broken in both directions.
How to do it properly
Most important first.
- Write the whole sequence down before starting, including the contract step and who will do it. The unfinished contract is the most common outcome of this pattern.
- Make every step independently deployable and independently revertible. If a step needs the next one to be safe, it is not a step.
- Put the read switch behind a flag rather than a deploy. Switching reads is the step most likely to reveal a problem, and a flag reverts in seconds (Feature Flags: Deploy Is Not Release).
- Prefer dual writes in application code over database triggers: the application version is visible in review, deploys with the code, and rolls back with it.
- Let the compatibility window be long. There is no prize for contracting the same day, and there is a real penalty for contracting while an old instance is still alive.
- Before contracting, prove nothing reads the old shape — with a metric or a log line emitted at the old read path, not by reading the code (The Audit Trail).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The pattern itself is the containment. Each step is either additive (no effect on running code), reversible by a deploy, or reversible by a flag — except contract, which is contained only by the evidence you gathered before running it.
What can go wrong
- Contract shipped in the same release as the read switch, which recreates the exact incompatibility the pattern exists to prevent.
- The dual-write path writes only one shape on some code path — a background job, an admin tool, a batch importer — so the new column is silently incomplete (Background Jobs and Workers).
- The backfill completes, but rows written during the backfill by a code version deployed before the dual-write release are missing. The verification query has to cover the whole table, not just the backfilled range.
- The sequence stalls after the read switch. The old column stays forever, new engineers cannot tell which is authoritative, and both are half-maintained.
- A rollback to a pre-dual-write version during the window, which stops populating the new shape without anyone noticing until the contract step fails its verification.
- Using a trigger for dual writes and forgetting it exists — it survives application rollbacks, which is exactly what makes it hard to reason about.
- "Expand/contract means adding a column and dropping the old one in the same migration." That is the incompatible change with extra statements. The pattern is about deploy boundaries, not about SQL.
- "We use blue/green deploys, so we do not need this." Blue/green gives you two application versions against one database, which is the compatibility problem stated more sharply, not solved (Blue/Green: Paying for the Fastest Rollback There Is).
- "The backfill is the hard part." The backfill is the slow part. The hard part is knowing when nothing reads the old shape any more.
- "Contract is optional." Skipping it leaves a schema that lies about its own intent, and the next engineer has to reconstruct which column is authoritative from application code.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- After the dual-write deploy: a count of rows where the new shape is populated rises monotonically with new writes, from every write path.
- After the backfill:
SELECT count(*) FROM t WHERE new_col IS NULL AND old_col IS NOT NULLreturns zero, run to completion rather than sampled. - A consistency check comparing old and new shape on a sample of rows, run repeatedly during the window, not once.
- Before contracting: the counter on the old read path has been zero for longer than your longest-lived deployable artifact, including batch jobs that run weekly.
- After the read switch: error rate and latency for the affected endpoints unchanged against the pre-switch baseline (Canary Analysis: Compared Against What?).
- Expand: nothing to roll back. An unused column is inert.
- Dual-write deploy: ordinary application rollback. The new column stops being populated, which is safe because nothing reads it yet.
- Backfill: idempotent by construction, so a rollback means stopping it and re-running later.
- Read switch: flip the flag. This is the reason the switch should be a flag and not a deploy.
- Contract: no rollback. This is the one irreversible step in the sequence, which is precisely why it is last, separate, and gated on evidence rather than on confidence (Destructive Migrations).
- Automate the backfill: batching, throttling, checkpointing and resumption are exactly the kind of repeatable work that should not be a person running a loop in a terminal.
- Automate the verification queries and put them on a dashboard, so "is the new column complete" is a lookup rather than an investigation.
- Keep human: the decision to switch reads, and the decision to contract. Both depend on evidence about what is still running, which is a judgement about your own fleet (Manual Production Changes).
- A one-line rename becomes six steps across at least three deploys, and for a large table the backfill can span days.
- The schema is in an intermediate state for the whole window: two columns meaning the same thing, and an ambiguity for anyone reading the schema without the context.
- Dual writes cost a little write throughput and add a code path that can diverge.
- The pattern trades a short, sharp risk for a long, low one — and long low risks are the kind organisations forget to finish.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe pattern is independent of engine and of ORM. It follows from one fact that holds everywhere: schema and code deploy separately, so every incompatible change needs a compatible intermediate state.
- DATABASE-SPECIFICOnly the cost of each step is engine-specific. Adding the new column is metadata-only on modern PostgreSQL and MySQL for a nullable column; the backfill's cost profile differs because PostgreSQL's MVCC writes a new row version per update and needs vacuum afterwards, whereas InnoDB updates in place and grows undo instead (UPDATE, DELETE and Dead Tuples).
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.