Plan and Code
The decisions taken before and during writing — change shape, size, reversibility and coexistence — determine how safely it can ship.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What has to be decided before the first line is written for the change to be shippable without heroics?
Most changes that are hard to deploy were hard to deploy from the moment they were designed. By the time the pull request exists, the deployment risk is already fixed, and the delivery pipeline can only reveal it.
Plan the feature, write the code, then work out how to ship it. Deployment is a delivery concern and comes after the engineering.
A change that renames a column and reads the new name in the same commit cannot be rolled back once deployed, because the code and the schema moved together (A Migration and a Deploy Are One Event).
- A change that renames a column and reads the new name in the same commit cannot be rolled back once deployed, because the code and the schema moved together (A Migration and a Deploy Are One Event).
- A change whose old and new versions cannot both run at once forces an all-at-once deploy, which removes every progressive strategy from the table (Version Coexistence: N and N+1, in Both Directions).
- A three-week branch produces a diff nobody can review and a merge that conflicts semantically with work that landed in the meantime (Long-Lived Branches).
- A change with no flag has exactly two states — shipped to everyone or not shipped — so the blast radius is decided before verification starts (Reducing Blast Radius).
- When it breaks in production, a bundled change gives you five suspects and no way to bisect (Change Correlation).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Deployment risk is a property of the change shape, not of the pipeline. The pipeline can slow a risky change down; it cannot make it reversible.
- Two properties decide almost everything. Coexistence: can version N and N+1 run simultaneously against the same data and the same consumers? Reversibility: if N+1 is withdrawn, does the system return to a working state?
- Any change that breaks coexistence forces a stop-the-world deploy. Any change that breaks reversibility means the only recovery path is forward, under time pressure, with a system that is currently wrong (Roll Forward: When Going Back Is the Harder Option).
- Slicing a change restores both properties. A rename becomes: add the new column, dual-write, backfill, read new behind a flag, remove old. Each slice is individually reversible and individually coexistable.
- A feature flag decouples deploy from release, so code can reach production while the behaviour stays off. That converts a deployment event into a configuration change with a much smaller blast radius (Feature Flags: Deploy Is Not Release).
Choosing the shape of the change
This is the plan-time decision that constrains everything downstream. There is no default answer; there is a set of criteria and a cost attached to each option.
You have a change that alters user-visible behaviour and touches persisted data. What shape do you give it?
when Small, additive, coexistable, and easy to reverse by redeploying the previous artifact.
cost Blast radius is everyone at once; you rely entirely on rollback speed and on verification catching it.
when The change touches schema or a contract other services depend on.
cost Several deploys instead of one, and the intermediate states must be designed and kept working (Expand, Migrate, Contract).
when You want deploy and release separated, or you want to expose it to a subset first.
cost A runtime branch, a config surface, and a flag someone must eventually delete (Feature Flags: Deploy Is Not Release).
when High uncertainty about behaviour under real traffic, and a signal exists that would show harm early.
cost The most machinery: flag, cohorts, comparison against a baseline, and someone watching (Progressive Delivery: Exposure as a Dial).
when Genuinely no coexistence is possible — a wire-format break with an unavoidable dual-version period.
cost Downtime or degraded service, plus rehearsal. Rare, and usually a sign an earlier design decision removed the cheaper options.
Slicing a rename, concretely
The canonical example, because it is the most common change that people accidentally make irreversible. One version reads old_name; the next reads new_name. Written as one change, the code and the schema move together and neither can move back alone.
Written as slices, every intermediate state is a working system that both versions can run against. That is the whole trick, and it generalises to any contract change: interfaces widen before they narrow.
commit 1: ALTER TABLE users RENAME old_name TO new_name; code reads new_name deploy: migration runs, then instances roll -> old instances read old_name => errors -> rollback: code goes back, schema does not
1. add column new_name (nullable) -- old code unaffected 2. write both, read old_name -- both versions correct 3. backfill new_name in batches -- reversible, resumable 4. read new_name behind a flag -- flip and unflip freely 5. stop writing old_name, then drop it -- only after step 4 is proven
In the coupled version there is a window — however short — where instances of two versions disagree about the schema, and after the migration there is no state the old artifact can run against. In the sliced version every step leaves a system both versions can serve, so rollback is always "deploy the previous digest" and never "restore from backup".
The planning failures that show up at deploy time
Each of these is a decision made during planning or coding whose bill arrives during the rollout. They are listed by how frequently they cause a rollback rather than by severity.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Schema and code changed in one commit | Errors during the rollout window; rollback leaves the schema ahead | Coexistence broken by design | Slice into expand/migrate/contract; treat the migration as its own release (Expand, Migrate, Contract) |
| Enum or message field added, consumers not updated | Downstream service rejects or drops messages | Producer shipped before consumers could tolerate the new value | Consumers tolerate unknown values first, then the producer emits them (Version Coexistence: N and N+1, in Both Directions) |
| Refactor bundled with a behaviour change | Regression after deploy, unclear which half caused it | Two changes, one deployable unit | Split and redeploy separately; bisect is only possible on separated changes |
| Branch open for weeks | Merge conflicts, then a failure nothing in review predicted | Semantic divergence that textual merge cannot detect (Long-Lived Branches) | Integrate daily behind a flag; keep main the integration point |
| New dependency call added on a hot path | Tail latency rises after deploy; occasional timeouts | A synchronous call with no timeout added where local testing never saw contention | Timeout and fallback before shipping; verify against a baseline on canary |
| No flag on an uncertain change | Full rollback needed for a problem affecting a small cohort | The only available lever was the deployment itself | Flag it; a config flip recovers in seconds where a redeploy takes minutes |
How to do it properly
Most important first.
- Decide the rollout shape at plan time: flagged, progressive, all-at-once, or requires-a-window. Write it in the change description, because it constrains the design.
- Split any change that touches schema into expand, migrate and contract phases before writing code, not after the review comments (Expand, Migrate, Contract).
- Keep changes small enough that a reviewer can hold the whole thing in their head and an operator can identify it as a suspect (Change Size: Why Small Changes Are Safer, and When They Are Not).
- Never bundle a refactor with a behaviour change. When the deploy goes wrong you will not know which half did it.
- Design for the old version to keep working. Additive first, remove later, and only after the old path is provably unused.
- Write the verification plan while planning: which signal will show this working, and what value counts as normal (Verify in Production).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A flag, or slicing that keeps every increment individually reversible. An unsliced, unflagged change has nothing containing it beyond the deployment strategy.
What can go wrong
- A flag that is never removed, so the codebase accumulates dead branches and the flag itself becomes a permanent, untested code path.
- Slicing done in the repository but not in delivery — five commits merged and deployed as one event, which has all the risk of the unsliced change.
- Coexistence assumed rather than tested; the new version writes a field the old version chokes on, and the failure appears only during the rollout window.
- Planning ceremony that produces a document nobody reads, adding delay without changing the shape of the change.
- "Small changes mean small features." It means small deployable units. The feature can be large; the increments that reach production should not be.
- "Feature flags remove deployment risk." They move it. A flag flip is itself a production change with its own blast radius, and it can be wrong (A Config Change Is a Production Change).
- "We will work out the rollback when we need it." The moment you need it is the worst moment to design it.
- "Planning slows us down." Slicing rarely slows down delivery; it slows down the first deploy and speeds up every recovery.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The change description names its rollout shape and its rollback path before implementation starts.
- Each merged commit could be deployed on its own without breaking the version already running.
- You can point at the flag and say who currently sees the new behaviour.
- Reviews are finishing rather than being abandoned — a proxy for changes being the right size.
- Before merge: abandon or reshape the change; cost is only the work done.
- For a flagged change: turn the flag off. This is a config change with a much smaller blast radius than a redeploy (Feature Flags: Deploy Is Not Release).
- For a sliced schema change: roll back to the previous slice, which by construction still works against both schema states.
- For an unsliced coupled change: there is no clean rollback, which is the argument the plan should have made.
- Automate the mechanical checks on change shape — migration linting for destructive statements, diff-size warnings, checks that a new flag has an owner and an expiry.
- Keep human: whether a change is safe to bundle, whether a flag is worth its complexity, and whether the rollback story is honest (The Automation Trap).
- Automate flag hygiene reporting: which flags are fully rolled out, which have not moved in months, which have no owner.
- Slicing multiplies the number of deploys. Five safe deploys take longer in wall-clock time than one risky one, and the team feels that cost immediately while the benefit is invisible.
- Feature flags add a runtime branch, a configuration surface and a testing burden — the flagged-off path is a code path nobody exercises.
- Additive-only schema changes leave the database carrying columns that are no longer read, until someone does the contract step that everyone postpones.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALCoexistence and reversibility are properties of any change to a running system, on any platform. What differs is how long the two versions must coexist — seconds for a small rolling deploy, days for a mobile client you cannot force to upgrade.
- SCALE-SPECIFICOn a single-instance service with a maintenance window, coexistence is not required and this collapses to "take it down and back up". It becomes non-negotiable the moment two instances of different versions can serve the same request.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — designing a change so that the confidence you need can actually be established before it ships.