Rollback: Only Useful If It Is Actually Safe
Going back to the previous version is the fastest way to end user impact — until the change made the previous version invalid, at which point what you are doing is not a rollback.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Can I actually go back, how long does it take, and what does going back fail to undo?
When a change causes user impact, the fastest way to stop the impact is to stop running the change. That is true only if the previous version is still a valid program for the current state of the system, and there are several ordinary changes that quietly make it invalid.
Rollback is redeploying the previous artifact. The pipeline has a button; we press it if something goes wrong.
The button reverses the artifact. It does not reverse the schema, the config, the flags, the data written since, or anything sent to a third party.
- The button reverses the artifact. It does not reverse the schema, the config, the flags, the data written since, or anything sent to a third party.
- A rollback across a destructive migration is not a rollback. The old code will run against a schema missing what it reads, and the result is a second outage on top of the first (Destructive Migrations).
- Rollback time equals rollout time for a rolling deploy, which people discover while user impact continues. For a large fleet with health gates, that is not fast.
- The previous artifact may not exist any more — pruned by retention, or a mutable tag that now points at the thing you are trying to escape (Artifact Retention, Tags Versus Digests).
- Rolling back to a version that is itself unhealthy under current conditions — because a dependency has since changed, or a certificate expired, or configuration moved on — produces an incident with two causes.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A rollback is a deployment whose target happens to be older, so it inherits every property of your deployment strategy: its duration, its mixed-version window, its readiness gating and its failure modes (Deployment Strategies).
- It is valid exactly when the previous version is still correct against the *current* state: current schema, current config, current data shapes, current dependency versions. Every one of those is a way for the previous version to stop being valid without anyone deciding it should.
- The property that makes rollback possible is forward compatibility — old code tolerating new data — which is the direction that gets exercised only during rollbacks (Version Coexistence: N and N+1, in Both Directions).
- Speed comes from the reversal layer, not from the word. A router change is seconds; a rolling redeploy is a rollout; a flag flip is immediate; an infrastructure change may have no reversal at all (Destructive Changes: What a Rename Really Does).
- What is never reversed: rows written, messages published and consumed, emails sent, webhooks delivered, payments taken, caches populated, and anything a third party has already acted on.
Four things called rollback, with very different guarantees
When an incident channel says "roll it back", it can mean any of these, and the difference matters in the first minute. Establishing which one is available for the change in question is work that should already have been done.
How are we going to stop the new behaviour?
when The behaviour is flagged and the off path still works.
cost Seconds, no pipeline, and no reversal of anything the feature wrote. Requires the off branch to have been exercised recently enough to trust (Feature Flags: Deploy Is Not Release).
when A blue/green or weighted setup is in place and the previous version is still running and healthy.
cost The fastest infrastructural reversal, and it needs the previous environment kept alive — which costs money every day it is not needed (Blue/Green: Paying for the Fastest Rollback There Is).
when The ordinary case: the previous digest is retained and still valid against current schema and config.
cost A full rollout duration, during which impact continues, plus a mixed-version window in the reverse direction (Version Coexistence: N and N+1, in Both Directions).
when The previous version is no longer valid — a destructive migration ran, side effects were emitted, a dependency moved — or fixing is genuinely faster than reverting.
cost Shipping a change under time pressure with less evidence than usual. Sometimes the only option, which is why it must be a planned option (Roll Forward: When Going Back Is the Harder Option).
when Reversal is slow or unavailable and impact must stop now: shed load, disable the affected endpoint, fail over, serve degraded.
cost Users lose functionality rather than getting correct behaviour, and the underlying change is still deployed (Load Shedding).
The ways a rollback stops being a rollback
Each row is an ordinary, reasonable change that removed the rollback option without anyone noticing. That is the pattern worth internalising: rollback availability decays through normal work, so it has to be maintained deliberately.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Contract step of a migration ran with the release | Rolling back the code produces errors on every request | The old version reads a column that no longer exists | Separate the contract step from the release that stops using the column, by at least one rollback window (Expand, Migrate, Contract) |
| New version wrote data in a shape the old one cannot parse | Rollback completes, then errors on records created in the last hour | Forward compatibility was never a requirement anyone checked | Require tolerant readers in the old version before shipping the new shape (Version Coexistence: N and N+1, in Both Directions) |
| Config change shipped alongside the code | Old version starts and immediately fails validation | Only the artifact was reverted; configuration moved forward | Version config with the release and revert them together (A Config Change Is a Production Change) |
| Deploy referenced a mutable tag | Rollback appears to succeed and the same broken code is running | The tag was moved; it no longer identifies the artifact anyone believes it does | Deploy and roll back by digest (Tags Versus Digests) |
| Retention pruned older images | Rollback fails: the artifact does not exist | Retention policy shorter than the realistic rollback window | Retain every deployed digest for longer than any plausible rollback horizon (Artifact Retention) |
| Release emitted webhooks and emails | Rollback restores behaviour; partners have already acted on the wrong data | External side effects are outside every reversal mechanism you own | Identify side-effect-emitting changes in advance and plan compensation rather than reversal (Roll Forward: When Going Back Is the Harder Option) |
| A dependency was upgraded in the same window | The old version does not start, or fails against the new dependency version | Rollback of one component in a system that moved on around it | Roll back the set that changed together, or treat the combination as forward-only |
A rollback under pressure, including the part that is not reversed
The purpose of this reconstruction is the gap between T+9m and T+14m. Impact ended when traffic left the new version; the incident continued because a rollback is a mitigation, not a repair.
- T+0changeRollout reaches 100%. Deploy pipeline reports success (A Successful Deploy Is Not Evidence of a Healthy System).
- T+4msignalError rate on one endpoint rises above baseline. Alert fires on the symptom, not on the deploy (Alert on Symptoms, Not on Causes).
- T+5mactionResponder checks what changed. The deploy annotation is on the dashboard, so the suspect is immediate (Change Correlation).
- T+6mactionDiagnostics captured — a few failing requests, logs, a heap snapshot — before the failing state disappears (Production Debugging).
- T+7mactionDecision to roll back. Not to investigate first: mitigate, then understand (Stop the Harm Before You Understand It).
- T+7m30sactionRollback begins. Because this is a rolling deploy, it will take about as long as the rollout did.
- T+9mrecoveryEnough instances back on the previous version that error rate falls below the alert threshold. Impact is ending, gradually rather than instantly.
- T+14mrecoveryRollback complete. Error rate at baseline. The mitigation is done.
- T+16msignalThe residue: rows written in the new shape during the exposure window, and a batch of webhooks already delivered to partners. Neither is reversed by anything that just happened.
- T+40mactionCompensating work planned for the residue; investigation of the defect begins now that nothing is on fire (Postmortems).
Durations are illustrative of ordering rather than measured. The load-bearing rows are T+6m — capture evidence before it vanishes — and T+16m, the part a rollback does not touch.
How to do it properly
Most important first.
- Decide the rollback plan before the deploy, as part of the change, and write down which of the four kinds it is: routing switch, redeploy, flag flip, or "none, this is one-way".
- Keep the previous artifact immutable and addressable by digest, and keep it long enough that it outlives any window in which you might want it (Tags Versus Digests).
- Preserve rollback validity deliberately: no destructive migration until the rollback window has closed, and no config change that the old version cannot read (Expand, Migrate, Contract).
- Rehearse it. A rollback performed once outside an incident turns an unknown duration into a measured one, and finds the artifact that no longer starts.
- Roll back to end impact, then investigate. Diagnosing before mitigating extends user impact for the sake of understanding you can obtain afterwards (Stop the Harm Before You Understand It).
- Roll back config, flags and code together where they were changed together — a partial reversal produces a state that has never existed (A Config Change Is a Production Change).
- Know explicitly which changes are not reversible, and treat those as a different risk class before they ship, not during the incident (Roll Forward: When Going Back Is the Harder Option).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A rollback is a full deployment and carries a full deployment's blast radius — it is the operation most likely to be run under pressure, by a tired person, without a canary. It is contained only by being rehearsed, by the previous version still being valid, and by using the same gated path as any other deploy rather than a manual shortcut.
What can go wrong
- Rollback across a contracted schema: old code queries a column that no longer exists, and the mitigation becomes the outage.
- Rollback of code without config, so the old version runs with configuration keys it does not understand or is missing ones it requires.
- The previous image is gone, or the tag was overwritten, so the rollback target is not what anyone thinks it is.
- A rollback that takes as long as the rollout, during which impact continues and everyone assumes the mitigation is already in effect.
- A rollback that fails halfway and leaves a fleet split across two versions with nobody driving it either way.
- Rollback treated as resolution: impact ends, the investigation never happens, and the same change ships again next week.
- Repeated rollbacks masking a persistent defect, so the release that finally goes out carries a large accumulated batch (Change Size: Why Small Changes Are Safer, and When They Are Not).
- "We can always roll back." You can until a contract step, an irreversible migration, an external side effect or a dependency upgrade removes the option — and none of those announce themselves.
- "Rollback undoes the change." It stops the new code running. Everything the new code did is still done.
- "Rollback is instant." It is exactly as fast as your reversal layer: seconds for a router or a flag, a full rollout duration for a rolling redeploy.
- "The rollback finished, so the incident is over." The impact may have ended; the incident ends when you have understood it and stopped it recurring (Postmortems).
- "Rolling back is an admission of failure." Rolling back quickly is the cheapest possible outcome for a bad change. Making it socially expensive is how organisations end up debugging in production instead.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- User-visible impact ended, confirmed by the same signal that detected it — not by the deploy tool reporting success (A Successful Deploy Is Not Evidence of a Healthy System).
- The version now serving is the one you intended, verified by digest rather than by tag.
- A measured rollback duration from a rehearsal, so the number used during an incident is a fact.
- This lesson is about rollback, so the recursive question is what happens when the rollback itself fails: you are now further from a known-good state than when you started, with less time.
- The answer is to have a defined third option in advance — force a specific known-good version at full traffic, disable the feature entirely with a flag, or shed load and mitigate at the edge while you fix forward (Load Shedding).
- Never let the rollback path depend only on the system that is currently broken. If the deployment pipeline is down, there must be a way to get a known-good version running without it.
- Automate the mechanics and the trigger: an automatic abort on a failing canary step is a rollback made by a machine in seconds rather than by a human in minutes (Canary Analysis: Compared Against What?).
- Automate the preservation of validity: block a destructive migration in CI while the previous release is still a rollback target.
- Keep the decision to roll back a running production system with a human when the signal is ambiguous — and make the bias toward rolling back, because reversal is usually cheaper than investigation under load.
- Keeping rollback available constrains what you may ship: no destructive changes for the length of the window, redundant columns and code paths maintained meanwhile.
- Rolling back quickly means you often roll back changes that would have been fine, paying in deploy churn to avoid rare expensive outcomes. That is usually the right trade and it is a real cost.
- A rollback ends impact and also ends your access to the failing state, which can make diagnosis harder — capture what you need before reversing (Production Debugging).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- PLATFORM-SPECIFICReversal time is a property of the mechanism: a load balancer or serverless alias switch is near-immediate, a rolling redeploy takes a rollout, a recreate takes a second outage window, and an edge or DNS change is eventual. The same word describes operations that differ by orders of magnitude, so measure yours.
- GENERALThe validity condition — the previous version must still be correct for the current state — holds on every platform and is what actually decides whether rollback is available.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.