Rolling Back Data
Reverting the transformation code does not revert the tables it wrote. A data rollback is either a restore from a retained snapshot or a re-run of the previous logic over the affected range — and both of them are forward operations.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A bad model shipped four hours ago and has been overwriting a fact table ever since. What does "roll it back" actually mean, and what is still wrong after you have done it?
Everyone who read the table during the bad window and cannot be un-told: the dashboard that showed a wrong figure in a leadership meeting, the downstream model that has already absorbed the bad rows into its own history, and the exported extract sitting in somebody's spreadsheet. Restoring the table serves the future; the past needs a message (Data Incidents).
The unit of a data rollback is a dataset and a range — this table, these partitions, this period — not a release. A deployment is atomic and a dataset is not: half a table can be wrong while the other half is fine, which is a shape code rollback never has (Partitioning).
Revert the commit, redeploy, and consider the incident closed. This is the correct and complete procedure for a stateless service, it is the reflex of every engineer who has operated one, and it is why this lesson exists.
The reverted code is now correct and the table still holds four hours of rows the bad code wrote. Nothing about a deployment touches data already written, so the dashboard keeps showing the wrong number after the fix is live (Atomic Publish).
- The reverted code is now correct and the table still holds four hours of rows the bad code wrote. Nothing about a deployment touches data already written, so the dashboard keeps showing the wrong number after the fix is live (Atomic Publish).
- The bad model ran incrementally, appending rather than replacing, so the wrong rows sit alongside the right ones with no marker distinguishing them. There is no version to revert to inside the table (Full Refresh vs Incremental).
- Three downstream models consumed the bad output during the window and wrote it into their own history. Reverting one table leaves a cascade of derived tables that each need their own repair, in dependency order (The Transformation DAG).
- The change also altered the schema — a renamed column, a changed type. Rolling the code back means the next run writes the old shape into a table that consumers have already adapted to the new one, so the rollback breaks a different set of consumers than the change did (Breaking Schema Changes).
- The source has moved on. The bad window covered a period whose upstream extract has since been compacted, and re-running the previous logic no longer reads the same inputs it read then (Keeping Raw History: The Recovery Position and the Liability).
- Someone restores yesterday's snapshot of the table to undo the bad model, and in doing so also erases eight hours of perfectly good data that landed after the snapshot was taken. The rollback is now its own incident (What Backfills Break).
What is actually happening
- Code and data have different lifecycles, and rollback is the point where the difference becomes expensive. Code is replaced; the running version is the only version and reverting it is total. Data is accumulated; the table is the sum of every write ever applied to it, and a deployment cannot subtract one (Mutable Servers and Immutable Images).
- There are therefore exactly two ways to undo a bad write. Restore: put back a retained earlier state of the dataset — a table-format snapshot, a copy, a partition backup. Recompute: run the previous logic over the affected range and overwrite what the bad logic produced (Reprocessing vs Retrying).
- Both are *forward* operations. Nothing is being undone in the sense a version-control revert undoes; new bytes are being written that happen to equal older bytes. That distinction matters because both operations can themselves fail, and both need the same validation as any other publish (Validating a Backfill Before You Publish).
- Restore is bounded by retention, and retention is the real limit on how far back a rollback can reach. A table format's snapshot history is expired by a maintenance process, and after expiry the earlier state is not slow to recover — it is gone (Open Table Formats).
- Recompute is bounded by input availability and determinism. It reproduces the earlier output only if the raw inputs for that range still exist unchanged and the previous logic had no dependency on wall-clock time, current dimension state, or anything else that has moved (Deterministic Replay: Making the Schedule Reproducible).
- A rollback propagates along lineage in dependency order. Every downstream model that ran against the bad output holds derived bad rows, so the repair is a topological walk from the corrupted node outward, not a single table operation (Impact Analysis).
- None of this recalls what consumers already read. The window between the bad publish and the repair is a period in which decisions were made on wrong numbers, and the only mechanism that addresses it is telling people (Debugging a Data Incident).
Two lifecycles, one word
Everything an engineer knows about rollback comes from operating code, and code has a property data does not: the running version is the *only* version. Replace the artifact and every subsequent request is served by the previous behaviour, completely, immediately, with no residue. There is nothing left over from the bad version because the bad version was never anywhere except in memory.
A dataset is the accumulated result of every write ever applied to it. The bad model did not merely behave badly — it left behind rows, and those rows are indistinguishable from every other row in the table. There is no artifact to swap. Reverting the code changes what the *next* run writes and touches nothing that has already been written, which is why the dashboard is still wrong after the deploy goes green.
The diagram makes the split explicit. The deployment path and the data path diverge at the moment of the bad run and never rejoin: one is repaired by a revert, the other requires a separate, deliberate write. Two incidents, one cause, and only one of them is closed by the tool that reports success.
Revert the commit, redeploy, confirm the pipeline is green, close the incident. Elapsed time is minutes and the process is familiar to everyone in the channel.
Revert the commit so no further damage accrues; identify the affected datasets and the affected range from the publish history; repair each of them — restore or recompute — in dependency order, validating before each publish; then tell the consumers who read during the window.
The deployment and the data have separate states, and only the first is restored by a revert. Every row the bad version wrote survives the rollback untouched, downstream models have already absorbed those rows into their own history, and consumers who read during the window took decisions the repair cannot reach. Closing on a green pipeline closes on the half of the incident that was easy.
Restore or recompute
There are two mechanisms and they fail for opposite reasons. Restore puts back a retained earlier state and is limited by retention: past the expiry horizon the earlier state does not exist and no amount of urgency creates it. Recompute runs the previous logic over the affected range and is limited by inputs: if the raw data for that range has been compacted, mutated or dropped, the run reads something different from what it read the first time.
They also differ in what they produce. A restore gives you the *previous* data — which includes the previous bugs, the previous metric definition, and the absence of everything correct that landed after the snapshot. A recompute gives you the *output of the logic you chose* over today's inputs, which is usually what you actually want and is slower to obtain.
The choice is rarely either-or during a real incident. The common sequence is restore first because it is fast, tell consumers the dataset is behind, then recompute properly and publish the corrected version once it has validated. That sequence costs a visible freshness regression and buys a short window of wrong data, which is nearly always the right trade (The Freshness SLO).
A model wrote wrong rows to a fact table for four hours. What repairs the table?
when The table is in a format that keeps commit history, the bad publish is a small number of versions back, and speed matters more than completeness.
cost Discards every correct write since that version, including data from unrelated pipelines that share the table. Bounded by snapshot retention, which may be shorter than you think (Open Table Formats).
when Raw inputs for the range are retained and the model is deterministic and range-bounded. The default answer for a well-built pipeline.
cost Takes as long as the range is wide, competes with the current schedule for compute, and reproduces the earlier output only if the inputs have not moved (Reprocessing vs Retrying).
when The previous logic was also wrong, or the change was a genuine improvement with one defect in it.
cost The longest path — the fix must be written and reviewed while the table is wrong — and the resulting numbers match neither the pre-change nor the post-change series, which needs explaining (Semantic Changes).
when The repair will take hours and consumers need something coherent in the meantime.
cost Two visible datasets and a switch to remember to undo, plus consumers who will find the quarantined one later and use it (Data Marts).
when The affected range is genuinely immaterial to every consumer, and you can say who they are.
cost A permanent discontinuity in the series that nobody will remember in six months. Only defensible when written into the dataset's documentation (Dataset Documentation).
Snapshot expiry defaults, retention windows and the exact syntax for restoring a previous table version differ between table formats and between versions of the same format, and managed platforms often apply their own expiry policy on top. Confirm your actual retention window by reading the table's current version history rather than by reading a default in a document — and confirm it before an incident, not during one.
Schema rollback is not symmetric
A logic change can be reverted with a recompute. A schema change frequently cannot, because the population it breaks on the way back is not the population it broke on the way out. Adding a column is safe: nothing that worked before stops working. Removing that column during a rollback breaks every consumer who started using it in the interval — and the more successful the change was, the more consumers that is.
The asymmetry is the entire argument for expand-and-contract. Deploy additively, backfill, let consumers migrate, and only then remove the old shape. At every point in that sequence there is a state to go back to that does not break anybody, which is what "having a rollback" actually means for a schema (Expand and Contract Migrations).
The rows below are the ones that make a rollback dangerous rather than tedious. Notice that two of the three impacts are silent: a consumer reading a column that has quietly gone back to its old semantics gets no error, and a pipeline that coerced a type on the way out will coerce it back on the way in. Only the third one — the removal — fails loudly, and it is the only one anybody plans for.
- order_id
- customer_id
- amount_minor
- placed_at
- status
- order_id
- customer_id
- amount_gross_minor
- amount_net_minor
- placed_at
- status
change The change replaced amount_minor with an explicit gross/net pair. The rollback removes the pair and restores the single column — which is not the inverse operation it looks like, because consumers adopted the new columns during the four days the change was live.
| Consumer | Effect | How it shows up |
|---|---|---|
| Finance model that migrated to `amount_net_minor` | Column disappears; the model fails on its next run with a resolution error. The loudest impact and the only one anybody anticipated. | Loudly — it raises |
| BI dashboard still reading `amount_minor` | The column returns and reports gross again, as it always did. Nothing breaks, and the reported figure jumps by the tax and fee difference on the day of the rollback — which reads as a business change (Two Dashboards, Two Numbers). | Silently — no error, wrong result |
| Downstream mart built during the four-day window | Holds a history in which the metric is net for four days and gross on either side of it, permanently, with no marker in the data recording where the boundary is (Semantic Changes). | Silently — no error, wrong result |
| An extract someone took on day two | Sits in a spreadsheet defined on the new semantics, with a filename that says nothing about which definition produced it. Unreachable by any rollback (Who Actually Consumes This Data). | Silently — no error, wrong result |
| Schema registry / contract check | Flags the removal as a breaking change — correctly — and will block the rollback if enforcement is on. This is the check working as designed, and during an incident it is experienced as an obstacle (Contract Enforcement). | Loudly — it raises |
The ways the rollback is itself the incident
Rollbacks are performed under time pressure, by people who are already tired, on datasets that are already wrong. That combination produces a recognisable set of second incidents, and every one of them is preventable by a step somebody skipped because it felt slow.
The pattern in the cause column is that a data rollback is a publish. It has a range, it has inputs, it has validation, it has an atomic swap and it has consumers — and treating it as an emergency exception to those steps is what turns a four-hour incident into a two-day one.
The last row is the one that is not technical and is the most frequently skipped. Nobody feels heroic sending a message that says the number was wrong for four hours, and it is the only part of the repair that reaches the people who actually made decisions on the bad data (Trusting Data).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Revert deployed, incident closed. | The dashboard still shows the wrong number the next morning. | The deployment tooling reported success and nothing in it has any relationship to the rows already written. | Treat every data-affecting revert as two work items, and do not let the deployment tool close the second one ("What Changed?" — Deploy Markers and the Invisible Deploys). |
| Snapshot restore to yesterday. | Eight hours of correct data written by unrelated pipelines vanishes. | A table-level restore is indiscriminate: it returns every row to the snapshot state, not only the rows the bad model touched. | Restore at partition or range granularity where the format allows it, or recompute the affected range instead of restoring the table (Partitioning). |
| Recompute of a three-day range. | The output differs from both the bad version and the expected previous version. | The raw inputs for that range were compacted and deduplicated since, so the previous logic read different data. | Pin the recompute to an immutable raw snapshot and record which input version it read (Keeping Raw History: The Recovery Position and the Liability). |
| Fact table repaired; marts left alone. | The corrected figure appears in one dashboard and the wrong one persists in three others. | Downstream models absorbed the bad rows into their own materialised history and nothing recomputes them automatically. | Walk the lineage graph downstream and repair in dependency order, validating each node before the next (Impact Analysis). |
| Urgent rollback published without validation. | A second defect, introduced by the repair, discovered while the first is still being explained. | The repair was treated as an exception to the publish protocol because it was urgent. | Publish the repair to a side location, validate, then swap. The swap is the fast part; the validation is what makes it safe (Atomic Publish). |
| Rollback needed six days after the bad publish. | No earlier version of the table exists. | Snapshot expiry ran on its schedule, and the retention window was shorter than the time it took a human to notice. | Set snapshot retention from realistic detection time and publish the remaining window per dataset so it is a known number (Restore Testing). |
| Everything repaired correctly; nobody told. | A consumer re-runs a report, gets a different number than last week, and opens an incident of their own. | The repair addressed the data and not the people who had already read it. | Make consumer notification a required, named step in the runbook, with the affected window and the datasets stated plainly (Data Incidents). |
How to build it
Most important first.
- Make every model idempotent and range-bounded so that recompute is always available. A pipeline that can rebuild any stated range from raw inputs has a rollback path for every incident; one that appends without a replaceable unit does not have one at all (Idempotent Data Pipelines).
- Publish atomically to a versioned target — a table format commit, a swap, a partition replace — so that a bad publish is one identifiable version rather than a smear across a table. The rollback unit and the publish unit should be the same unit (Atomic Publish).
- Retain snapshots for at least as long as your realistic detection time, and decide that number deliberately. Most data incidents are found by a human days after they start, and a snapshot retention shorter than that is a rollback capability that will not be there when it is needed (RPO & RTO).
- Keep raw inputs immutable and retained, because recompute is the only rollback that works after snapshots expire and the only one that produces *corrected* rather than *previous* data (Keeping Raw History: The Recovery Position and the Liability).
- Separate the schema change from the logic change and deploy them apart. Additive-first, backfill, then switch consumers, then remove — so that at every point there is a version to go back to that does not break a different set of consumers (Expand and Contract Migrations).
- Record which code version produced each published version of a dataset. Without that edge, "roll back to before the bad model" is a question nobody can answer precisely and the range is guessed (Data Lineage).
- Validate the rollback before publishing it exactly as you would validate any other write. A restore or a recompute is a publish, and an unvalidated repair during an incident is how a second incident starts (Validating a Backfill Before You Publish).
- Write the consumer notification into the runbook as a required step, not a courtesy. The people who read the bad number are the part of the incident that no technical mechanism reaches (Who Actually Consumes This Data).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A code revert guarantees that future runs use the previous logic. It guarantees nothing about any row already written, and the gap between those two facts is the entire subject of this lesson (Four Ways to Replace Running Code).
- A snapshot restore guarantees the dataset returns to exactly the state at that snapshot — including the absence of everything correct that arrived afterwards. It is precise and it is indiscriminate (Backup Strategy).
- A recompute guarantees the output of the chosen logic over the chosen range against the inputs *as they are now*. It does not guarantee reproduction of the earlier output, because the inputs may have changed (Reprocessing vs Retrying).
- Neither guarantees anything about downstream datasets. Their correctness is restored only by repeating the operation on each of them in dependency order (The Transformation DAG).
- What is explicitly not guaranteed: reversal of consumption. Extracts taken, decisions made, models trained and reports sent during the bad window are unaffected by any rollback (Trusting Data).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Before publishing a rollback, run the dataset's full validation suite against the candidate output in a location consumers are not reading. A repair that is not validated is a change with more urgency and less scrutiny than the one that caused the incident (Data Tests).
- Compare the candidate against the pre-incident data for a range the incident did not touch. If a range that was never affected now differs, the rollback logic is wrong and the blast radius just grew (Validating a Backfill Before You Publish).
- The check misses semantic regression: restoring the previous version restores the previous *definition* too, which may itself have been the thing somebody was fixing. A rollback that undoes a correct metric change reconciles perfectly and is wrong (Semantic Changes).
- It also misses everything downstream. A validated fact table with unrepaired marts above it is a green check on the one node nobody was reading (Impact Analysis).
- A rollback consumes the freshness budget of every dataset it touches. Recomputing a range holds the pipeline busy and delays the current period, so a rollback during a busy window frequently breaches the freshness objective while fixing the correctness one (The Freshness SLO).
- A snapshot restore is typically fast — a metadata operation in a table format that points the table at an earlier version — and it makes the dataset *older*, which reads as a freshness regression on every monitor watching it (Freshness Monitoring).
- A recompute is as slow as the range is wide, and it competes for the same compute the current schedule needs. Running it against a side location and swapping in keeps consumers reading something coherent throughout (Atomic Publish).
- The honest sequencing during an incident is often to restore quickly to a known-good older state, tell consumers the data is behind, and recompute the correct version afterwards. Two moves, and it beats a long window of confidently wrong data (Data Incidents).
- A rollback that reverts a schema change is not symmetric with the change. Adding a column is backward compatible; removing it during a rollback breaks every consumer who adopted it in the interval, so the rollback is a breaking change on a different population (Backward Compatibility).
- Rolling logic back over a range whose schema has since changed produces output in an older shape written into a newer table. Whether that fails loudly or coerces silently depends on the format's evolution rules, and silent coercion is the common case (Schema Evolution).
- Any rollback that reverts a metric definition needs announcing in the same terms the change was announced in. Consumers noticed the number move once; they will notice it move back and will trust neither value (The Metrics Layer).
- Long-lived snapshots eventually hold schemas the current readers do not expect. A restore from far enough back is a schema migration wearing a rollback's clothes (Open Table Formats).
- The rollback *is* the recovery, which is why it deserves the same care as a release: bounded range, side location, validation, atomic swap, and a record of what was published and why (Planning a Backfill).
- Recovering from a bad rollback is the same procedure again with a different target version, which is why retaining a snapshot of the *bad* state before overwriting it matters — without it, a rollback that turns out to be the wrong call cannot be examined afterwards (Backup Strategy).
- When recompute is impossible because inputs are gone, restore is the only path; when restore is impossible because snapshots expired, recompute is the only path. Losing both at once is the condition every retention decision is quietly choosing between (Data Retention).
- Repair the cascade in dependency order, and re-validate each node before moving to the next. Repairing a leaf before its parent produces a dataset that is correct for exactly as long as it takes the parent to run (The Transformation DAG).
What can go wrong
- The code is reverted, the incident is closed, and the wrong rows are still in the table. The single most common data rollback failure and it happens because the deployment tooling reported success ("What Changed?" — Deploy Markers and the Invisible Deploys).
- A snapshot restore that also discards correct data written after the snapshot, converting a bounded incident into an unbounded one (What Backfills Break).
- A recompute against inputs that have changed since, producing output that is neither the old value nor the value the fixed code would produce today (Reprocessing vs Retrying).
- The cascade forgotten: the fact table is repaired and four marts still hold derived bad rows, which are discovered a week later by a different team (Impact Analysis).
- Snapshot retention shorter than detection time, so the rollback capability everybody assumed existed expired before the incident was noticed (Data Retention).
- The mitigation failing: an urgent rollback published without validation, introducing a second defect while the first one is still being communicated (Validating a Backfill Before You Publish).
- Consumers never told, so the corrected number appears without explanation and is read as a second error rather than a fix (Trusting Data).
- "We rolled back the deployment, so the data is fixed." The deployment governs what future runs do. Rows already written are unaffected by it, and the table will keep serving them until something writes over them (Atomic Publish).
- "Blue-green means we can roll back instantly." Blue-green swaps which code serves traffic. Both colours wrote to the same tables, so there is no green copy of the data to swap back to unless you built one deliberately (Blue-Green Deployments).
- "Time travel means we can always go back." Only within retention, only for datasets in a format that keeps versions, and only to a state that also lacks everything correct that arrived since. It is a real capability with three sharp edges (Open Table Formats).
- "Rolling back is safer than fixing forward." Frequently the opposite. A restore discards good data written after the snapshot, and a reverted metric definition may undo a correction somebody made deliberately. Rollback is a change and carries a change's risk (Semantic Changes).
- "We fixed the table, so the incident is over." Consumers read the wrong number during the window and some of them acted on it. The last step of a data rollback is a message, and it is the step most often skipped (Data Incidents).
- A snapshot restore can resurrect rows that were deleted for a privacy request, because the snapshot predates the deletion. Deletion processes must therefore cover retained versions, or the restore path quietly becomes a compliance failure (Deletion Requests).
- Long snapshot retention extends the retention of everything in the dataset, including data whose stated retention period has expired. The rollback window and the retention policy are the same number and are usually set by different people (Data Retention).
- A rollback that restores a previous access configuration alongside the data can re-expose columns that were masked in the interval. Access rules should be applied at read time rather than baked into the stored version, so that a restore of data is not a restore of permissions (Row and Column Security).
Operating it
- The published version history of each dataset — version identifier, publish time, code version, row count — as a queryable table. It is what makes "roll back to before the bad model" a precise instruction rather than an estimate (Metadata: Technical, Operational and Business).
- Deployment markers on the same axis as dataset metrics, so a step change in a row count or a distribution can be lined up against the release that caused it ("What Changed?" — Deploy Markers and the Invisible Deploys).
- Snapshot retention remaining per dataset, published as a number. It is the rollback window, and teams routinely discover its actual size during the incident that needs it (Restore Testing).
- The lineage graph downstream of the corrupted node, which is the work list for the cascade and the answer to "what else is wrong" (Data Lineage).
- An incident timeline recording when the bad version was published, when it was detected, when the repair published, and who was told — the raw material for shortening the next one (Reading a Timeline: Observation Order Is Not Causal Order).
- At 10x datasets, the cascade is no longer walkable by hand and column-level lineage stops being a nice-to-have — the difference between repairing four downstream models and forty is whether the graph exists (Column-Level Lineage).
- At 10x data volume per partition, recompute stops being the fast option and snapshot restore becomes the only response that fits inside an incident, which pushes the retention decision from an afterthought to an architectural one (RPO & RTO).
- At 100x, rollback becomes a platform capability rather than a per-pipeline procedure: a standard publish protocol, a standard version record and a standard repair path, because bespoke rollback logic per pipeline cannot be kept correct (Data Platform Engineering).
- Consumer count does not change the mechanics and changes the communication entirely. Eighty consumers means the notification step is the long pole, and it is the step with no tooling (Data Discovery).
- A snapshot restore in a table format is usually a metadata commit and is close to free in compute. Its cost was paid in advance, continuously, as retained storage for versions nobody read (Storage Lifecycle).
- A recompute costs the full processing of the affected range, at a moment when the current schedule also needs compute, so the real cost includes the delay it imposes on everything else (Compute Waste).
- The cascade multiplies both: every downstream model over the affected range, in dependency order, each with its own validation pass (Scan Cost).
- Retaining snapshots long enough to cover realistic detection time is a standing storage cost that buys an option. It is one of the few places in a data platform where paying for something you hope never to use is straightforwardly correct (What Actually Drives Data Platform Cost).
- Snapshot retention buys a fast rollback and costs retained storage for every version, continuously, for datasets that will almost never need it.
- Recompute-based rollback buys correctness — it produces the right answer rather than the previous one — and costs raw retention, deterministic pipelines, and time proportional to the range.
- Atomic versioned publishing buys a clean rollback unit and costs a publish protocol every pipeline must follow, including the ad-hoc ones, which is where the discipline usually breaks.
- Fast restore first, correct recompute afterwards buys a short window of wrong data and costs a visible freshness regression plus a second communication to consumers. It is usually the right call and it always feels wrong at the time (The Freshness SLO).
Dataset review questions
This lesson uses the shared review exercise.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe asymmetry between reverting code and reverting data follows from data being accumulated rather than replaced, so it holds on every stack; what varies is only which of the two repair mechanisms is cheap, and on a platform with neither versioned publishing nor retained raw inputs the honest answer is that there is no rollback path at all.
- FORMAT-SPECIFICIceberg, Delta and Hudi keep a commit history that makes restoring a previous table state a metadata operation, bounded by whatever the maintenance process has not yet expired; plain Parquet directories on object storage have no version concept, so the equivalent capability must be built by hand as copies or dated partitions, and the rollback window is whatever someone remembered to keep.
- TOOL-SPECIFICTransformation frameworks that rebuild a model in full on every run give a rollback for free — revert the code and the next run overwrites the bad output — while incremental materialisations do not, because the bad rows were appended and no subsequent run replaces them. Which materialisation a model uses therefore decides whether a code revert is sufficient, and that is rarely how the choice is made.
- ORG-SPECIFICWhether the consumer-notification step actually happens depends on whether the team has a channel to its consumers at all; platforms with a catalog and named dataset owners can send it in minutes, and platforms without one discover during the incident that they do not know who reads the table.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns rollback for code — artifact versioning, blue-green and canary releases, the deploy pipeline and the revert. Every intuition from there is correct about the deployment and wrong about the tables, and this lesson exists because that domain's reflexes are the ones an engineer arrives with.
- — DevOps / Production Engineering also owns the migration discipline this borrows — expand and contract, additive-first, deprecate before removing — which is the only technique that makes a schema rollback safe rather than merely possible.
- — Distributed Systems owns why a repair cannot be atomic across a lineage graph: the datasets are separate systems with separate commits, so a cascade is repaired node by node and there is always a window in which some nodes are corrected and others are not.