Partial and Logical Data Recovery

1. 1
   Stop the bleeding

   Disable the job, revert the deploy, revoke the credential, flip the flag off.

   fails by Recovery starts while damage continues, so the scope keeps moving.

   evidence The write rate of the damaging pattern drops to zero on a dashboard.

2. 2
   Preserve evidence

   Snapshot the current, damaged state before touching it.

   fails by A hasty repair destroys the only record of what the damage looked like.

   evidence A snapshot identifier recorded in the incident timeline.

3. 3
   Scope

   Determine which rows, which columns, and the start instant, from the data and the change record.

   fails by Guessing; a correction that is too wide damages good data, too narrow leaves the incident half-fixed.

   evidence A query that returns the affected set with a count, agreed by two people.

4. 4
   Side restore

   Restore to a separate instance at a point just before the damage began.

   fails by Chosen point falls inside the damage window; or quota, network or key problems block the restore (Restore Drills).

   evidence A spot check on the side copy confirms known-good values for affected rows.

5. 5
   Diff

   Compare side copy and live for the scoped set; classify differences into damage and legitimate later writes.

   fails by Treating every difference as damage, which overwrites correct new work.

   evidence A reviewed diff summary with counts per category.

6. 6
   Dry run

   Execute the correction with writes disabled and report what it would change.

   fails by Skipped under time pressure — the most expensive shortcut in this procedure.

   evidence Dry-run output matching the expected count from the scope step.

7. 7
   Repair in batches

   Apply the correction in bounded, resumable, idempotent batches.

   fails by A single unbounded statement that locks the table, saturates the database, or cannot be stopped.

   evidence Per-batch records of rows changed; live error and latency signals stay flat (Operating a Production Database).

8. 8
   Repair downstream

   Invalidate caches, reindex search, recompute derived data, handle external side effects.

   fails by Database correct, product still wrong, users still seeing stale values.

   evidence Reads through the application return corrected values.

9. 9
   Assert and record

   Run invariants, then write down exactly what changed.

   fails by Nobody can reconstruct the final state during the postmortem.

   evidence Invariant checks pass; a change record exists (Postmortems).

UPDATE orders
   SET status = 'paid'
 WHERE status = 'pending';
-- no bound, no dry run, no batches, no record of prior values

-- 1. scope, agreed and counted first
CREATE TABLE repair_scope AS
SELECT o.id, o.status AS old_status, s.status AS good_status
  FROM orders o
  JOIN side_restore.orders s USING (id)
 WHERE o.updated_at >= :damage_start
   AND o.updated_at <  :damage_end
   AND o.status <> s.status;

-- 2. dry run: this count must match the agreed scope
SELECT count(*) FROM repair_scope;

-- 3. repair in bounded, resumable batches
UPDATE orders o
   SET status = r.good_status, updated_at = now()
  FROM (SELECT * FROM repair_scope
         WHERE NOT repaired ORDER BY id LIMIT :batch) r
 WHERE o.id = r.id;
-- mark repaired, pause, re-check live signals, continue

The left statement has no bound, no record of what it changed, no way to stop halfway and no way to undo. The right one keeps the prior value for every touched row, so the repair itself is reversible; it matches a count agreed in advance, so a scope error is caught before any write; and it yields between batches, so the database keeps serving traffic while the repair runs.