Disaster Recovery as an Operation
A disaster is a class of event, not a size of one. DR is the standing capability to reach a known-good state, chosen per failure class.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What actually counts as a disaster, and what capability answers each kind?
Teams plan for the dramatic case — a region disappearing — and are taken out by the ordinary one: a bad migration, a deleted bucket, an expired credential, a corrupted index. The plan covers the rare event and not the likely one.
Disaster recovery means having a plan for the data centre burning down. We use a cloud provider with multiple availability zones, so that is handled.
Multi-zone infrastructure protects against a zone failing. It faithfully replicates a bad DELETE to every zone at once.
- Multi-zone infrastructure protects against a zone failing. It faithfully replicates a bad DELETE to every zone at once.
- Most real "disasters" are self-inflicted and logical: a migration that dropped the wrong column, an automation that deleted the wrong resources, a credential rotation that locked everyone out.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Disaster recovery is a capability, not a document: an inventory of what state exists, a recovery path per class of loss, tested evidence that each path works, and a named person who can invoke it.
- The classes fail differently and need different answers. Infrastructure loss needs redundancy or rebuild; data loss needs backups and restore; logical corruption needs point-in-time recovery and reconciliation; access loss needs an out-of-band path; dependency loss needs degradation.
- Every recovery path is a trade between standing cost and time-to-recover. Paying continuously for warm capacity buys speed; paying nothing until the event costs time you have to spend during it.
- The plan must include the systems you use to run recovery. If the runbook lives in a wiki hosted on the failed infrastructure, or the incident channel depends on the identity provider that is down, recovery starts by recovering the recovery tools.
- DR overlaps with incident response but is not the same activity: incident response stops user impact; DR restores a known-good state. The same event usually needs both, in that order (Stop the Harm Before You Understand It).
Five classes of disaster, and the tool for each
Redundancy answers exactly one row of this table. That is the single most useful thing to notice about it: the architecture people point at when asked about DR handles the least likely class.
| Class | Example | What answers it | What does not |
|---|---|---|---|
| Infrastructure loss | Zone or region unavailable | Redundancy, failover, rebuild from IaC | Backups alone — restore is far slower than failover |
| Data loss | Table dropped, bucket emptied | Backups plus a drilled restore | Replication — it copies the deletion |
| Logical corruption | Bad migration or bad code writes wrong values | Point-in-time recovery, side restore, reconciliation | Snapshots taken after the damage began |
| Access loss | Identity provider down, key or credential lost | Break-glass path, out-of-band credentials | Anything requiring the lost login |
| Dependency loss | Payment provider or upstream API down | Degradation, queueing, feature disable | Failover — your infrastructure is healthy |
DR shapes, from cold to always-on
These are the standard shapes, and they form a spectrum of standing cost against recovery time. No shape is correct on its own — the correct one is the cheapest that meets the objective you actually committed to.
Note that the first shape depends entirely on restore working, which is the whole argument of Restore Drills. The last shape is not a DR strategy so much as an architecture, and it brings a permanent operational tax.
How much standing cost does this system's recovery objective justify?
when The system can be down for as long as a restore plus a rebuild takes, and the data volume makes that bounded.
cost Cheapest to hold, slowest to recover; recovery time scales with data volume and is dominated by transfer and index rebuild.
when Data is replicated continuously but compute is off until needed.
cost Pays for storage and replication continuously. Recovery still requires provisioning and scaling compute, which is where the recovery-region capacity question bites.
when A scaled-down copy of the system runs continuously and can be scaled up.
cost Continuous duplicate infrastructure at reduced size, plus the operational burden of keeping two environments in step.
when Both sides serve traffic and either can absorb all of it.
cost Highest standing cost and highest complexity: data consistency across regions, deploys everywhere, and enough headroom in each side to carry the whole load (Operating in More Than One Region).
The recovery path has its own dependencies
Recovery is a system, and it has an availability of its own. Drawing it is worth the two minutes, because the drawing is where people notice that the runbook, the credentials and the coordination channel all live inside the thing that failed.
- Every node on that path is a dependency you must have tested. A runbook in a wiki behind the failed identity provider is not a runbook.
- The declaration node is human on purpose. Automating it means automating a judgement about ambiguous evidence (How to Automate Something).
How to do it properly
Most important first.
- Inventory state first. List every store that holds something you cannot regenerate from a repository, with an owner and a recovery path per item. Most gaps are found here, before any strategy is chosen.
- Pick a DR shape per system, driven by its objectives rather than by a single company-wide standard (RTO and RPO).
- Write down the trigger and the authority: what observation makes this a disaster, and who can declare it and invoke the plan (Roles During an Incident).
- Keep the runbook reachable when the primary is not — a copy outside the affected infrastructure, in a form a phone can open (Runbooks).
- Rehearse the classes you are most likely to hit, and rehearse the declaration itself, not only the technical steps (Restore Drills).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The plan itself is what contains a disaster. Where the plan does not cover a class of loss, nothing does.
What can go wrong
- The plan assumes a full-region loss and offers nothing for a single corrupted table.
- The recovery procedure depends on a tool, credential or document that the disaster removed.
- "We are multi-zone, so we have DR." Zone redundancy is high availability — it handles infrastructure loss within a region and nothing about data or logical damage (High Availability in Cloud & Infrastructure).
- "Disaster means the region is gone." Most declared disasters are data or access events inside a perfectly healthy region.
- "The provider handles it." The provider handles their infrastructure. Your data, configuration, access model and application are yours (Shared Responsibility in Cloud & Infrastructure).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A state inventory that names every non-regenerable store, its owner and its recovery path.
- For each recovery path, the date of the most recent successful exercise.
- A runbook accessible from a device and network that do not depend on production.
- A named role that can declare a disaster, and at least one rehearsal in which someone actually did.
- Invoking DR is itself a large, risky change: failing over, restoring, or rebuilding can each make things worse. The plan needs an abort path — what to do if the restore target is also bad, or the standby is unhealthy.
- Failback is the harder half and is routinely unplanned. While you run in the recovery state, new writes accumulate there and must be reconciled before returning (Region Failover).
- Automate the mechanical: provisioning replacement infrastructure, restoring from a chosen copy, rehydrating configuration and secrets, and health-checking the result.
- Keep human: the declaration, the choice of recovery point, the decision to fail over versus wait, and the decision to fail back. Automated failover on an ambiguous signal is its own outage mode (The Automation Trap).
- Every reduction in recovery time costs standing money — idle capacity, duplicated data, cross-region transfer — spent continuously against an event that may not come (Idle Capacity).
- Automated failover reduces recovery time and adds a new failure mode: failing over for a signal that was not a disaster.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- ORG-SPECIFICWhat counts as a disaster, who may declare one, and what must be communicated to whom are organisational decisions. A regulated environment may have mandated declaration criteria and reporting deadlines; a small product team may have none, and should still name a person.
- CLOUD-SPECIFICZone and region semantics, cross-region replication features, and what a provider guarantees during a large outage vary by provider and by service. Capacity in the recovery region is not reserved for you unless you paid to reserve it — during a wide outage everyone tries to launch at once.
- GENERALThe classification of loss — infrastructure, data, logical, access, dependency — holds anywhere, including on-premise, and is what stops a plan from covering only the dramatic case.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — what guarantees survive a partition, and why a plan that assumes clean, total failures is planning for the easy case.