Reliability & Disaster Recovery

RPO & RTO

Two numbers that turn "we need it to be reliable" into an architecture. RPO is how much data you can afford to lose; RTO is how long you can afford to be down. Derive the design from the numbers, never the numbers from the design.

▶ Run the lab

The question this answers

Infrastructure question

How much data may we lose, and how long may we be down — and what does each answer force us to build?

Application requirement

The business needs to state what an outage costs before engineering can decide what to spend. "As little as possible" is not a requirement; it is the absence of one, and it produces either an unaffordable design or an unpleasant surprise.

What it provides

Two numbers per system that make the reliability conversation concrete, comparable across services, and directly translatable into a topology and a bill.

Application RequirementInfrastructure RequirementComputeNetworkStorageIdentityDeploymentScalingReliabilityObservabilitySecurityCostTrade-offs

Two numbers, two different questions

RPO — Recovery Point Objective — is how much data you can afford to lose, measured backwards in time from the disaster. An RPO of one hour means that after recovery, up to one hour of the most recent writes may be gone, and the business has agreed that this is survivable. RPO is determined almost entirely by how often you take an independent copy: hourly backups give an RPO of roughly one hour, continuous replication gives seconds, and synchronous replication gives zero at the cost of write latency.

RTO — Recovery Time Objective — is how long you can afford to be down, measured forwards from the disaster. An RTO of four hours means the service must be serving again within four hours of the event starting — including detection, decision, restore and verification, not just the part where someone types commands. RTO is determined by how much infrastructure is already running and how automated the recovery is.

They are independent. A system can have an RPO of five seconds and an RTO of eight hours: continuous replication means almost nothing is lost, and a manual rebuild means it takes all day to serve again. The reverse also exists: a hot standby with nightly backups recovers in minutes and can lose a day of data if the failure was corruption rather than infrastructure. Confusing the two produces designs that solve the wrong half — and the half that was left unsolved is discovered during the incident.

TierRPORTOWhat that requiresRelative cost
Best effort24 h48 hNightly backups, rebuild from code by handVery low
Standard1–4 h4–8 hFrequent backups, automated rebuild, a rehearsed restoreLow
Business critical5–15 min1 hContinuous backup or log shipping; pilot-light standby ready to scaleMedium
Mission criticalseconds5–15 minContinuous async replication; warm standby; automated failover (Active-Passive Failover)High
Zero loss0< 1 minSynchronous replication and active-active. Every write pays the replication round trip (Active-Active)Very high
What each RPO/RTO pair forces you to build, and roughly what it costs

Derive the architecture from the numbers, not the numbers from the architecture

The productive direction is: the business states what an hour of downtime and an hour of lost data cost, engineering states what each tier costs, and the two are compared. The unproductive direction — overwhelmingly the more common one — is that an architecture is built first and its RPO and RTO are measured afterwards and written into a document as though they had been requirements.

Make the conversation concrete with scenarios rather than adjectives. *"The payment ledger loses fifteen minutes of transactions. What happens?"* — for a payments business the answer involves reconciliation with an external provider, customer disputes and possibly a regulator, so the RPO must be seconds. *"The recommendation cache is empty for six hours."* — recommendations are worse, nothing is lost permanently, and the RPO is effectively infinite because the data is derived. Same company, same disaster, requirements three orders of magnitude apart.

That last observation is the reason to tier by data set rather than by system. Uniform requirements across an estate are always wrong in one of two directions: either you are paying mission-critical prices for logs and derived data, or you are protecting the payment ledger the same way you protect an analytics cache. Tiering is what makes the total affordable.

One system, three tiers. The requirement, not the component, decides the mechanism.PROVIDER-NEUTRAL
Region EU
Application tier (stateless)private— RPO n/a, RTO minutes — rebuilt from an image, holds nothing
Payment ledger — RPO seconds / RTO 15 minprivate— synchronous standby + continuous log archive + cross-account backup
User profiles — RPO 1 h / RTO 4 hprivate— hourly snapshots + point-in-time recovery; restore is rehearsed quarterly
Recommendation cache — RPO n/a / RTO 6 hprivate— derived data: no backup at all. Recomputed. Say so explicitly in the plan
Uploaded documents — RPO 15 min / RTO 2 hprivate— versioning + cross-region replication; irreplaceable, customer-supplied
Backup vault — separate accountprivate— the copy that survives a compromise of everything above
Application tier (stateless)Payment ledger — RPO seconds / RTO 15 min
Application tier (stateless)User profiles — RPO 1 h / RTO 4 h
Application tier (stateless)Recommendation cache — RPO n/a / RTO 6 h
Application tier (stateless)Uploaded documents — RPO 15 min / RTO 2 h
Payment ledger — RPO seconds / RTO 15 minBackup vault — separate account· continuous archivecrosses boundary
User profiles — RPO 1 h / RTO 4 hBackup vault — separate account· hourlycrosses boundary
Uploaded documents — RPO 15 min / RTO 2 hBackup vault — separate account· versioned replicationcrosses boundary

The cost curve is not linear, and the last mile is the steep part

Moving from a 24-hour RPO to a one-hour RPO is usually cheap: take backups more often. Moving from one hour to one minute means continuous log shipping and the infrastructure to receive it. Moving from one minute to zero means synchronous replication, which adds the replication round trip to every single write — a latency tax paid on every transaction forever, in exchange for a guarantee that matters during an event that may never occur.

RTO has the same shape. Hours are cheap: restore from backup into an environment you build when needed. Tens of minutes need a pilot light. Minutes need a warm standby. Sub-minute needs active-active and the organization to run it. Each step multiplies both money and operational burden, and the last step multiplies engineering complexity most of all.

The most valuable thing this framing produces is permission to say no. When a stakeholder asks for zero data loss and instant recovery, the answer is not "no" but "here is what that costs, and here is what a fifteen-minute RTO costs instead" — a factor of ten or more in the total. Presented as a price rather than a technical objection, the requirement usually relaxes on its own.

What buying down each number costs. Relative weights, not currency.COST-VARIES
RPO 24h → 1h usage
driven by backup frequency × data size · Cheapest improvement available. Usually a configuration change.
RPO 1h → minutes usage
driven by continuous log shipping + storage for it · Point-in-time recovery is often included with managed databases — check before building it.
RPO minutes → 0 · surprisefixed
driven by synchronous replication · Adds the replication round trip to every write, forever. The cost is latency as much as money.
RTO days → hours fixed
driven by automated rebuild from IaC + a rehearsed restore · Mostly engineering time, and it pays for itself the first time it is used.
RTO hours → minutes fixed
driven by standby infrastructure running continuously · The jump from restoring to failing over. This is where the money goes.
RTO minutes → seconds · surprisefixed
driven by active-active across regions · More than double the infrastructure plus a permanent distributed-data problem.
Rehearsal fixed
driven by engineering time, quarterly · The line without which every number above is aspirational.

Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.

Key points

  • RPO is how much data you may lose; RTO is how long you may be down. They are independent and require different mechanisms.
  • RPO is set by how often you take an independent copy; RTO is set by how much is already running and how automated recovery is.
  • Derive the architecture from the numbers. Measuring your existing design and calling the result a requirement is backwards.
  • RTO includes detection, decision and verification — not just the restore command.
  • Tier by data set, not by system: a payment ledger and a recommendation cache in the same application need requirements orders of magnitude apart.
  • The cost curve steepens sharply at the end; the last increment toward zero is where most of the money is.
  • An RPO or RTO that has never been measured in a rehearsal is a hope, not an objective.

The loop, answered

Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.

How it works
  • RPO is bounded by the interval between independent copies: backup frequency, log-shipping cadence, or replication lag.
  • Synchronous replication drives RPO to zero by acknowledging a write only after a second copy has it — which is why it costs latency.
  • RTO is the sum of detect + decide + prepare + restore + verify + cut over; each term is separately measurable and separately reducible.
  • Restore duration scales with data volume and restore throughput, which is why RTO and data size are linked.
  • Standing infrastructure trades money for RTO: the more that is already running, the fewer steps remain during the incident.
What you still own
  • Assign RPO and RTO per data set and record them somewhere the team actually reads — a service catalogue, not a slide.
  • Measure both during rehearsals and publish the measured numbers next to the target ones. The gap is the honest state of your DR.
  • Re-derive when the data grows: an RTO that assumed a two-hour restore of 200 GB is wrong at 2 TB, and nothing will tell you.
  • Review after every incident. Real events reveal which term of the RTO is actually largest, and it is usually detection or decision.
  • Make the price of each tier visible so stakeholders choose rather than wish.
How it fails
  • Aspirational numbers written in a document and never measured — the most common failure, and invisible until the disaster.
  • RPO defeated by backup frequency: hourly backups cannot deliver a five-minute RPO no matter what the plan claims.
  • RTO blown by detection: the recovery took twenty minutes and nobody noticed the problem for two hours.
  • Uniform requirements across the estate, so either the critical data is under-protected or the whole bill is inflated by protecting logs.
  • Restore time that grew silently with the data until the documented RTO became arithmetically impossible.
  • RPO measured against replication while the actual disaster was corruption, which replication propagated instantly.
How it scales
  • RTO degrades as data grows, because restore duration is proportional to volume. The number must be re-derived, not inherited.
  • RPO is unaffected by data size but is affected by write volume, since replication lag grows under write bursts.
  • More systems means more recovery coordination; a tight estate-wide RTO gets harder faster than the component count grows.
  • Tiering is what keeps the total affordable at scale: uniform mission-critical requirements across hundreds of data sets is not a budget anyone approves.
Security
  • RPO for a compromise scenario depends on how far back your backups reach, because an intrusion may be discovered weeks later. Retention is a security requirement, not just a reliability one.
  • Immutable backups make the RPO real against an attacker who can delete — without immutability, your effective RPO during a compromise is unbounded (Backup Strategy).
  • RTO during a security incident is longer by design: you must rebuild rather than restore in place, and rotate every credential first (The Infrastructure Supply Chain).
  • Recovery procedures need pre-provisioned, audited break-glass access, or the RTO includes an access-request queue (Audit Trails).
Cost shape
  • RPO cost is driven by copy frequency and, at zero, by write latency on every transaction.
  • RTO cost is driven by how much infrastructure runs continuously without serving traffic.
  • Both curves steepen sharply at the last increment; the difference between "minutes" and "zero" is often larger than everything before it.
  • Tiering is the main cost lever available, and it requires the business to rank its own data — which is the hard part.
What to watch
  • Measured RPO: current replication lag and age of the most recent verified backup, side by side with the target.
  • Measured RTO from the last rehearsal, broken into stages so you can see which term dominates.
  • Restore duration trend as data grows, which is the early warning that a documented RTO has quietly become impossible.
  • Coverage: which data sets have an assigned tier and which have never been classified. The unclassified ones are the risk.
  • The signal that lies: the RPO written in the plan. Only the measured one is real.
Simpler alternatives
  • A single availability target instead of two numbers, for a small system where the distinction genuinely does not change the design. Simpler and honest at small scale.
  • Accepting an unbounded RPO for derived data — caches, search indexes, analytics — and recording that decision so nobody later builds backups for it.
  • Provider-managed point-in-time recovery, which delivers a strong RPO with no infrastructure of your own and is the cheapest first move for most teams.
  • For very small systems, "restore from last night's backup and accept a day" is a legitimate, complete DR strategy. Write it down and stop there.
What adopting this costs
  • Every step toward zero RPO costs either money or write latency, and synchronous replication costs both.
  • Every step toward zero RTO costs standing infrastructure that serves nothing on ordinary days.
  • Tiering saves money and costs classification effort plus the discipline to keep classifications current.
  • Ambitious numbers that are never rehearsed are worse than modest ones that are, because they create confidence without capability.

RPO and RTO: pick the requirement, then pay for it

RPO and RTO: pick the requirement, then pay for it
RPO is how much data you may lose; RTO is how long you may be down. Both are requirements someone signs, and every second you shave off either one has a bill and an operational burden attached.
workload
backup frequency
replication
failover
what this tier commits you to
backup storage + retention usage
standby capacity kept running fixed
cross-domain replication traffic usage
write latency tax on every commit fixed
automation, runbooks, restore drills fixed
RPO required
15 min
RPO achieved
24 h
RTO required
1 h
RTO achieved
unknown
RPO  ← replication:  sync = 0 · async = the lag (≈5 s here, and it grows under write load) · none = the backup interval
RTO  ← detection + decision + promotion or restore + repoint. Restore is the slow one: it moves bytes.
Run the failure to see how much data is gone and how long recovery takes.
The whole recovery path runs through a restore that has never been executed. That does not make the RTO 4 h — it makes it unknown, and unknown covers the cases that actually happen: the backup only contained one of the two databases, the encryption key was in the environment that burned, restore throughput is a quarter of what someone assumed, or the job has been failing silently for six weeks. A backup you have never restored is a hypothesis. Turn on the drill toggle and the number becomes a measurement.
ILLUSTRATIVECOST-VARIES

What people believe, and what is true

Claim

RPO and RTO are basically the same thing.

Reality

One is data loss, the other is downtime, and they are set by different mechanisms. A system can lose almost nothing and still take a day to serve again.

Claim

Our RTO is one hour because a restore takes one hour.

Reality

RTO starts when the disaster starts. Detection and the decision to declare are usually larger than the restore itself.

Claim

We should aim for zero on both.

Reality

Zero RPO taxes every write with a replication round trip and zero RTO requires active-active. State the price and let the business choose.

Apply it