The question this answers
I have one copy of the data and it works. What does a second copy actually buy me?
Replication by itself guarantees exactly one thing: more than one copy of the data exists. Every stronger claim — that a read sees the latest write, that failover loses nothing, that two replicas agree — is a property of the replication *protocol*, not of having copies.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A replica knows the contents of its own storage and the last change it applied from its source. It does not know whether that is current, whether its source has accepted writes since, or whether its source is still alive. "I am up to date" is always an inference from the absence of news, and absence of news is exactly what a partition produces.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Four motives that are usually collapsed into one
People say "we replicate for availability" and mean any of four different things. They are worth separating because each one implies a different acceptable staleness, a different failover behaviour, and a different bill.
The important column below is the third. Read scaling and read-latency locality are satisfied by stale copies; survive-a-node-loss and survive-a-region-loss are not, because a stale copy that becomes the source of truth silently discards committed writes.
| Motive | What it needs | Is a stale copy acceptable? |
|---|---|---|
| Serve more reads than one machine cantypical | Any number of followers, lag-tolerant reads | Yes — this is the cheapest case and the one people over-engineer |
| Serve reads near the usertypical | A copy inside each region the users are in | Yes, if the application can name a staleness bound it tolerates |
| Survive losing a machine without losing writesprotocol | A committed write durable on more than one node *before* the client is told it committed | No — this is what makes the write path slower |
| Survive losing a whole region or datacentreassumption | Copies in independent failure domains, plus a promotion procedure that is safe | No, and the distance sets a floor on write latency |
The cost is agreement, not disk
Disks are cheap; the expensive part of a second copy is that the system now has two versions of the truth and must have an answer for which one counts. Every subsequent lesson in this module is a different answer to that question, and each answer moves the cost somewhere else — onto write latency, onto a loss window at failover, or onto the application, which must merge conflicts.
This is why "just add a read replica" is not free even when the replica is only read from. The moment a read can be served by a copy, some read somewhere returns a value older than a write the same user already saw — and the bug report will not say "replication lag", it will say "I saved it and it disappeared".
Replication is not backup, and the difference is the failure it survives
A replica applies your changes as fast as it can, including the change that deleted the table. That is what it is for. A backup is a copy that is deliberately *behind*, and the gap is the whole product — it is what lets you recover from a logical error rather than a hardware one.
The clean way to hold this: replication protects against a failure of the machine, backup protects against a failure of the software or the operator, and they defend against disjoint sets of incidents. Every team that has replaced backups with replicas discovers this during the incident, not before it.
- Hardware loss, power loss, host eviction — replication covers this; backups are slow recovery for it.
- Bad migration, accidental
DELETE, application bug corrupting rows — backups cover this; replicas faithfully reproduce it in milliseconds. - Ransomware or credential compromise — neither covers this unless a copy is offline or immutable.
Key points
- Replication guarantees only that copies exist. Everything else comes from the protocol layered on top.
- Four motives — read throughput, read locality, node-loss durability, region-loss durability — want different protocols and tolerate different staleness.
- Read scaling is satisfied by stale copies; write durability is not, and that is why durability is the one that costs write latency.
- The real cost of a copy is not storage, it is having two versions of the truth and needing a rule for which counts.
- Replicas are not backups: a replica reproduces your
DELETEfaithfully and immediately.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Choose which node or nodes may accept a write — one (leader-based), several (multi-leader), or any (leaderless).
- • The accepting node records the change durably in an ordered form: a log entry, a WAL record, a versioned value.
- • The change is propagated to the other copies, either before acknowledging the client (synchronous) or after (asynchronous).
- • Each replica applies changes in an order that preserves whatever guarantee the protocol claims.
- • Reads are routed to a copy chosen by a policy — leader only, any replica, or a quorum of replicas — and that routing policy is what determines the staleness a reader can see.
- • The replication stream stalls: the source is fine, the replica is fine, and the gap grows unbounded.
- • A replica falls so far behind that its source has already discarded the log segments it needs, and it cannot catch up without a full re-seed.
- • The source dies with changes that reached no other copy.
- • Two copies both accept a write for the same key and neither is wrong by the protocol.
- • A replica applies changes in a different order than its source and diverges silently — no error, just different bytes.
- • Growing lag with no errors: replica apply-lag climbs from milliseconds to minutes while every dashboard stays green, and support tickets arrive describing data that "went backwards".
- • Failover data loss: a promoted replica serves a version of reality that is missing the last few seconds of committed writes; the operator sees orders acknowledged to customers that no longer exist in the database.
- • Silent divergence: two replicas return different values for the same key with no error on either, discovered only when a checksum or reconciliation job is run.
- • Re-seed storm: a replica that fell out of the retention window requires a full copy, and the resulting bulk read saturates the source at exactly the moment it is already under pressure.
- • Replicated corruption: a bad application write or an accidental mass delete is faithfully applied on every copy within milliseconds, so the number of replicas provides no protection at all.
- • Read scaling requires no coordination at all: the follower asks for changes and applies them, with no agreement about what is current.
- • Durability across failure domains requires agreement *before* the client is acknowledged — that is the one place replication buys latency directly.
- • Failover requires agreement about who is now the leader, which is the point where replication stops being a data problem and becomes a consensus problem — see Leader Election: Choosing One, and Knowing You Were Chosen and Split-Brain: Two Nodes, Both Certain They Are In Charge.
- • Each surviving copy still holds a self-consistent prefix of history, if the protocol applies changes in order — which is a stronger assumption than it sounds.
- • Reads can still be served from any surviving copy, at an unbounded staleness once the source is gone.
- • Anything acknowledged to a client but not yet propagated is at risk, and no copy can tell you afterwards how much that was.
- • Detect: measure apply lag at the replica in *data* terms (position gap) as well as time terms, because a quiet system has near-zero time lag while badly behind.
- • Contain: stop routing reads to a replica once its lag exceeds the bound the application was written against, rather than serving increasingly stale answers.
- • Recover: let the replica catch up from the log if it is inside the retention window; re-seed from a snapshot if it is not.
- • Reconcile: compare copies with a checksum or Merkle-style comparison — see Anti-Entropy: Repairing Divergence Nobody Reported and Merkle Trees: Finding the Difference Without Reading the Data — rather than assuming propagation implies equality.
- • Verify: after a failover, explicitly determine and record how much acknowledged data was lost, and reconcile it against upstream systems that were told the writes succeeded.
- • Replica apply lag in both seconds and log positions, per replica, not averaged across the fleet.
- • Whether the replication stream is *connected* — a stalled stream and a fast one both report low lag if lag is measured naively as "time since last applied event".
- • Retention headroom: how far a replica may fall behind before a full re-seed becomes necessary.
- • Divergence checks: a periodic comparison of a sampled key range across copies, alerting on inequality rather than on lag.
- • Post-failover write-loss accounting: the highest position acknowledged on the old leader versus the highest present on the new one.
- • Read-dominant workloads where a bounded staleness is genuinely acceptable and can be stated as a number.
- • Any system whose availability target cannot be met by a single machine's maintenance and failure profile.
- • Geographically distributed users whose read latency is dominated by distance rather than by query cost.
- • Durability requirements that exceed what one storage device and one power domain can offer.
- • Write-heavy workloads with a small read fraction: every copy pays the full write cost and returns nothing on the investment.
- • Systems whose users always read what they just wrote, where an asynchronous replica introduces a user-visible bug for no gain.
- • Small datasets on modern hardware where a single well-provisioned node with fast restore meets the actual availability target more cheaply.
- • Teams without a tested promotion procedure — an untested failover is a second way to lose the system, not a way to save it.
- • One node with a fast, *tested* restore path — for many internal systems a 10-minute recovery objective is honest and radically cheaper.
- • Vertical scaling first: a bigger machine removes the read-throughput motive entirely, with no new consistency question. See archLinks
horizontal-vs-vertical-scaling. - • A cache in front of a single primary, which buys read throughput without a second source of truth — at the cost of an invalidation problem instead. See A Cache Across Machines Is a Replica With No Replication Protocol.
- • Partitioning rather than replicating, when the pressure is dataset size rather than availability. See Why Partition: Four Ceilings, Four Different Answers.
Four motives, four different protocols
✕ A read quorum can miss the write entirely
There is a legal read that returns a value older than an acknowledged write.
- Quorums are drawn from the same N home replicas — no sloppy quorum, no hinted handoff to a stand-in node.
- Membership is stable: every participant agrees which N nodes hold this key while the read and the write are in flight.
- A write that reached W replicas is durable on all W — an acknowledgement is not withdrawn by a later crash.
- The reader can tell which of the returned values is newest — a version, a vector clock or a monotonic timestamp, not a wall clock it merely trusts.
- R + W = 2 <= N = 3, so a read quorum and a write quorum can be disjoint. A read may legally return a value older than an acknowledged write, however briefly.
- 2W = 2 <= N = 3, so two concurrent writes can each reach a quorum without meeting. The system must merge them; it cannot order them.
A sloppy quorum accepts W acknowledgements from nodes outside the home set during a partition. The count is met, the overlap is not, and the read misses the write.
| What it needs | Staleness it tolerates | What the copy costs | |
|---|---|---|---|
| Read throughputtypical | One leader, asynchronous read replicas. Writes never wait for a replica. | Seconds. A product page a second behind is still a product page. | Read-your-writes breaks the moment a user is routed to a lagging replica. |
| Read localitytypical | Asynchronous replicas placed near the readers. | One WAN one-way delay, continuously, plus whatever the link is doing today. | Reads got faster; a write is still one region away from its leader. |
| Durability against node lossassumption | Quorum or semi-synchronous replication inside one region. | None for acknowledged writes — that is the whole point of paying for it. | Every write now waits for a second machine, and stalls when that machine stalls. |
| Durability against region lossassumption | Synchronous replication across regions, majority acknowledgement. | None for acknowledged writes. The speed of light is now inside the write path. | Write latency floors at the inter-region round trip and cannot be tuned below it. |
What people believe, and what is true
Replication is our backup strategy.
A replica applies the destructive change as eagerly as any other. Replication defends against machine loss; backup defends against logical error. They cover disjoint incidents.
More replicas means more durability.
Only if the copies fail independently and only if the write is acknowledged after reaching more than one of them. Five asynchronous replicas in one rack add read capacity and no durability at all.
A replica with zero reported lag is up to date.
Lag reported as "time since the last event applied" reads zero on a stalled stream with no traffic. Position gap is the honest measure.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
A second copy buys availability, read capacity, read locality or durability — but which one you get depends entirely on the protocol, not on the copy.
Practical
Name the motive first, then pick the protocol. If the motive is read throughput, asynchronous followers are correct and cheap. If the motive is not losing acknowledged writes, you must pay for the acknowledgement to wait on more than one node, and no amount of asynchronous replicas substitutes for that.
Advanced
Replication converts a single-copy availability problem into a multi-copy agreement problem, and agreement is the thing that cannot be bought cheaply. The rest of this domain is the price list: Synchronous Replication: Paying Latency for a Durability Guarantee pays in latency, Asynchronous Replication: The Loss Window You Chose pays in a loss window, Multi-Leader Replication: Accepting Writes in More Than One Place and Leaderless Replication: Every Replica Accepts Writes pay by handing conflicts to the application.
Apply it
- ⚡ A social feed at 50:1 read:write, single region, users tolerate a few seconds of staleness — what replication arrangement is correct, and what is over-engineering here?
- ⚡ A payments ledger where an acknowledged write must never be lost. Where exactly in the write path does the extra latency have to appear?
- 💬 You add a read replica to relieve the primary. What class of bug have you just introduced, and to which users?
- 💬 Your team says replication means you no longer need backups. What do you say?
- 💬 How would you measure whether a replica is actually keeping up, given that "lag in seconds" reads zero on an idle stalled stream?