The question this answers
What does it take for "your write succeeded" to survive the immediate death of the node that said it?
Any write acknowledged to the client is durable on at least the configured number of replicas at the moment of acknowledgement. If the leader is lost immediately afterwards, that write is present on a node eligible for promotion — assuming the promotion procedure only promotes nodes that have it.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The leader knows which followers have confirmed receipt up to which position — this is real, locally verifiable knowledge, unlike almost everything else in this domain. What it does not know is whether a *non*-responding follower has the entry: an unacknowledged entry may be applied, in flight, or lost, and the leader cannot distinguish these (A Timeout Tells You Nothing About Whether It Happened again, at the replication layer).
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
What the acknowledgement is actually claiming
In asynchronous replication, "200 OK" means *one node has it*. That is a claim about one disk in one failure domain, and the correct reading is "your write is as durable as this machine is". Synchronous replication changes the claim to "at least k+1 independent nodes have it", which is the only form in which an acknowledgement survives the loss of the acknowledging node.
Notice the shape of the trade precisely: you have not made writes safer, you have made the acknowledgement honest. The work is the same; what changed is when you are told it is done. That reframing matters because it explains why the latency cost is unavoidable rather than an implementation inefficiency.
The availability inversion nobody expects
Here is the counterintuitive part, and the reason naive fully-synchronous setups get rolled back after their first incident. A synchronous follower is a dependency of the write path. If it stalls — GC pause, disk saturation, network hiccup — writes stall with it. You added a replica to improve availability and lowered write availability, because you now need two nodes up instead of one.
This is why real systems almost never use "all followers must ack". They use one of two shapes: semi-synchronous (wait for any one of several followers, so any single stall is absorbed) or quorum (wait for a majority, so a minority of stalls is absorbed and the promotion rule is designed to match). See Quorums: What R + W > N Does and Does Not Buy.
| Arrangement | Nodes required for a write | Loss window on leader death |
|---|---|---|
| Fully asynchronousprotocol | 1 (the leader) | Up to the current replication lag |
| Semi-synchronous, any 1 of 3 followersassumption | 2 of 4 | Zero, if promotion only considers acked followers |
| Majority quorum, N=5protocol | 3 of 5 | Zero, and safe promotion is derivable rather than procedural |
| All followers must ackprotocol | All of them | Zero — and any single slow node halts all writes |
"Durable" needs a definition, and vendors differ on it
A follower can acknowledge a replicated entry at four different moments, and systems ship with different defaults. Received into memory — survives nothing but the network. Written to the OS page cache — survives a process crash, not a power loss. Flushed to the device — survives power loss, subject to the device honouring the flush. Applied to the state machine and visible to readers — the only level at which a promoted follower can immediately serve the write.
The gap between "acked" and "visible" is the one that surprises people: a follower may confirm durability of a log entry it has not yet applied, so a read on that follower immediately after promotion can still miss the write for a moment. If your correctness argument depends on the write being *readable* rather than merely *recoverable*, check which of the four your system means.
- Ask what your system's "synchronous" actually waits for — the answer is often page cache, not fsync.
- Ask whether promotion considers apply position or receive position; the difference is a real read-after-failover gap.
- On cloud block storage, an fsync is already a network round trip to a replicated volume — so you may be paying for durability twice without gaining independence.
Key points
- Synchronous replication does not make writes safer; it makes the acknowledgement honest across machine loss.
- Every synchronous follower is a dependency of the write path — you traded write availability for durability.
- Fully synchronous ("all must ack") is almost always wrong; semi-synchronous or quorum absorbs a single stall.
- The latency cost includes a network round trip and, across regions, the speed of light. It is not tunable away.
- "Durable" has at least four meanings — received, page-cached, flushed, applied — and they fail differently.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The client sends a write; the leader logs it durably and assigns it a position.
- • The leader sends the entry to followers and does *not* yet acknowledge the client.
- • Each follower persists the entry to whatever level the system defines as durable, then reports its position.
- • When the configured number of confirmations has arrived, the leader acknowledges the client.
- • If confirmations do not arrive within a timeout, the leader either blocks, fails the write, or degrades to asynchronous — and which of these it does is the most important line in the configuration.
- • A synchronous follower stalls, and every write blocks behind it.
- • The follower acknowledges receipt but has not flushed, and a correlated power event loses both copies.
- • The leader crashes after the follower persisted but before the client was acknowledged — the write exists and the client believes it failed.
- • The system silently degrades to asynchronous on timeout, so the durability guarantee is absent precisely during the incident it was bought for.
- • The follower is in the same rack, power domain or availability zone, so both copies share a failure.
- • Write stalls with no failing node: p99 write latency jumps to the replication timeout while every node reports healthy. The cause is one follower's disk, and the symptom appears on the leader.
- • Silent degradation: the system falls back to asynchronous after a replication timeout and logs it at INFO. The operator discovers the guarantee was off only after a failover loses data.
- • Phantom write: the leader dies between follower persistence and client acknowledgement. The client retries, and unless the operation is idempotent the effect happens twice — the write existed all along. See The Retry Is a Decision, Not a Reflex.
- • Correlated loss: both copies were in the same availability zone, the zone lost power, and the acknowledged writes are gone despite synchronous replication having worked exactly as configured.
- • Throughput collapse under fan-out: a leader waiting on many synchronous followers has its throughput bounded by the slowest follower at any moment, so tail latency at one node becomes throughput at the cluster. See perfLinks
tail-latency.
- • One round trip per write, added to the critical path. This is the cheapest possible form of coordination — it is not consensus, because the leader alone decides.
- • The coordination cost is paid on every write, unlike leader election which is paid rarely; this is the tradeoff shape to compare against Quorums: What R + W > N Does and Does Not Buy and consensus.
- • Across regions the coordination cost has a hard physical floor: the round-trip time between the regions. See The One Number You Cannot Optimise and perfLinks
cross-region-latency.
- • If a synchronous follower fails, writes stop (or degrade) — but no acknowledged write is lost.
- • If the leader fails, every acknowledged write is present on at least one promotable node, provided promotion respects the acknowledgement rule.
- • Reads on followers remain available and remain stale in the same way as before — synchronous replication constrains the write path, not the read path.
- • Detect: alert on replication *wait time* on the leader, not just on follower lag — the leader-side wait is the direct measure of the availability you are spending.
- • Contain: remove a persistently slow follower from the synchronous set explicitly rather than letting timeouts do it implicitly, so the change is a decision with a record.
- • Recover: let the removed follower catch up asynchronously, then re-add it to the synchronous set only once its lag is stably near zero.
- • Reconcile: after any degradation window, determine which writes were acknowledged without the guarantee and treat that window as at-risk.
- • Verify: kill a synchronous follower under load in a test environment and confirm the system does what you believe — block, fail, or degrade. The default is frequently not what the team assumed.
- • Leader-side time spent waiting for follower acknowledgements, as a distribution, not a mean.
- • A counter for every fallback from synchronous to asynchronous mode, with duration — this must be an alert, not a log line.
- • The identity of the current synchronous follower set, and how often it changes.
- • Which durability level the follower ack represents, verified rather than assumed, ideally by a power-cut test.
- • Failure-domain independence of the synchronous set: are the acking nodes actually in separate zones?
- • Data where a lost acknowledged write is a business incident rather than an annoyance — ledgers, payments, order placement, identity.
- • Any system where the client cannot re-derive the write, so a loss is unrecoverable rather than merely inconvenient.
- • Failover architectures where the promotion must be automatic and safe, since automatic promotion is only safe if the promoted node provably has the acknowledged writes.
- • High-volume, low-value writes — telemetry, analytics events, cache fills — where the coordination is a permanent tax on the hot path for data nobody would notice losing.
- • Cross-region synchronous replication for latency-sensitive user writes: the physics makes the write path visibly slow, and there is no configuration that fixes it.
- • Small clusters where the synchronous follower set is a single node, which halves your write availability for a modest durability gain.
- • Semi-synchronous: wait for any one of several followers, keeping the guarantee while absorbing single-node stalls.
- • Quorum acknowledgement, which makes the promotion rule derivable from the write rule rather than a separate procedure. See Quorums: What R + W > N Does and Does Not Buy.
- • Asynchronous replication plus an explicit, measured, accepted loss window — an honest choice for many workloads. See Asynchronous Replication: The Loss Window You Chose.
- • Write to a durable queue first and let the database follow, moving the durability boundary earlier and making the write path append-only. See The Log Is Not a Queue.
- • Local-region synchronous replication with asynchronous cross-region, which buys zone-loss durability without paying intercontinental latency per write. See Active-Passive: Simple to Reason About, Rarely Tested.
What a synchronous acknowledgement claims, and what it costs
What people believe, and what is true
Synchronous replication makes writes safe.
It makes the acknowledgement honest. The window where a write can be lost is not eliminated — it is moved to before the acknowledgement, where the client still knows the outcome is unresolved.
More synchronous replicas means more durability.
Beyond independence of failure domains, additional synchronous replicas mostly buy you additional ways for a write to stall. Durability comes from independence, not from count.
If it acked, it survived a power cut.
Only if "ack" means fsync and the device honours flush. Page-cache acknowledgement survives process death and not power loss, and this is a common default.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
The leader waits for a follower to confirm before telling the client "done". The wait is what makes the acknowledgement survive the leader.
Practical
Use semi-synchronous or quorum acknowledgement rather than "all followers". Verify what your system does on replication timeout, alert on that fallback, and check that the acking nodes are in different failure domains than the leader.
Advanced
The write-loss guarantee is a joint property of the acknowledgement rule and the promotion rule; neither is sufficient alone. This is the argument that makes majority quorums attractive — with W and R both majorities, any promotable set necessarily intersects any acknowledged set, so promotion safety follows from arithmetic rather than from an operator's runbook. See Quorums: What R + W > N Does and Does Not Buy and The Raft Log: Commit Index, Divergence and Reconciliation.
Apply it
- 💬 Your database is configured for synchronous replication and you still lost acknowledged writes during a zone outage. Give two explanations.
- 💬 What exactly does your system's follower acknowledgement mean — received, cached, flushed, or applied? Why does the difference matter for automatic failover?
- 💬 Adding a synchronous replica improved durability and made the service less available. Explain to a sceptical manager why that is expected rather than a bug.