The question this answers
What kinds of failure is my design actually built to survive?
A protocol guarantees its properties only under a stated failure model and a stated timing model, with a stated bound on how many participants may fail. Outside those bounds it guarantees nothing — not degraded behaviour, nothing. "Raft is safe" means "Raft preserves safety while a majority of nodes are non-faulty and messages are eventually delivered".
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows what it observed: a message that arrived, a message that did not, a value that disagreed with its own. It cannot classify the failure that produced the observation — an omitted message and a crashed sender look identical, and a corrupted message and a malicious one are the same bytes. The model is an assumption you make, not a fact you detect.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The ladder
Crash-stop. A node works correctly until it stops, and then never returns. This is the simplest model and the one most textbook algorithms are presented against. It is also unrealistic in one important way: real nodes come back.
Crash-recovery. A node may stop and later restart, losing anything it held only in memory but retaining what it wrote durably. This is the model most production systems actually operate in, and it introduces the whole subject of what must be flushed to disk before a message is sent — because a node that promised something, crashed and forgot is indistinguishable from one that lied.
Omission. A node stays up but fails to send or receive some messages: a full queue, a dropped packet, an overloaded thread pool that never dequeues. Notice that from the outside, omission and crash are the same observation. Notice also that omission can be *partial* — some peers get replies and others do not — which crash never is.
Timing. A node does everything correctly but too slowly: a garbage collection pause, a disk that has become slow, a noisy neighbour. This is the model that breaks lease-based and timeout-based designs, because the assumption they rest on is precisely a bound on delay.
Byzantine. A node may do anything at all: send different answers to different peers, send well-formed but false data, or behave correctly until the moment it matters. Corruption and compromise both live here. Byzantine Failures, and Why You Probably Do Not Assume Them treats it properly.
| Admits | Nodes needed to tolerate f | Typically assumed by | |
|---|---|---|---|
| Crash-stopprotocol | Node halts permanently | 2f+1 for consensus | Textbook algorithms |
| Crash-recoveryprotocol | Halt, then return with durable state only | 2f+1, plus durability rules | Raft, Paxos, most databases |
| Omissionassumption | Messages silently dropped, possibly per-peer | 2f+1, handled as delay | Most production systems, implicitly |
| Timingassumption | Arbitrary slowness, unbounded pauses | Same count; needs fencing to stay safe | Anything using leases or timeouts |
| Byzantineprotocol | Arbitrary or malicious behaviour | 3f+1, plus signatures | Blockchains, avionics, some financial infrastructure |
The timing model matters as much as the failure model
Alongside "how may nodes misbehave" sits "what may the network do to time", and the two together determine what is possible. Synchronous: known upper bounds on message delay and on relative processing speed. Under this model a perfect failure detector exists — wait for the bound, and silence means death. Almost nothing on commodity infrastructure is synchronous.
Asynchronous: no bounds at all. Under this model no perfect failure detector exists, and the FLP result says deterministic consensus is impossible even with a single crash failure. This is where most of the impossibility results live.
Partially synchronous: the network behaves within some bound *eventually*, but you never know when that period starts, and before it the network may do anything. This is the model real systems are engineered for, and it is the honest one. It explains the shape of practically every production protocol: they are designed so that safety holds always, in any timing, and liveness holds only during the good periods. A Raft cluster can stall indefinitely under sustained bad timing; what it will not do is elect two leaders that both commit.
This safety/liveness split is worth internalising, because it is how experienced engineers read a guarantee. "Never returns a stale value" is a safety claim and should hold under the worst timing. "Elects a leader within a second" is a liveness claim and holds only when the network cooperates. A system that promises both unconditionally is over-promising.
What real infrastructure actually does
The pragmatic position taken by most distributed databases and coordination services is: assume crash-recovery plus omission plus timing; do not assume Byzantine. In other words, tolerate nodes that stop, come back, silently drop messages and pause for arbitrary periods — but assume that a node which *does* answer answers honestly according to the protocol.
That is a defensible choice, and it is worth knowing why. Byzantine tolerance costs 3f+1 nodes instead of 2f+1 — four nodes to tolerate one fault instead of three — plus cryptographic signatures on messages, plus an extra communication round, plus far more complex code. Against an operator who controls all the machines, the threats that model addresses are better handled by other means: checksums against corruption, TLS against tampering in transit, access control against compromise.
The gap that leaves is real and worth naming: silent data corruption. A disk or a memory error can produce a node that is honest but wrong, which is a Byzantine fault by any formal definition. The standard response is not a Byzantine protocol but checksums at every layer — which is a targeted defence against the specific Byzantine behaviour that actually occurs, at a tiny fraction of the cost.
- Crash-recovery: require durability before acknowledging anything you may be asked about later.
- Omission: treat every silence as possible delay, never as confirmed absence.
- Timing: never let an expired lease alone authorise exclusive action — fence it.
- Byzantine, narrowly: checksum data end to end, because corruption is the one arbitrary fault that occurs routinely.
- Byzantine, broadly: only if participants are outside your trust boundary.
Where implicit models cause incidents
Most teams have never written their failure model down, which means it is implicit in the code and usually inconsistent between components. The characteristic symptoms are worth recognising. A design that assumes crash-stop breaks when a node returns from a long pause and resumes work someone else took over — that is a timing fault meeting a crash-stop assumption, and the result is The Stale Lock Holder: A Paused Process Does Not Know It Was Paused.
A design that assumes crash-recovery but does not flush before acknowledging breaks on power loss: the node returns having forgotten a commitment it made, which turns a crash fault into a Byzantine one from the protocol’s perspective. This is why fsync semantics are load-bearing in consensus implementations rather than a performance detail.
A design that assumes no omission breaks when a health check passes on a node whose worker pool is wedged, because the health endpoint and the work path are different code. The node is "up" by every measurement and does no work — omission failure, presenting as a mysterious throughput drop with green dashboards.
Writing the model down costs an afternoon and changes what you build. The question to answer for each component: may it come back after failing, may it be arbitrarily slow, may it drop messages selectively, and do you trust what it says?
Key points
- A protocol is correct only relative to a failure model, a timing model, and a bound on how many nodes may fail.
- The ladder — crash-stop, crash-recovery, omission, timing, Byzantine — admits progressively more behaviour at progressively higher cost.
- The timing model matters as much: safety should hold under any timing, liveness only during good periods.
- Most infrastructure assumes crash-recovery plus omission plus timing, and explicitly not Byzantine.
- Silent data corruption is a real Byzantine fault handled cheaply with checksums rather than with a Byzantine protocol.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • State which components may fail and in which of the five ways.
- • State the timing assumption: are you relying on any bound on delay, and where?
- • State how many simultaneous failures the design tolerates, and what happens at f+1.
- • Separate the properties into safety (must hold always) and liveness (may stall under bad timing).
- • Check every mechanism against the model — a lease is a timing assumption, a health check is an omission assumption.
- • A node returns from the dead with stale in-memory state and acts on it.
- • A node acknowledges a write it had not made durable, then restarts having forgotten it.
- • A node drops messages to one peer and not others, so different observers form different views.
- • A pause exceeds every timeout in the system, and every component independently concludes the node is gone.
- • A disk returns corrupted data that passes every application-level check because none exists.
- • Resurrected worker: a node judged dead resumes and duplicates work. The operator sees two sets of side effects for one job, with the second set beginning exactly one pause-length after the first.
- • Forgotten acknowledgement: a node confirms a write, loses power, and returns without it. The operator sees a committed write missing after a restart, and a consensus cluster that refuses to make progress because its log diverged.
- • Wedged-but-healthy: liveness probes pass, work does not progress. The operator sees zero errors, green health, and a queue whose age grows without bound.
- • Silent corruption: a replica serves subtly wrong bytes. The operator sees a checksum mismatch during a repair job, or — with no checksums — a customer report and no explanation.
- • The failure model directly sets the quorum arithmetic: 2f+1 for crash and omission faults, 3f+1 for Byzantine.
- • A weaker timing assumption does not change the node count but does change what you may conclude from silence, and therefore whether fencing is required.
- • Every additional fault tolerated costs nodes, and each node costs a round-trip participant on the write path.
- • Within the model and the fault bound, the protocol’s safety properties hold and liveness may be delayed.
- • At exactly the fault bound, safety typically still holds and progress stops.
- • Beyond the bound, the protocol offers nothing — not degraded correctness, nothing. This is why "we lost two of three nodes" is qualitatively different from "we lost one".
- • Detect: monitor how close you are to the fault bound, not just whether you are past it — one node down in a three-node cluster is the alert.
- • Contain: prevent a recovering node from acting on pre-failure state until it has re-synchronised.
- • Recover: restore redundancy before doing anything else, because the window of reduced redundancy is where the second failure lands.
- • Reconcile: verify data integrity on the recovered node rather than assuming a clean start means clean data.
- • Verify: test the model’s boundary deliberately — kill f nodes and confirm the system behaves as claimed.
- • Current fault tolerance headroom: how many more nodes can be lost before the guarantee stops holding.
- • Distinguish liveness probes from work-progress signals; the gap between them is where omission failures hide.
- • Checksum mismatch counts at every layer that has them — the honest measure of how much silent corruption is occurring.
- • Durability confirmation latency (
fsynctime), because a design that quietly stopped flushing looks faster and is unsafe.
- • Whenever a protocol’s guarantee is being evaluated, adopted or debugged — the model is the fine print.
- • When choosing between systems, because two products claiming "strong consistency" may assume different models.
- • Formal model selection for a stateless service behind a load balancer is disproportionate; restart-on-failure is the whole design.
- • Adopting Byzantine tolerance where all participants are inside one trust boundary buys an order of magnitude of complexity against a threat that access control already covers.
- • Reduce the number of components that need a model: state in one store with one replication protocol is one model to reason about instead of five.
- • Buy the model rather than build it: a managed consensus service or database has already made these choices and documented them.
- • Narrow the Byzantine case to its real instance — corruption — and address it with checksums instead of a protocol.
- • Make failures cheap rather than tolerated: if a job can simply be re-run from scratch, the model barely matters.
A protocol is correct only relative to a model
What people believe, and what is true
Raft tolerates any two failures in a five-node cluster.
It tolerates two *crash* failures. A node that is up and answering incorrectly is outside the model, and one such node can violate safety.
The failure model is an academic concern.
It is the fine print on every guarantee you rely on. Most surprising outages are the system behaving exactly as specified, outside the model someone assumed.
A slow node is not a failure.
In a timing model it is precisely a failure, and it is the one that breaks lease-based designs, because they assume a bound on delay.
We do not need Byzantine tolerance because nobody is attacking us.
Correct for malice, incomplete for corruption. Bit flips and buggy firmware produce arbitrary behaviour with no attacker involved.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Say what may fail and how before claiming anything is correct. Crash, crash-and-return, drop messages, be slow, or misbehave arbitrarily — each costs more to tolerate.
Practical
Write down, per component: may it return after failing, may it be arbitrarily slow, may it drop messages selectively, do you trust its answers, and how many can fail at once. Then audit your mechanisms against it — leases assume timing bounds, health checks assume no omission, and acknowledgements assume durability.
Advanced
FLP shows that in a fully asynchronous system, no deterministic protocol can guarantee consensus with even one crash failure — not because the algorithms are inadequate but because no algorithm can distinguish a crashed process from a slow one, so it must either risk waiting forever or risk deciding wrongly. Real systems escape this by weakening the requirement rather than the problem: they adopt partial synchrony and guarantee safety always but liveness only during synchronous periods, or they add randomisation and guarantee termination with probability one. Recognising which escape a system took tells you exactly how it will behave on a bad network day.
Apply it
- ⚡ A three-node cluster loses one node to a hardware fault and a second to a long GC pause at the same time. Walk through what the model says happens.
- 💬 What failure model does your primary datastore assume, and what happens outside it?
- 💬 Why does Byzantine tolerance need 3f+1 rather than 2f+1?
- 💬 What is the difference between a safety property and a liveness property, and which one may a partition break?