Failure Models
9 lessons. Every one names the guarantee it claims, what a node can know, and how it fails.
A single machine fails all at once — the process is running or it is not. A distributed system fails in pieces, at different times, and each surviving piece has a different and incomplete picture of which pieces those were. Every other idea in this domain is a response to that.
Q · Why is "some of it is broken" so much harder to handle than "all of it is broken"?
Service A calls Service B and the call times out. The single most common mistake in distributed systems is treating that as "it failed". It is not a failure result — it is the absence of a result, and five different realities produce it.
Q · My request to another service timed out. Did the work happen?
Before you can say a protocol is correct you have to say what it is correct against. Crash-stop, crash-recovery, omission, timing and Byzantine are the standard ladder — each admits more behaviours, and each costs more to tolerate. Choosing one is a design decision that most teams make implicitly.
Q · What kinds of failure is my design actually built to survive?
A failure detector converts silence into a suspicion. It is worth being precise about how weak that conversion is: five different situations produce the same silence, and one of them is that the observer, not the target, is the isolated one.
Q · My monitoring says the node stopped sending heartbeats. What have I actually learned?
This is the hardest distinction in practice and, in an asynchronous network, it is formally undecidable. Not difficult — undecidable. Every design that acts on "the node is dead" is acting on a guess, and the useful question is what happens when the guess is wrong.
Q · How do I tell whether that node has crashed or is merely slow?
A Byzantine fault is a component that does something arbitrary rather than simply stopping: answering differently to different peers, returning well-formed nonsense, or behaving correctly until it matters. Tolerating it costs 3f+1 nodes, signatures and an extra round — which is why almost nobody does, and why that choice deserves to be explicit.
Q · What if a node does not just fail, but lies — and should my system care?
A fault domain is the set of things that go away at the same time for the same reason. Process, machine, rack, zone, region — and also the ones that are not on the diagram: a shared config store, a certificate authority, a control plane, a deploy pipeline.
Q · When this thing fails, what else goes with it?
Availability arithmetic — three replicas at 99.9% gives nine nines — assumes failures are independent. They are not. The interesting failures have a common cause, and a common cause takes all the replicas at once regardless of how many there are.
Q · Why did all three replicas fail at the same time?
Three replicas on one machine look replicated on the diagram and are not. Redundancy is a count of copies; resilience is whether the system survives a specific failure. Only placement, failure independence and an exercised recovery path turn the first into the second.
Q · I have three copies of everything. Why did the outage still happen?