Membership & Discovery
6 lessons. Every one names the guarantee it claims, what a node can know, and how it fails.
A service registry answers "where is payments?" — and the answer is never a fact. It is a belief, formed from heartbeats that may be late, replicated through a store with its own consistency model, and cached at every hop on the way to the caller. The staleness of the four layers adds up, and the sum is how long you keep sending traffic to a machine that is gone.
Q · When my client asks a registry where a service is, what is that answer actually worth?
"Which nodes are in the cluster right now?" looks like a question with an answer. It is not. Every node holds a local view assembled from the absence of messages, views disagree, and no node can distinguish a crashed peer from a slow one. The engineering question is not how to get the right answer — it is which decisions are allowed to depend on a belief that may be wrong.
Q · Which nodes are in my cluster right now, and what am I allowed to do with that answer?
Each node picks a few random peers each round and exchanges what it knows. Information spreads like an infection: exponentially at first, reaching everyone in O(log N) rounds, at a per-node cost that does not grow with the cluster. The honest price is that it is probabilistic and eventual — no node can ever know that a fact has reached everyone, or when it will.
Q · How does a fact reach every node in a large cluster without anyone talking to everyone?
Replicas drift apart — a write that reached two of three, a node that was down for an hour, a hint that expired, a bit that rotted. Read repair fixes what gets read and abandons everything else. Anti-entropy is the background process that compares replicas systematically and repairs what nobody asked for, and it is the only reason your replication factor still means something a year in.
Q · My replicas have drifted apart and nobody noticed. What brings them back together?
Two replicas hold a billion keys and want to know which ones differ. Comparing them key by key costs a billion comparisons and a terabyte of transfer. A hash tree over key ranges answers the same question with one 32-byte exchange when they match, and about fifteen exchanges to locate a single difference when they do not. It is one of the genuinely beautiful ideas in distributed systems, and the details of getting it wrong are where all the operational pain is.
Q · How do two replicas find which of a billion keys differ, without sending each other a billion keys?
A binary failure detector forces you to pick one timeout, and that timeout is simultaneously too twitchy for the cheap decisions and too slow for the expensive ones. A graded detector — one that reports how suspicious a node is rather than whether it is dead — lets each consumer choose its own threshold where it actually knows the cost of being wrong.
Q · How long should I wait before deciding a node is dead — and why is that the wrong question?