Coordination
8 lessons. Every one names the guarantee it claims, what a node can know, and how it fails.
The price of agreement is usually quoted in milliseconds. That is the smaller half. The real cost is structural: a node that must confirm with its peers before acting cannot act at all when those peers are unreachable — so every coordination point you add multiplies your availability by someone else’s.
Q · What does it actually cost me to make two nodes agree before either acts?
Before buying agreement, ask three questions: can a violation be repaired afterwards, do the operations commute, and can ownership be partitioned so one node decides alone? A surprising number of "we need a distributed lock" problems dissolve under one of them — and a stubborn residue genuinely does not.
Q · Can I restructure this so nodes act independently instead of agreeing first?
Every coordination mechanism in this domain exists to protect some statement that must never become false. Name that statement first — precisely, with its scope — and the architecture follows from it. Skip that step and you will choose a mechanism, then reverse-engineer a justification.
Q · What must never become false in my system, and what does that requirement force me to build?
A distributed lock is not a mutex that happens to be over a network. A mutex is a correctness mechanism backed by hardware; a distributed lock is a hint backed by a timeout, and the difference decides whether you may use it for efficiency or for correctness.
Q · When can I trust a distributed lock, and what is it actually protecting?
A lock that never expires deadlocks the first time a holder dies. A lease fixes that by making authority time-bounded — and in doing so it converts a distributed-systems problem into a clock problem, which is a trade worth understanding before you make it.
Q · How does authority get revoked from a node that has stopped responding — and what does the expiry actually assume?
The canonical distributed-systems accident: A takes a lock, A pauses, the lease expires, B takes the lock, A resumes believing nothing happened, and both act. Nobody is at fault, no error is logged, and the resolution is not a longer timeout.
Q · A process holding a lock stalls for a minute and then wakes up. What does it believe, and what does it do?
etcd, ZooKeeper and Consul are marketed as different products and are, underneath, the same four primitives over a consensus-backed key-value store. Learn the primitives — compare-and-swap, ephemeral keys, watches, leases — and every one of them becomes a configuration detail.
Q · What does a coordination service actually give me that a database does not?
A unique constraint in one database is a solved problem. Spread the data across shards and it becomes the hardest kind of invariant there is — a negation over global state, which cannot be checked locally by anyone. Four designs exist, and each fails differently.
Q · How do I guarantee a username is unique when no single node holds all the usernames?