The Cluster That Kept Re-Deciding Who Was In It
Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.
The symptom
What was reported, before anyone knew what was happening.
What each party saw, and what each concluded
The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.
| Who | What they could actually see | What they concluded | Verdict |
|---|---|---|---|
| Node 17 | Three consecutive gossip heartbeats from node 31 missed. | Node 31 is dead. Mark it failed and start re-replicating its ranges. | ✕ wrong |
| Node 31 | Its own process healthy, disk fine, and its heartbeats being sent. | I am alive and a member. Rejoin and claim my ranges back. | ✓ right |
| Gossip layer | Membership updates propagating and being superseded within seconds. | Converging on the latest view, as designed. | ✕ wrong |
| On-call engineer | Constant rebalancing with no node down. | A bug in the rebalancer’s termination condition. | ✕ wrong |
3 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.
Commit before you read on
No node ever crashed and the gossip protocol converged correctly every time. Commit before reading on: identify the feedback loop, and say which of its two halves you would break first.
Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.