Two Leaders, Both Certain
Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.
The symptom
What was reported, before anyone knew what was happening.
What each party saw, and what each concluded
The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.
| Who | What they could actually see | What they concluded | Verdict |
|---|---|---|---|
| Node A (the original leader, rack 1) | Heartbeat acks from its two rack-1 followers continuing normally. No acks from the two rack-2 nodes. | I still hold a majority of a 5-node cluster (3 of 5). I am leader; the other two crashed. | ✕ wrong |
| Node D (rack 2) | No heartbeat from the leader for 3 election timeouts. Node E reachable and voting. | The leader is gone. Stand for election in a higher term and, on winning, begin serving writes. | ✕ wrong |
| The load balancer | Both A and D returning 200 on /healthz and both reporting role: leader. | Two healthy leaders; route to whichever is closer. | ✕ wrong |
| Operator | Zero errors, zero elections in the alerting metric, and stock counts drifting downward on eleven SKUs. | An application bug in the decrement path. | ✕ wrong |
4 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.
Commit before you read on
A five-node Raft cluster cannot have two leaders in the same term, and this one did not. Commit before reading on: what specifically allowed D to win an election while A still believed — correctly, from its side — that it had three of five?
Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.