The Lock Holder That Came Back From the Dead
Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.
The symptom
What was reported, before anyone knew what was happening.
What each party saw, and what each concluded
The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.
| Who | What they could actually see | What they concluded | Verdict |
|---|---|---|---|
| Worker 1 | Lock acquired at 02:00:00. Wrote rows 1 to 12,000. Continued writing after 02:00:14. | I hold the lock; I am the only writer. | ✕ wrong |
| The lock service | Key export-lock set by worker 1 at 02:00:00 with TTL 10s. No refresh. Key expired at 02:00:10. | The holder is gone; the lock is free. | ✕ wrong |
| Worker 2 | Acquire succeeded at 02:00:11. | No one else is writing the export. | ✕ wrong |
| Object store | Two clients appending to the same object, interleaved. | Nothing — it has no opinion. Both writes were valid and both were accepted. | ✓ right |
3 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.
Commit before you read on
Worker 1 never crashed and its process log shows a continuous run. Commit to an explanation before reading on: how did a live, healthy worker lose a lock it was actively holding, and why did it not notice?
Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.