The Homepage That Fell Over Every Hour on the Hour
Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.
The symptom
What was reported, before anyone knew what was happening.
What each party saw, and what each concluded
The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.
| Who | What they could actually see | What they concluded | Verdict |
|---|---|---|---|
| Web tier | Cache miss on homepage:featured; call the recommendation service to rebuild. | A normal miss. Rebuild and repopulate. | ✕ wrong |
| Recommendation service | 9,000 identical requests for the same computation within 400ms. | A traffic spike. | ✕ wrong |
| Cache | A key reaching its TTL and being evicted, exactly as configured. | Nothing — it behaved correctly. | ✓ right |
| DBA | The same expensive aggregate query executing thousands of times in one burst. | The query needs an index; the application is not caching. | ✕ wrong |
3 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.
Commit before you read on
The application caches this value aggressively and the cache hit rate is 99.94%. Commit before reading on: explain how a 99.94% hit rate is compatible with 9,000 simultaneous misses, and why adding an index would not have helped.
Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.