All Green, All Useless
Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.
The symptom
What was reported, before anyone knew what was happening.
What each party saw, and what each concluded
The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.
| Who | What they could actually see | What they concluded | Verdict |
|---|---|---|---|
| Load balancer | GET /healthz returning 200 in 3ms from all 24 instances. | All instances are healthy; send traffic to all of them. | ✕ wrong |
| Checkout instance | Its own process running, HTTP server listening, and every database call failing with "connection pool exhausted". | I am up. /healthz returns 200 because it checks that I am up. | ✕ wrong |
| Database | 400 connections held open, almost all idle in transaction. | At the configured limit; refuse new connections. | ✓ right |
| On-call engineer | Green health across the fleet, errors at the edge. | The problem is upstream of checkout — the gateway or the CDN. | ✕ wrong |
3 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.
Commit before you read on
The health check answered its question honestly every time. Commit before reading on: state the question /healthz was actually answering, the question the load balancer needed answered, and what happens if you simply make the check test the database.
Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.