Run the incident, then rewrite the runbook that made it worse
Checkout success rate has fallen from 99.4% to 71%. The on-call engineer follows the runbook: "1. Check the dashboard. 2. If it looks bad, restart the pods. 3. If that does not work, escalate." They restart the pods. Success rate recovers to 88% for six minutes, then falls to 64%. Two other engineers have joined and are typing in the same terminal session.
The fix that looks right
Restarting the pods, because it works — briefly. Connection pools reset, queued waiters clear, the success rate jumps, and the graph provides exactly the confirmation that the action was correct. Six minutes later it decays, and the natural response is to restart again, which works again, slightly less. The team can spend an hour in that loop, each iteration reinforcing the belief that restarts help and each one destroying the pool state that would have shown the real cause. A runbook that prescribes an unconditional restart institutionalises this: it makes evidence destruction the documented first response.