p99 doubled and nothing changed

Observation order is not causal order. The first thing anyone noticed is rarely the first thing that happened.

1. Mon 14:00A composite index is added to search-db as part of scheduled optimisation work. Query latency improves slightly.ddl audit log
2. Thu 08:12p99 latency alert fires: 1,910 ms against a 940 ms baseline. p50 is 84 ms, unchanged.request_duration
3. Thu 08:15No deploys in nine days. No config changes. No traffic anomaly. The obvious explanations are all unavailable.deploy markers
4. Thu 08:20Suspicion falls on Monday's index, on the grounds that it is the only recorded change to anything.incident channel
5. Thu 08:34Database metrics are flat: query latency, buffer hit ratio and lock waits are all unchanged since Monday.db metrics
6. Thu 08:51p99 is split by instance. Thirty-eight instances read 890 ms; two read over 9,000 ms.request_duration by instance
7. Thu 09:02The two slow instances share a hypervisor host. CPU steal time on that host is 22%; disk read latency is 40× the fleet median.host metrics
8. Thu 09:18Both instances are drained and replaced. Fleet p99 returns to 910 ms within six minutes.request_duration