A 90-second cache restart became a 40-minute outage

Observation order is not causal order. The first thing anyone noticed is rarely the first thing that happened.

1. 09:41Error rate on catalog-api crosses 2%. First page fires.error_rate
2. 09:42p99 latency jumps from 240 ms to 4.1 s across the whole fleet at once. No deploy in the last six days.request_duration
3. 09:43catalog-db CPU reads 70%. The on-call notes there is "still headroom" and looks elsewhere.db cpu
4. 09:44Cache hit rate is found to have fallen from 96% to 15%.cache_hit_ratio
5. 09:46Managed Redis event log shows node redis-002 completed an automatic failover at 09:39, taking 87 seconds.cloud provider events
6. 09:48Redis is healthy again and hit rate is recovering — but error rate keeps climbing, now 31%.cache_hit_ratio
7. 09:52Inbound request rate at the load balancer is 2.4× normal. Analytics shows real user sessions flat.lb request rate
8. 10:04Database connection pool is fully saturated on every instance; 800+ requests queued fleet-wide waiting for a connection.pool_wait_time
9. 10:21Traffic is shed at the load balancer. Error rate falls within four minutes and the system recovers on its own.incident channel