intermediate
A Small Drop and a Large Outage
Read the evidence before you read the options. The signals are presented the way a dashboard would present them — nothing is labelled with the answer.
The report
At 09:12 the product API started returning errors. It recovered on its own around 09:40. Nobody deployed anything. The team lead wants to know whether this will happen again, and the incident channel has settled on "the database fell over".
The system
Timeline of readings, 09:00 – 09:45ILLUSTRATIVE
| Signal | Value | What it tells you |
|---|---|---|
| Request rate | 2,100 req/s throughout | Incoming traffic is flat across the entire window. |
| Cache hit rate | 96% → 41% at 09:11, recovering to 94% by 09:38 | The proportion of requests served from cache fell by more than half, then climbed back. |
| Redis node count | 3 → 2 at 09:11, back to 3 at 09:26 | One cache node left the cluster and rejoined fifteen minutes later. |
| Database query rate | 84 q/s → 1,240 q/s at 09:12 | Queries reaching the database increased roughly fifteenfold. |
| DB CPU | 24% → 99% at 09:13 | The database saturated within a minute of the query increase. |
| DB connection pool | 20 of 20, 600+ waiting | All connections checked out with a large queue behind them. |
| API error rate | 0.1% → 34% at 09:15 | A third of requests failed, starting two minutes after the query increase. |
What is the constraint?