advanced
The Tail Doubled, the Median Did Not
Read the evidence before you read the options. The signals are presented the way a dashboard would present them — nothing is labelled with the answer.
The report
A weekly performance review flagged that p99 on the search API has gone from 380 ms to 810 ms over the past two weeks. p50 has not moved. Nobody reported an incident and error rates are unchanged. The team is not sure whether this is worth investigating.
Search API — two week comparisonILLUSTRATIVE
| Signal | Value | What it tells you |
|---|---|---|
| p50 latency | 74 ms → 76 ms | The typical request is essentially unchanged. |
| p95 latency | 210 ms → 244 ms | The 95th percentile has risen modestly. |
| p99 latency | 380 ms → 810 ms | The 99th percentile has more than doubled. |
| Request rate | 3,200 req/s → 3,310 req/s | Traffic is up 3%. |
| Error rate | 0.04% → 0.04% | Unchanged. |
| CPU (fleet average) | 41% → 43% | Compute usage is broadly unchanged. |
| GC pause time (p99) | 12 ms → 13 ms | Collector pauses are stable. |
| Deploys in window | 6 | Six releases went out over the two weeks. |
Slowest 1% of requests, sliced by dimensionILLUSTRATIVE
| Signal | Value | What it tells you |
|---|---|---|
| By instance | evenly spread across all 24 | No single host accounts for a disproportionate share. |
| By time of day | evenly spread | Slow requests are not clustered in any particular hour. |
| By tenant | 78% from tenant `acme-corp` | Four in five slow requests come from a single customer. |
| Tenant `acme-corp` share of traffic | 2.4% | That customer accounts for roughly one request in forty overall. |
| Tenant `acme-corp` p50 | 620 ms | For this customer, the typical request is already slow. |
| `acme-corp` document count | 890k → 4.1 M over 3 weeks | Their indexed dataset grew roughly fivefold. |
What is the constraint?