Symptom → What to Check
The reverse index of a production incident. Each row opens the lesson that explains the mechanism, the misreading it invites, and what the fix costs.
First moves
| You see | Check this |
|---|---|
| "It's slow" with no other detail | Split duration by endpoint; compare p50 vs p99 before forming any theoryFrom Symptom to Root Cause → |
| Latency stepped at a specific minute | Overlay deploy, config and feature-flag markers — "what changed?" pays first"What Changed?" — Deploy Markers and the Invisible Deploys → |
| Two metrics moved together | Look for the third cause before claiming one caused the otherCorrelation Is Not the Root Cause → |
| Average latency looks fine, users complain | Read percentiles; the mean is an average over experiences nobody hadThe Average Was Fine and Users Were Not → |
Latency shape
| You see | Check this |
|---|---|
| p99 bad, p50 flat | Group the slowest 1% by instance, tenant, key and time — find the dimensionTail Latency: Why p50 Being Fine Does Not Help → |
| Latency rises sharply for a small traffic increase | Check utilisation: you are in the knee of the queueing curveQueueing: Why Systems Get Slow Before They Get Broken → |
| A gap in the trace with no span in it | Instrument the wait: that gap is queueing for a thread, connection or workerQueueing: Why Systems Get Slow Before They Get Broken → |
| One dependency dominates the trace | Find the critical path before optimizing anything parallel to itThe Critical Path Is the Only Path That Pays → |
| A stack of identical spans in one request | N+1: batch the queries, and predict the end-to-end effect firstThe Comb: N+1 as a Visible Shape → |
| Latency scales with number of dependencies | Fan-out tail amplification — per-service p99 becomes per-request p50Fan-Out: Waiting for the Slowest of Seven → |
Resources
| You see | Check this |
|---|---|
| CPU pinned, throughput flat | Profile during the slow window; read the widest framesSelf Time, Total Time, and Where the CPU Went → |
| CPU high but the profile is all runtime frames | Allocation pressure, not algorithmic cost — allocate less before tuning the collectorGarbage Collection: Pause, Throughput, Footprint — Pick Two → |
| CPU low, latency high | It is waiting, not working: locks, pools, I/O or a dependencyLow CPU, High Latency: Lock Contention → |
| Memory grows under flat traffic | Heap comparison over time; distinguish a leak from an unbounded cacheLeak or Unbounded Cache? The Question That Picks the Fix → |
| Periodic latency spikes, evenly spread | Stop-the-world pauses: collector, checkpoint, compaction or cronGarbage Collection: Pause, Throughput, Footprint — Pick Two → |
| One instance slower than its peers | Split every latency metric by instance before averaging them togetherTail Latency: Why p50 Being Fine Does Not Help → |
Data layer
| You see | Check this |
|---|---|
| Query slow, database CPU high | EXPLAIN ANALYZE; compare rows scanned against rows returnedThe Slow Query Workflow → |
| Query slow, database idle, app pool full | Pool saturation — the queue is on your side of the wireConnection Pool Saturation: Waiting in Front of an Idle Database → |
| Query slow, database idle, lock waits high | Transactions serialising on a hot row or critical sectionLow CPU, High Latency: Lock Contention → |
| An index made it slower | Selectivity and cache residency decide; a scan can beat an indexAn Index Scan Is Not Automatically Faster → |
| Reads return stale data after a write | Replication lag — measure it as a capacity signal, not a curiosityReplication Lag: Reads That Are Correct and Stale → |
| Hit rate dropped a little, database load exploded | Misses multiply backend traffic; 96%→60% is ten times the queriesA 95% Hit Rate Tells You Almost Nothing → |
| A popular key expired and everything fell over | Cache stampede — coalesce requests, stagger TTLs, serve stale while revalidatingCache Stampede: Everyone Misses at Once → |
| One shard hot, the rest idle | Key skew — split, replicate or client-cache the hot entriesHot Keys: When Aggregate Metrics Hide a Saturated Node → |
Queues
| You see | Check this |
|---|---|
| Queue depth climbing | Compare arrival rate against completion rate; compute time to drainThe Backlog Arithmetic: Four Levers and a Drain Time → |
| Queue depth stable, users still waiting | Read oldest-message age — depth is not the user-facing numberDepth Is Not an Emergency; Age Is → |
| A dependency failed and load doubled | Retry storm: backoff, jitter and a retry budget, not more retriesRetry Storms: The Load You Generated Yourself → |
| Workers pinned at 100%, queue growing | Confirm the workers are the constraint before scaling themTwenty Workers, All Busy, Five Hundred Waiting → |
Frontend
| You see | Check this |
|---|---|
| Server fast, page slow | Open the browser waterfall; compare lab timings against real-user dataReading the Browser Waterfall → |
| Fast on a laptop, slow on a phone | JavaScript parse and execute cost, not download sizeJavaScript Costs Four Times, Not Once → |
| Content jumps around while loading | Layout shift from unsized media and late-injected contentCore Web Vitals as Signals, Not Scores → |
| The page freezes when you click | Long tasks blocking the event loopEvent-Loop Lag: One Callback, Everybody Waits → |
Capacity & testing
| You see | Check this |
|---|---|
| Fine at average load, fails at peak | Compute utilisation at peak; headroom is what you actually buyHeadroom: The Capacity You Deliberately Do Not Use → |
| Autoscaling did not save us | Measure the lag from metric to ready instance against the spike durationAutoscaling Lag: The Gap Where the Outage Lives → |
| The load test looked great, production did not | Warm caches, cold JIT, unrealistic payloads, or coordinated omissionCoordinated Omission: When the Load Generator Lies → |
| A benchmark says the new version is faster | Check warm-up, variance, environment and whether it measures the real pathBenchmark Fallacies: Confident Numbers That Are Wrong → |
| p95 moved from 180 ms to 260 ms between releases | Establish whether it exceeds normal variance before calling it a regressionRegression or Tuesday? Telling a Real Change from Noise → |
Operating
| You see | Check this |
|---|---|
| Nobody knows if this is bad | Define an SLI and an SLO so "bad" has a number attachedSLOs: A Target, a Window, and a Reason → |
| Alerts fire constantly and nobody reads them | Alert on user-visible symptoms with an action attached, not on thresholdsAlerts Worth Waking Someone For → |
| The dashboard has 200 charts and answers nothing | Build around six questions: healthy, traffic, slow, errors, dependency, capacityDashboards Built Around Questions → |
| We fixed it and something else broke | Expect the bottleneck to move; predict where before you deployThe Bottleneck Moves After Every Fix → |
| The agent run takes 20 seconds | Count steps and round trips before blaming any single model callReading an Agent Run as a Trace → |