Which Signal Actually Means "The Database Is Slow"
Nine numbers all get reported as "the database is slow" and they mean completely different things. Query duration measured at the application, split by statement, is the one that confirms it — and database CPU is the one that misleads most often.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Nine numbers, and only some of them are evidence
When latency rises and the widest span is a database call, the trace has told you *where the time was spent waiting*, not *what the constraint is*. Those are different claims. The span covers everything between the application asking for a connection and the last row arriving: pool wait, network hop, parse and plan, lock acquisition, buffer misses, disk reads, result serialization. Exactly one of those is "the database is out of capacity", and the other six are things a database dashboard often shows as green.
So the first move is not to open the database dashboard. It is to split the application-side query duration by statement and ask which statements moved. If every statement got slower by roughly the same amount, you are looking at something shared — pool queueing, the network path, a noisy neighbour, a failing disk. If one statement family moved and the rest are flat, you have a query problem and the workflow in The Slow Query Workflow applies.
The panel below is the triage read. Notice that CPU is normal and the system is still badly broken: the smoking gun is that active connections equal pool size while the database itself is bored. That combination means the waiting is happening *in front of* the database, which is a different fix entirely — see Connection Pool Saturation: Waiting in Front of an Idle Database.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| App-side query duration p99 | 840 ms (was 45 ms) | Time from "give me a connection" to "last row received". The symptom, measured where the user feels it. | suspect |
| Database-side statement duration p99 | 14 ms (was 12 ms) | The engine executed the statement in 14 ms. It never saw the other 826 ms. | smoking gun |
| Database CPU | 31% | Not out of compute. Rules out "the database needs a bigger instance" as the primary story. | normal |
| Active connections / pool size | 20 / 20 | Every connection is checked out, continuously. Requests are queueing for a connection, not for the database. | smoking gun |
| Lock wait time | 2 ms avg | Not blocking on row locks this time. Rules out the Low CPU, High Latency: Lock Contention shape. | normal |
| Buffer / cache hit ratio | 99.2% | Working set still in memory; not falling back to disk reads. | normal |
| Rows scanned / rows returned | 1.1 | Queries are selective. No sudden full scan. | normal |
| Replication lag | 80 ms | Replicas are current; reads served from them are fresh enough. | normal |
| Disk read latency | 0.4 ms | Storage is fine. Rules out the I/O story. | normal |
Read them in an order that rules things out
Signals are worth more for what they eliminate than for what they confirm. Every reading above rules a specific hypothesis in or out, and a triage that starts from "which of these is highest" will chase whichever number happens to be noisy. Start from the two-sided comparison instead: application-side duration against database-side duration. The gap between them *is* the queueing, and it is invisible from inside the engine.
The matrix below is the order worth working in. It is deliberately front-loaded with the cheap comparisons — the ones that need no query analysis and immediately halve the search space. Only when the gap is small (the engine really is spending the time) does it become worth opening a plan, and at that point The Slow Query Workflow takes over.
One caution that costs teams hours: a database can be the *victim* rather than the cause. A cache that stopped serving hits pushes its entire miss traffic downstream, and the database dutifully reports high load while behaving correctly (Cache Stampede: Everyone Misses at Once). A retrying client multiplies its own load (Retry Storms: The Load You Generated Yourself). In both cases every database signal is red and every database fix is wrong.
| Compare | If it looks like this | Rules in | Rules out |
|---|---|---|---|
| App-side duration vs DB-side duration | Large gap (840 ms vs 14 ms) | Queueing in front of the engine: pool exhaustion, client-side saturation, network path | Query cost, plan regressions, index problems |
| App-side duration vs DB-side duration | Gap is small (60 ms vs 55 ms) | The engine really is spending the time — go to the plan | Pool sizing, connection churn |
| Per-statement duration | One statement family moved, rest flat | A specific query, plan or data-volume change | Shared resource exhaustion |
| Per-statement duration | Everything moved together | Shared constraint: CPU, I/O, locks, pool, replica promotion, noisy neighbour | A single bad query |
| CPU vs lock wait | CPU low, lock wait high | Low CPU, High Latency: Lock Contention — serialization on hot rows | Capacity: a bigger instance changes nothing |
| CPU vs buffer hit ratio | CPU high, hit ratio collapsed | Working set no longer fits memory; reads hitting storage | Application-side problems |
| Rows scanned vs rows returned | Ratio jumped from ~1 to thousands | A plan flip or a missing index — see An Index Scan Is Not Automatically Faster | Infrastructure faults |
| DB load vs upstream cache hit rate | DB load up, cache hit rate down | The database is a victim of cache behavior, not the cause | Anything fixable inside the database |
The application's clock is the honest one
Every database exposes statement timing, and every statement timing excludes the part of the request most likely to be broken. The engine starts its clock when it receives the statement on an already-established connection. It cannot see the 800 ms the request spent waiting in the application for a free pool slot, the TLS handshake on a cold connection, or the time the driver spent parsing a 4 MB result set into objects.
This is why the instrumentation that matters is a span around the *whole* database interaction on the application side, and separately a metric for pool acquisition wait. With both, the gap becomes a number you can alert on rather than a hypothesis. Without them, an incident where the pool is exhausted looks exactly like an incident where the database is overloaded, and the two have opposite fixes: fewer, faster queries versus more connections — and more connections can make an overloaded database worse.
The excerpt below is the same request seen from both sides. Nothing in the server-side view is wrong; it is simply answering a narrower question than the one being asked.
APPLICATION SPAN (what the user waited for)
db.query GET /orders/42 840 ms
├─ pool.acquire 812 ms ← queued for a free connection
├─ net.roundtrip 2 ms
├─ server execution 14 ms ← all the database ever reports
└─ driver.deserialize (312 rows) 12 ms
DATABASE VIEW (pg_stat_statements / slow query log)
SELECT * FROM orders WHERE id = $1
calls 18420
mean_exec_time 14.0 ms ← "the database is healthy"
max_exec_time 31.0 ms
The engine is telling the truth. It is answering
"how long did execution take?", not
"how long did the user wait for this data?"Key points
- The widest span in a trace tells you where time was spent waiting, not what the constraint is — the two are routinely different.
- Compare application-side query duration against database-side statement duration first; the gap is queueing the engine cannot see.
- Database CPU is the most misleading single signal: bottlenecked at 20% (locks, pool) and healthy at 85% (batch job) are both normal.
- If every statement slowed by a similar amount, look for a shared constraint; if one family moved, look at that query.
- A database can be the victim — cache miss storms and retry storms make every database signal red while every database fix is wrong.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1User → application: checkout p99 goes from 300 ms to 1.4 s; traces show the database span consuming 840 ms of it.
- 2Application → database: the engine reports 14 ms mean execution for the same statements — no plan change, no new index need.
- 3Application → pool: active connections sit at 20 of 20 for the whole window; acquisition wait p99 is 812 ms.
- 4Pool → root cause: concurrency arriving at the service exceeds what the pool can serve, so requests queue in the application while the database idles at 31% CPU.
- 5Root cause → misdiagnosis risk: the trace named the database, the database dashboard is green, and a team without pool metrics concludes "the database is flaky" and resizes the instance.
- • "The database span is the widest, so the database is the bottleneck." The span includes pool wait, network, execution and deserialization; only one of those is the database.
- • "Database CPU is only 30%, so the database is fine." Lock waits, pool exhaustion and single-hot-partition workloads all bottleneck at low CPU.
- • "Database CPU is 85%, so we need a bigger instance." High CPU during a nightly batch with headroom to spare is expected; the question is whether user-facing statements are queueing behind it.
- • "Hit ratio is 99%, so memory is fine." A 99% hit ratio with 40× more queries still means far more disk reads than yesterday — ratios hide volume.
- • "All the database numbers are red, so fix the database." When an upstream cache fails, every database number is red and the fix is upstream.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • A span on the application side wrapping the entire database call, including connection acquisition, tagged with a normalized statement name — not the endpoint.
- • A separate histogram for pool acquisition wait time. This is the single highest-value database metric most teams do not have.
- • Per-statement p50/p95/p99 from the engine (`pg_stat_statements`, the MySQL slow log, or equivalent), so "which statement moved" is answerable in seconds.
- • Rows examined versus rows returned per statement family — a ratio, tracked over time, catches plan flips before users do.
- • Lock wait time, active connections against pool size, buffer/cache hit ratio, and replica lag as four separate series, never merged into one "database health" number.
- • Instrument the gap: application-side span plus pool-acquisition histogram. Until the gap is measurable, every incident of this shape is a coin flip between two opposite remedies.
- • Split query duration by normalized statement, not by endpoint, so "which statement moved" is a dashboard question rather than an investigation.
- • Track rows-examined-to-returned as a ratio per statement family; it is the cheapest early warning for plan regressions.
- • Alert on pool-acquisition wait and lock wait directly, since both bottleneck the system while leaving CPU dashboards green.
- • Record the upstream cache hit rate on the same dashboard as database load, so victim-versus-cause is one glance rather than one meeting.
- • Reproduce the ambiguity: run a load test that exhausts the pool without loading the database, and confirm your dashboards distinguish it from a genuine query regression.
- • After instrumenting, replay the last incident's window and check that the gap between application and engine timing is visible without opening a trace.
- • Confirm each new metric moves independently — if pool wait and execution time always move together, the instrumentation is measuring the same thing twice.
- • Per-statement metrics carry cardinality cost: normalize statements to a bounded set of names, or the metrics backend becomes the next incident ([[cardinality]]).
- • Application-side spans on every query add overhead on hot paths — sample them, and keep the pool-wait histogram unsampled since it is cheap and decisive.
- • More signals mean more triage surface. The matrix above is only useful if the team agrees on the reading order in advance, not during the incident.
- • An alert on application-side query p99 divided by engine-side p99: when the ratio exceeds a stated threshold, the waiting has moved outside the engine.
- • A dashboard panel that pairs every database signal with its ruling-out claim, so triage order survives the person who wrote it leaving.
- • A load-test scenario in CI that saturates the pool, asserting the pool-wait metric fires and the query-duration metric does not.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe nine readings are a teaching example of a coherent incident shape, not a capture from a real system. Real dashboards are noisier and rarely this internally consistent.
- DATABASE-SPECIFICStatement-timing views (
pg_stat_statements, the MySQL slow query log, Oracle AWR) expose different fields with different exclusions. What every engine shares is that its clock starts after the connection exists.