The Four Golden Signals
Latency, traffic, errors and saturation. Four numbers that describe almost any request-serving system well enough to know whether it is healthy and, when it is not, which direction to look. The value is not the list — it is that the four are read together.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The four, and why they are read together
Latency is how long requests take, at a percentile, split by success and failure — because fast failures can drag the average down and make a degrading service look like it is improving. Traffic is demand: requests per second, split by route. Errors is the fraction failing, including the ones that return 200 with a broken body. Saturation is how full the constraining resource is — the queue, the pool, the memory, the CPU — and it is the one that predicts the near future rather than describing the present.
Read individually, each one misleads. Latency alone cannot distinguish "slower because of more load" from "slower because of a code change" — that needs traffic beside it. Errors alone cannot distinguish "one bad client hammering a 400" from "the database is refusing connections". Saturation alone is the most common false alarm in the industry: high utilization with no queueing is a well-used system, not a sick one (see USE: Utilization, Saturation, Errors).
Read together, they form a small diagnostic grid. Latency up + traffic up + saturation up is a capacity story. Latency up + traffic flat + saturation flat points at a dependency or a code change. Errors up + latency *down* is fast-failing, often a dependency rejecting immediately or a circuit breaker doing its job. Learning to read the combination is worth more than any individual threshold.
| Latency | Traffic | Errors | Saturation | Most likely story |
|---|---|---|---|---|
| ↑ | ↑ | → | ↑ | Capacity: demand exceeded what the current fleet serves comfortably |
| ↑ | → | → | → | A dependency got slower, or a change shipped — look at spans and deploys |
| ↑ | → | → | ↑ (pool/queue only) | Queueing on a bounded resource: connections, workers, threads |
| ↓ | → | ↑ | → | Fast-failing: errors returning quickly and pulling the average down |
| → | → | ↑ | → | A specific route or client failing while the aggregate looks calm |
| ↑ p99 only | → | → | → | Tail-only problem: GC pauses, lock contention, a slow shard, retries |
One incident, all four
Here is the same checkout incident read as a golden-signals dashboard. The value of the format is how fast it eliminates. Traffic flat kills the capacity theory in one glance. Saturation normal on CPU and memory kills the "bigger instances" reflex. What remains is a latency increase concentrated in the tail with normal errors — which points at a dependency or a change, and sends the investigation to traces rather than to the infrastructure team.
Note the saturation row that matters is not CPU. CPU utilization is the most-watched and least-informative saturation signal in most request services; the constraining resource is far more often a connection pool, a worker pool or a queue (see Connection Pool Saturation: Waiting in Front of an Idle Database and Twenty Workers, All Busy, Five Hundred Waiting). A golden-signals dashboard that only charts CPU for saturation is missing the signal that would have caught this class of incident.
The errors row also earns its place by being unremarkable. Errors staying flat while latency triples is diagnostic information: the dependency is slow but still answering, which rules out a hard outage and points toward degradation or a retry amplification (see Retry Storms: The Load You Generated Yourself).
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Latency · p50 | 210ms (baseline 180ms) | Barely moved — this is not a broad slowdown | normal |
| Latency · p99 | 1,820ms (baseline 240ms) | The tail exploded while the median held: a subset of requests is affected | smoking gun |
| Traffic · rps | 1,205 (baseline 1,180) | Flat. Rules out a demand-driven slowdown immediately | normal |
| Errors · 5xx rate | 0.4% (baseline 0.3%) | Essentially unchanged — the dependency is slow, not down | normal |
| Saturation · CPU | 22% utilization | Not the constraint; adding instances would change nothing | normal |
| Saturation · outbound HTTP pool | 100% in use, 84 requests waiting | The real constraint: requests queue for a connection to the payment provider | smoking gun |
Where the frame does not fit
The golden signals were formulated for request-serving systems, and they transfer imperfectly to everything else. For a batch pipeline, "traffic" is records per run and "latency" is end-to-end job duration — but the signal that actually predicts failure is lateness against a schedule, which is not one of the four. For a queue consumer, the equivalent of latency is oldest-message age, not per-message processing time (see Depth Is Not an Emergency; Age Is).
Streaming systems need a fifth idea entirely: lag. A consumer keeping up perfectly at 08:00 and falling 40 minutes behind by 09:00 has flat latency, flat errors, and flat CPU — none of the four move, and the system is failing. The honest framing is that the golden signals are an excellent default for HTTP services and a starting point everywhere else, to be extended with the signal that actually predicts failure in that architecture.
The other adaptation worth making is per-route and per-dependency splits. A service-wide latency panel hides a single degraded endpoint inside an otherwise healthy aggregate. Splitting by route costs a small amount of cardinality (route is a bounded, low-cardinality label — unlike user id, see Label Sets That Survive a Year) and turns "the service is a bit slow" into "POST /checkout is very slow and everything else is fine".
- HTTP service → the four as written, split by route and status class.
- Queue consumer → replace latency with oldest-message age; saturation is worker-pool occupancy (see Six Queue Signals, Two That Wake You Up).
- Batch job → lateness against schedule and records-per-second; per-record latency rarely matters.
- Streaming consumer → consumer lag is the primary signal; none of the four capture it.
- Agent / LLM workload → time to first token and step count, not just total duration (see Inside One Model Call: Queue, First Token, Generation).
Key points
- Latency, traffic, errors, saturation — the value is in reading them together, not in the list.
- Split latency by success and failure; fast errors otherwise make a degrading service look healthier.
- Saturation is usually a pool or a queue, not CPU — a CPU-only saturation panel misses the most common constraint.
- Traffic flat during a latency spike eliminates the capacity theory in one glance, which is most of the value.
- The frame fits request services well and needs extending for queues, batch and streaming, where lag or lateness is the real signal.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Demand → service: request rate rises, or a dependency slows, changing how long each request occupies a worker.
- 2Service → bounded resource: concurrent in-flight work exceeds the pool or worker count, and requests begin to wait.
- 3Queueing → latency: waiting time is added to every request, so p99 rises sharply while p50 moves modestly.
- 4Latency → saturation feedback: slower requests hold resources longer, raising occupancy further — the loop that turns a slowdown into an outage.
- 5Saturation → errors: once waits exceed client timeouts, failures appear and clients retry, adding load to a system already at its limit.
- • "CPU is at 30%, so we have headroom." CPU is one saturation signal among several; the pool at 100% with a wait queue is the one that matters here.
- • "Average latency is fine." Averages hide tail problems, and a rise in fast errors can push an average *down* during a degradation (see The Average Was Fine and Users Were Not).
- • "Errors are flat, so nothing is wrong." Slow-but-successful is the most common serious degradation and produces no error signal at all.
- • "Latency and traffic both rose, so we need more capacity." Sometimes true; also consistent with a change that made each request more expensive. Check the deploy timeline before scaling.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Latency: p50, p95 and p99 of request duration, split by route and by success/failure, over a window with a visible baseline.
- • Traffic: requests per second by route, so a demand-driven change is distinguishable from a code-driven one.
- • Errors: failed fraction including semantic failures that return 2xx, split by error class rather than one aggregate rate.
- • Saturation: occupancy and wait time for the bounded resources — connection pool, worker pool, queue depth — not only CPU and memory.
- • Build one golden-signals panel per service, split by route, with deploy markers overlaid — before building anything more elaborate.
- • Chart saturation for the actual bounded resources (pools, workers, queues), not only host CPU and memory.
- • Separate latency by outcome so failures cannot flatter the success-path numbers.
- • Extend the frame where the architecture demands it: consumer lag, oldest-message age, schedule lateness.
- • Tie the latency and error panels to the SLI used for alerting so the dashboard and the page agree (see [[sli]]).
- • Ask someone unfamiliar with the service to answer "is it healthy?" from the panel in under ten seconds.
- • Replay a past incident against the new panel: would the four signals have pointed at the right layer?
- • Confirm every alert that pages a human maps to one of the four signals, or to an explicit extension of them.
- • Splitting by route multiplies series count; it is worth the cardinality, but it is not free (see [[cardinality]]).
- • Saturation is the hardest of the four to instrument well, because the constraining resource differs per service and changes over time.
- • A four-panel dashboard is deliberately incomplete; it points a direction and does not diagnose, which frustrates teams expecting an answer.
- • Make the golden-signals panel part of service creation, so no service launches without it.
- • Review the panel after each incident: if the four signals did not point at the layer, add the signal that would have.
- • Alert on the signals rather than on causes; symptom-based alerting survives architectural change (see Alerts Worth Waking Someone For).
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEAll readings are a constructed teaching example. The relationships between the signals are the lesson; the absolute values are not benchmarks.
- WORKLOAD-SPECIFICThe four map cleanly onto request-serving systems. Queue consumers, batch jobs and streaming pipelines need lag or lateness signals the original framing does not include.