Six Queue Signals, Two That Wake You Up
Arrival rate, processing rate, depth, oldest-message age, retry volume and dead-letter volume. Depth is the number everyone graphs and the number that explains the least; the rate pair tells you whether you are falling behind, and age tells you whether a human is already suffering.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The six numbers, read in order
A queue is the rare part of a distributed system where the physics is visible. Work arrives at some rate, work is consumed at some rate, and the difference accumulates. Everything else — depth, age, retries, dead letters — is a consequence of those two numbers and how long they have disagreed.
That is why the reading order matters. Start with arrival rate and processing rate together: if arrivals exceed processing, nothing else you look at will improve on its own, and the only question left is how long you have. Depth tells you how much disagreement has already accumulated. Oldest-message age converts that into the thing a user experiences: *how long has the unluckiest piece of work been waiting?*
Retry volume and dead-letter volume are the second tier — they tell you whether the queue is falling behind because there is more work, or because the *same* work keeps coming back. A doubling of arrival rate with flat unique-job count is a retry problem wearing a traffic costume, and scaling workers to meet it will happily burn money processing the same failures faster (see Retry Storms: The Load You Generated Yourself).
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| arrival rate | 10,000 jobs/s | Up 25% from the daily baseline of 8,000/s — real traffic growth, or retries in disguise? | suspect |
| processing rate | 8,000 jobs/s | Flat, at the same rate as yesterday. The consumers are not slower; there is simply more work than they can take. | smoking gun |
| queue depth | 1,240,000 | Large and growing — but on its own this is the *consequence*, not the diagnosis. It would look identical for a 10x spike that ended an hour ago. | normal |
| oldest message age | 4m 10s | The unluckiest job has waited over four minutes. If this queue backs a "your export is ready" email, the SLO is already breached. | smoking gun |
| retry rate | 120/s | 1.2% of arrivals. Steady, not climbing — this is background noise, not amplification. | normal |
| dead-letter rate | 3/s | Low and stable. Jobs are not failing permanently; they are waiting. | normal |
Why depth is the number everyone graphs and nobody can act on
Queue depth has no natural scale. Is 50,000 messages a lot? For a queue processing 100,000/s with sub-second jobs, that is half a second of work and completely healthy. For a queue processing 10/s with jobs that take a minute each, that is over a month of backlog and the business is effectively down.
The same ambiguity runs the other way: depth *falls* when consumers crash and producers give up, when a poison message stalls a partition and everything behind it stops being counted, and when someone purges the queue during an incident. A depth alert that fires on "too high" is silent for three of the worst failure modes a queue has.
This is the USE: Utilization, Saturation, Errors mistake in queue form — utilization-shaped thinking applied to something that needs a saturation-shaped signal. The fix is not to stop graphing depth; it is to alert on the two signals that carry their own scale: the rate ratio (dimensionless — is it above 1?) and oldest-message age (in seconds — compare it to the promise you made a user).
| Signal | Answers | Blind to | Alert or graph? |
|---|---|---|---|
| arrival ÷ processing rate | Are we falling behind *right now*? | How much damage has already accumulated | Alert — sustained ratio > 1 for N minutes |
| oldest message age | How long has the unluckiest work waited? | Whether the cause is volume or slowness | Alert — this is the user-facing number |
| queue depth | How much has accumulated | Job cost, so it has no fixed scale; also drops on consumer death | Graph — useful for time-to-drain, poor as a threshold |
| retry rate | Is the same work circulating? | Which dependency is rejecting it | Alert on the *ratio* to arrivals, not the raw count |
| dead-letter rate | What has permanently failed? | Silent stalls that never reach the DLQ at all | Alert — any sustained non-zero rate deserves a look |
| consumer count / utilization | Are the workers even running? | Whether workers are busy or blocked downstream | Alert on *zero* consumers; graph the rest |
The measurement that is easy to get wrong
Oldest-message age is the signal worth the most and the one most often computed incorrectly. The naive implementation records now - enqueued_at when a job is *dequeued* — which means a queue that has completely stalled reports an age of zero, because nothing is being dequeued to measure. The metric goes quiet exactly when the incident starts.
Measure it from the head of the queue instead: periodically peek at the oldest un-acknowledged message and emit now - its enqueue timestamp, whether or not anything is consuming. Some brokers expose this directly; where they do not, a small sampling loop that peeks without consuming is worth the effort. Publish it per priority class too — a shared queue where the "welcome email" backlog hides a stalled "payment reconciliation" job is a queue with one metric and two very different SLOs.
One honest caveat: on partitioned logs (Kafka-style), "the queue" is several independent queues, and the aggregate hides the failure. Consumer lag must be read per partition — one stalled partition among thirty is a 3% aggregate blip and a 100% outage for every user whose key hashes there. See Hot Keys: When Aggregate Metrics Hide a Saturated Node for the same shape in caches, and Architecture → Kafka-style logs for the partition model itself.
1# WRONG — goes silent exactly when the queue stalls2on_job_dequeued(job):3 emit_gauge("queue.age_seconds", now() - job.enqueued_at)4 # no dequeues -> no samples -> dashboard shows "no data", alert never fires5 6# RIGHT — sample the head, independent of consumption7every 10 seconds:8 for partition in queue.partitions: # per partition, never aggregated9 head = partition.peek() # does not consume10 age = head ? now() - head.enqueued_at : 011 emit_gauge("queue.head_age_seconds", age,12 labels={partition: partition.id, priority: partition.priority})13 14# Alert on the promise you made, not on a round number:15# head_age_seconds{priority="user_facing"} > 60 for 5m16# head_age_seconds{priority="batch"} > 1800 for 15mKey points
- Read the rate pair first: arrival vs processing is the only signal that says whether you are falling behind at this instant.
- Queue depth has no natural scale — it is high in healthy busy systems and drops when consumers die, so it is a poor alert threshold.
- Oldest-message age is the user-facing number: it converts accumulated backlog into "how long has someone been waiting".
- Measure age from the head of the queue, not at dequeue time, or the metric goes silent during exactly the stall you need it for.
- On partitioned queues, read lag per partition — one stalled partition is invisible in the aggregate and total for the users on it.
The Queueing Curve
Change an input and watch which number moves — and which one does not.
At ρ = 0.80 the queue is real but modest. This is the last comfortable zone — note how little headroom is left before the curve turns upward.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Producers → queue: arrival rate rises from 8,000/s to 10,000/s while unique-job count rises proportionally, so this is real work, not retries.
- 2Queue → consumers: processing rate holds flat at 8,000/s; consumer count and per-consumer throughput are unchanged, so consumers are not degraded.
- 3Rate pair → depth: the 2,000/s difference accumulates; depth climbs linearly, which is the signature of a capacity shortfall rather than a stall.
- 4Depth → age: head-of-queue age crosses 60s, breaching the user-facing promise even though zero jobs have failed.
- 5Age → users: support tickets arrive about "missing" emails that are not missing, only queued behind 1.2 million other jobs.
- • "Depth is huge, so the consumers are broken" — the consumers are processing at exactly their normal rate; the arrival side changed.
- • "Depth dropped, we are recovering" — depth also drops when consumers crash, when a queue is purged, and when producers start failing.
- • "No errors, so the queue is healthy" — a queue that is falling behind reports zero errors right up until the timeouts start.
- • "Aggregate consumer lag is 3%, that is fine" — on a partitioned log that can be one partition at 100% and twenty-nine at zero.
- • "Retries are up because traffic is up" — check the ratio; a rising retry *fraction* means the work is failing, not that there is more of it.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Arrival and processing rate as a pair, on one graph, same units: `rate(queue_enqueued_total[5m])` against `rate(queue_processed_total[5m])`.
- • Head-of-queue age sampled on a timer (`queue.head_age_seconds`), labelled by partition and priority class, not computed at dequeue.
- • Retry rate as a *ratio* to arrivals, so a doubling of traffic and a doubling of retries look different.
- • Consumer count and per-consumer utilization, to distinguish "workers are saturated" from "workers are gone".
- • Dead-letter rate, plus the age of the oldest DLQ entry — a DLQ nobody drains is a silent data-loss queue.
- • Alert on the rate ratio and head-of-queue age; demote depth to a graph used for time-to-drain arithmetic.
- • Emit head-of-queue age per partition and per priority class so a stalled slice cannot hide inside a healthy aggregate.
- • Separate user-facing work from batch work into different queues with different age SLOs, so one cannot starve the other.
- • Track unique jobs alongside total arrivals so retry amplification is distinguishable from traffic growth.
- • Give the dead-letter queue an owner, an age alert and a documented redrive procedure — otherwise it is where jobs go to be forgotten.
- • Replay a known stall in staging (stop consumers for 10 minutes): head-of-queue age must climb continuously; the old dequeue-time metric will show "no data".
- • Confirm the age alert fires before the user-visible promise is breached, not after — compare alert fire time to the SLO threshold on the same timeline.
- • After splitting queues, verify batch backlog growth leaves the user-facing queue's head age flat under the same total load.
- • Head-of-queue sampling costs a periodic peek against the broker; on very high-partition-count topics that is real API load and needs its own budget.
- • Splitting queues by priority multiplies the operational surface: more consumers to size, more alerts, more dashboards, more ways to misconfigure one.
- • Per-partition metrics multiply cardinality — a 200-partition topic with three labels is a real cost, and [[cardinality]] applies here too.
- • An alert on
head_age_secondsper priority class, tied to the documented SLO for that class rather than a round number. - • An alert on zero consumers and on sustained rate ratio > 1, both of which catch failure modes a depth threshold misses entirely.
- • A dashboard panel that always shows arrival and processing rate on the same axis — reviewers should never see depth without the rate pair beside it.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe 10,000/s and 8,000/s figures are chosen to make the arithmetic legible. Real queue rates span many orders of magnitude and the thresholds that matter come from your own SLO, not from these numbers.
- WORKLOAD-SPECIFICWhether a given depth is healthy depends entirely on per-job cost. The same depth is half a second of work or a month of it depending on the workload.
Misconceptions
Apply it
Where the depth lives
Per-partition lag only makes sense once you know that a Kafka-style topic is N independent ordered queues with N independent consumers.