Dead-Letter Queues
Somewhere for work that will never succeed, so that one poison message cannot consume the fleet — and a human who is expected to look.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What happens to a message that fails every time, and who finds out?
A job that cannot succeed must stop consuming capacity, must not be lost, and must reach someone who can decide what to do with it.
Retry until it works. Failures are usually transient, so retrying indefinitely is the safest default — nothing is ever lost.
A malformed payload, a deleted referent or a code bug fails identically on attempt one and attempt ten thousand. Retrying is not resilience against those; it is a loop.
- A malformed payload, a deleted referent or a code bug fails identically on attempt one and attempt ten thousand. Retrying is not resilience against those; it is a loop.
- Each attempt occupies a consumer. A steady trickle of poison messages consumes a growing share of the fleet until legitimate work is starved (Queue Backlog).
- Retries with no backoff hammer a dependency that may already be the reason for the failure (Retry Storms).
- On a partitioned log there is no retry-in-place: a consumer that keeps throwing never commits its offset, and the whole partition stops advancing behind it.
- Nothing surfaces. The failure is inside a retry loop, so the error rate looks constant rather than alarming, and nobody learns that a whole class of work has stopped completing.
What is actually happening
- A dead-letter queue is a separate destination for messages that have exhausted their retry budget. It converts an unbounded retry loop into a bounded one plus a durable record that a human can act on.
- The trigger is normally an attempt count: after N receives without an ack, the broker moves the message. Some brokers also dead-letter on message expiry or on a queue length limit, and some require the application to publish to the DLQ itself.
- The DLQ is not an error log. It holds the actual messages, so its distinguishing capability is replay: fix the bug, deploy, and put the messages back on the main queue to be processed normally.
- Replay is only safe because consumers are idempotent. Replaying into a non-idempotent consumer duplicates whatever the failing messages had already partially done (Job Idempotency).
- Retry policy and dead-letter policy are one decision. Retrying a permanent failure ten times with backoff wastes ten attempts and delays the DLQ by the sum of the backoffs; retrying a transient failure only twice dead-letters work that would have succeeded. The split follows the error taxonomy: retry what is transient, dead-letter what is not, and fail fast on what is definitively invalid (An Error Taxonomy That Maps Cause to Response).
- A DLQ with no alert is strictly worse than no DLQ, because it converts a loud failure into a silent one and creates the impression that the problem is handled.
From failure to human
The path a doomed message takes has six steps, and most implementations stop after three. The last three are where the value is: a message sitting in a DLQ that nobody is alerted about has the same outcome as a message that was dropped.
Note where classification sits. Deciding *early* that an error is permanent saves every subsequent retry — a message with a missing required field does not become valid on attempt four.
- 11. Attempt
The consumer runs and throws.
fails by Swallowing the error and acking anyway — the failure never enters this pipeline at all.
- 22. Classify
Transient (retry), permanent (dead-letter now), or invalid (drop with a log).
fails by Treating every error identically, so permanent failures burn the whole retry budget (An Error Taxonomy That Maps Cause to Response).
- 33. Retry with backoff
Re-delivers after an increasing, jittered delay.
fails by No backoff hammers the dependency; no cap means step 4 never happens (Backoff and Jitter).
- 44. Dead-letter
Moves the message plus its failure reason to a terminal queue.
fails by No DLQ configured, or misconfigured permissions so the move silently fails.
- 55. Alert
Tells a human that work has stopped completing.
fails by No alert — the single most common gap, and the one that makes steps 1 to 4 pointless (Alerts Worth Waking Someone For).
- 66. Triage and replay
Group by cause, fix, replay at a controlled rate.
fails by Replaying into a still-broken consumer, or all at once into a recovering dependency.
Steps 5 and 6 are the ones that turn storage into a process. A DLQ without them is a queue whose consumer is nobody.
Retry the transient, dead-letter the rest
Retry budget is a resource, and spending it on failures that cannot succeed is how a small problem consumes a fleet. The classification is not subtle: does another attempt, later, plausibly produce a different result?
The third column is the one that changes behaviour. A 400-shaped failure that dead-letters immediately frees a consumer for real work and puts the message in front of a human minutes earlier.
| Failure | Another attempt helps? | What to do | Why |
|---|---|---|---|
| Dependency timeout or 503 | Yes | Retry with backoff and jitter | The classic transient failure; most retries succeed (Retries) |
| Rate limited by a third party | Yes, after the reset | Retry, honouring the retry-after hint | Retrying sooner makes it worse (Rate Limiting) |
| Deadlock or serialisation failure | Yes, immediately | Retry a bounded number of times | The conflicting transaction has gone by the next attempt (Deadlocks in Application Code) |
| Malformed payload / schema mismatch | No | Dead-letter on the first attempt | It will fail identically forever; retrying wastes consumers |
| Referenced row deleted | No | Dead-letter, or ack with a log if it is expected | Nothing will bring it back; decide which, deliberately |
| A bug in the consumer | No, until deployed | Dead-letter, fix, replay | Replay is precisely why the message is preserved |
| Poison message on a partitioned log | No | Catch, produce to a DLQ topic, commit the offset | Otherwise the partition never advances |
Replay is a privileged, rate-limited operation
Replay is the capability that justifies keeping the messages rather than logging the errors. It is also the operation most likely to be performed at speed, under pressure, by someone who has just deployed a fix — which is exactly when a naive replay does damage.
The version on the right is not more complex for its own sake. Rate limiting protects the dependency that has just recovered, a claim stops two operators replaying the same batch, and the outcome counter is what tells you whether the fix actually worked instead of assuming it did.
for (const msg of await dlq.receiveAll()) {
await mainQueue.send(msg.body)
}
// The dependency that just came back up receives the entire
// backlog at once. Two operators run this concurrently.
// Nothing records which messages were replayed, or whether
// any of them succeeded this time.await replay.run({
source: 'orders-dlq',
filter: { errorClass: 'PaymentProviderTimeout' }, // group by cause first
dryRun: false, // the default is true
ratePerSecond: 50, // protect the recovering dependency
maxMessages: 40_000,
actor: currentUser.id, // authorized, and audited
})
// Claims each message before re-publishing, so a second
// operator running this replays nothing twice; counts
// outcomes so "did the fix work" is a number, not a hope.A replay re-executes real business effects at whatever rate the loop achieves. Rate limiting keeps the recovery from causing a second outage, filtering by error class means you replay only what the fix addressed, the claim makes concurrent operators safe, and the audited actor makes a privileged bulk operation traceable. Idempotent consumers are the prerequisite that makes any of it safe at all (Job Idempotency).
How to build it
Most important first.
- Cap attempts on every queue and route exhausted messages to a DLQ. There is no queue for which unbounded retry is the right default.
- Classify errors in the handler. A validation failure or a missing referent should go straight to the DLQ on the first attempt rather than consuming five retries it can never use (An Error Taxonomy That Maps Cause to Response).
- Use exponential backoff with jitter between attempts so retries spread out rather than synchronising (Backoff and Jitter).
- Alert on DLQ depth greater than zero, or on its rate of increase. This is the alert that makes the whole mechanism real (Alerts Worth Waking Someone For).
- Record why the message died — the last error, the attempt count, the consumer version, the correlation id — alongside the message. A DLQ full of payloads with no failure reason is a pile of work nobody can triage.
- Build replay as a first-class, rate-limited operation with a dry-run mode, not an ad-hoc script someone writes during an incident.
- Set DLQ retention long enough for a real triage cycle — long enough to survive a weekend and a fix — and remember that anything sensitive in those payloads is retained for that long too.
- Give the DLQ an owner. "Someone will look at it" is how a dead-letter queue becomes a place work goes to be forgotten.
What can go wrong
- A DLQ that exists and is never monitored — the most common outcome, and the one that feels safest.
- The DLQ consumer itself failing, so messages fail to arrive in the place failures are supposed to arrive.
- Replay into a still-broken consumer, so messages round-trip back to the DLQ with a higher attempt count and no progress.
- Replay of a large backlog all at once, overwhelming a dependency that has just recovered (Backpressure).
- Replay into a non-idempotent consumer, duplicating side effects the original attempts had already performed.
- Attempt limits set so high that a permanently-failing message occupies consumers for hours before dead-lettering.
- DLQ retention expiring before anyone triages, silently discarding the work.
- A poison message on a partitioned log with no application-level diversion, halting a partition entirely.
- A message dead-lettering while a concurrent duplicate delivery of the same message succeeds, so the DLQ entry describes work that actually completed.
- Replay racing an in-flight retry of the same message, producing two concurrent executions (Job Idempotency).
- A fix deploying mid-replay, so some messages are processed by the old consumer and some by the new one (Rolling Deployments).
- Two operators replaying the same DLQ batch simultaneously — the reason replay needs a claim of its own.
- A DLQ accumulates exactly the messages that were malformed, hostile or unexpected — the attacker-shaped ones. Treat access to it as access to production data (Audit Trails).
- Retention is a data-retention obligation. Payloads with personal data sitting in a DLQ for weeks are inside the scope of your retention policy whether or not anyone has noticed (Secrets in Logs).
- Replay is a privileged operation that re-executes business effects. Require authorization, log who replayed what, and rate-limit it (Authorization in Backends).
- A publicly reachable or over-permissioned DLQ lets an attacker read failed payloads or inject messages that replay directly into your consumers (Public Exposure, Read With Context).
- "We have a DLQ, so failures are handled." A DLQ is where failures wait. Nothing is handled until someone looks, which is why the alert is the mechanism and the queue is just storage.
- "Everything should be retried the same number of times." A validation error and a timeout are different failures. Retrying the first is pure waste (An Error Taxonomy That Maps Cause to Response).
- "Replay is safe." Replay is safe when consumers are idempotent and the rate is controlled. Otherwise it duplicates effects and re-breaks whatever just recovered.
- "The DLQ is empty, so everything is fine." An empty DLQ can also mean nothing is routing to it — verify the wiring with a deliberately poisoned message rather than assuming (A Test Strategy Chosen by What Each Layer Can Prove).
- "Dead-lettering loses data." Dead-lettering *preserves* the message. Unbounded retry combined with a retention limit is what loses it.
Operating it
- DLQ depth and arrival rate, with an alert on any sustained increase. Zero is the normal value and any other value is a question (Six Queue Signals, Two That Wake You Up).
- Dead-letter reason grouped by job type and error class. One malformed field usually accounts for most of a DLQ, and grouping shows it immediately.
- Attempt-count distribution on the main queue. A shift toward the limit predicts dead-lettering before it happens.
- Age of the oldest DLQ message — how long triage is actually taking, as opposed to how long you intend it to take (Depth Is Not an Emergency; Age Is).
- Replay outcomes as their own counters. Messages that return to the DLQ after a replay mean the fix did not work.
- Correlation id preserved from the original request, so a DLQ entry can be traced back to the user action that produced it (Correlation Ids That Survive Every Hop).
- At 10x, the DLQ needs triage tooling: filtering, grouping by error, bulk replay and bulk discard. Reading messages one at a time stops working.
- At 10x, replay must be rate-limited. Putting a large backlog back on the main queue at once is a self-inflicted load spike on a dependency that has just recovered.
- At 100x, per-tenant or per-job-type DLQs stop one noisy source from burying everything else (Bulkheads).
- Volume changes the triage question from "what is wrong with this message" to "what are the five error classes and which is largest" — and that requires the reason to be recorded in a structured field (Structured Logging).
- A DLQ bounds retry cost and creates an operational obligation: something a human must watch and act on.
- A low attempt limit dead-letters faster and dead-letters work that would have succeeded on the next attempt.
- A high attempt limit rides out longer outages and lets poison messages consume capacity for longer.
- Long retention buys triage time and extends how long sensitive payloads persist.
- First-class replay tooling is real engineering effort, and without it replay happens as an unreviewed script during an incident.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALBounded retry plus a terminal destination plus an alert is the pattern everywhere; only the plumbing differs.
- CLOUD-SPECIFICSQS attaches a DLQ via a redrive policy with a maximum receive count, and offers redrive back to the source queue. Pub/Sub uses a dead-letter topic after a maximum delivery attempt count, and requires the subscription's service account to have publish permission on it — a misconfiguration that silently disables dead-lettering. RabbitMQ routes via a dead-letter exchange configured by policy, triggered by a nack with requeue false, a message TTL, or a queue length limit. Kafka has no dead-letter concept at all: the application must catch, produce to a DLQ topic, and commit the offset, or the partition stops.
- SIMPLIFIEDPresents one DLQ per queue. Larger systems separate by error class or tenant, and often add a retry queue with a longer delay between the main queue and the DLQ, so a dependency outage lasting hours does not dead-letter everything in it.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.