The question this answers
The message failed five times and went to the DLQ. Now what — and who finds out?
A DLQ guarantees that a message which exhausted its retry policy is retained rather than dropped, and that the main pipeline continues. It guarantees nothing about anyone noticing, about the message ever being processed, or about it surviving beyond the DLQ’s own retention period — after which the loss is silent and permanent.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The broker knows a message exceeded its attempt limit. It does not know why, whether the cause was the message or the world, or whether the business can tolerate the message never being processed. All three are outside the system, which is why the DLQ terminates in a human decision rather than in an automated one — and why an unowned DLQ leaves that decision permanently unmade.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The four steps that make it a safety mechanism
Moving a failed message out of the hot path solves a liveness problem, and it is the only part most teams implement. The DLQ becomes a *correctness* mechanism only when four further things exist, each of which has an owner and a runbook entry.
Test the design by asking a single question: if one message landed in this DLQ at 3am on a Saturday, what would happen? If the honest answer is "nothing until someone happens to look", you do not have a DLQ. You have a folder where data goes to be forgotten, and it is worse than a hard failure, because a hard failure at least stops something visible.
- Alert. Non-zero DLQ depth pages an owning team. Not a dashboard — a page, or at minimum a ticket that cannot be closed without a decision. Depth thresholds hide the first failure, which is the one worth catching.
- Inspect. Someone can read the payload, the failure reason, the attempt history and the original correlation id, without SSH-ing into a broker console. If the exception was not captured alongside the message, inspection is guesswork.
- Fix. The root cause is addressed: a handler bug, a producer bug, a schema mismatch, or a dependency that was down. This step is where the DLQ pays for itself, because a DLQ is a very high-signal bug report.
- Decide: replay or discard. Explicitly, per batch, with the reasoning recorded. Not "leave it there" — leaving it there is a decision to discard, made by not deciding and executed by retention.
| Missing piece | Symptom | Discovered by |
|---|---|---|
| No alerttypical | Messages accumulate for months | A customer, or a finance reconciliation |
| Alert with no ownertypical | Fires, is acknowledged, nothing happens | The same alert, again, indefinitely |
| No failure reason storedtypical | DLQ has payloads but no cause; nobody can triage | Whoever is asked to fix it |
| No replay toolingtypical | Fix ships, messages stay dead-lettered | A quarter-end report that is wrong |
| Replay is not idempotentassumption | Replaying duplicates effects; team becomes afraid to replay | A duplicate charge during recovery |
| DLQ has its own retentionprotocol | Messages silently expire before triage | Nobody — this failure is invisible by construction |
What must be stored alongside the message
A payload on its own is close to useless for triage. By the time anyone looks, the logs have rotated, the deploy has changed, and the dependency is healthy again. Whatever context is not attached to the dead-lettered message is gone.
Most brokers attach some metadata automatically — a receive count, a first-received timestamp, the source queue. Almost none attach the *exception*, because the broker never saw it. Capturing the failure reason therefore has to be done by the consumer at the moment of failure, usually by publishing to the DLQ yourself rather than relying on the broker’s automatic routing. That is a real trade: automatic routing is simpler and survives a consumer crash; manual routing carries the diagnosis but is skipped entirely if the process dies.
The pragmatic answer used by mature pipelines is both: let the broker auto-route as the backstop, and additionally write a structured failure record — message id, correlation id, handler version, exception class and message, attempt count, timestamp — to a durable store keyed by message id. Then the DLQ holds the payload and the store holds the reason, and triage joins them.
1interface DeadLetterRecord {2 messageId: string // producer-chosen; the join key for replay dedup3 correlationId: string // back to the originating request4 sourceQueue: string5 payload: unknown // exactly as received, unparsed if parsing failed6 firstReceivedAt: string7 deadLetteredAt: string8 attempts: number9 failure: {10 class: string // e.g. 'ValidationError'11 message: string12 consumerVersion: string // which deploy failed — often the whole answer13 stack?: string14 }15 // Filled in by a human, or by the replay tool. An empty decision on a16 // record older than the triage SLO is itself the alert.17 decision?: { action: 'replayed' | 'discarded'; by: string; at: string; why: string }18}Replay is a distributed operation with its own failure modes
Replaying a DLQ is not "put the messages back". It is publishing old messages into a live system, and every property you relied on in the original stream is different this time.
Ordering is gone. The messages left the stream at different times and re-enter as a batch, interleaved with current traffic. A replayed OrderUpdated may now arrive after a newer update it should have preceded, so replay silently applies stale data unless handlers are version-aware. Idempotency matters more, not less — some dead-lettered messages failed *after* a partial side effect, so replaying repeats it. Rate matters: dumping 200,000 messages back into a pipeline sized for steady state is a self-inflicted load test on a system you have just finished repairing.
Replay also needs a destination decision. Back to the original queue is the obvious choice and is usually right; a dedicated recovery queue with its own consumers is better when the volume is large, because it isolates the recovery from live traffic and lets you rate-limit it independently. Either way, replay through the normal handler — never through a one-off script that reimplements the logic, because that script is untested code operating on your worst data.
Retention is the trap nobody plans for
The DLQ is a queue, and queues have retention. Fourteen days is a common default. A message dead-lettered on the Friday before a holiday shutdown can expire before anyone triages it, and its expiry generates no alert, no log line and no metric that distinguishes it from a message someone deliberately deleted.
This is the sharpest form of the lesson: an un-alerted DLQ converts a loud failure into a silent one, on a timer. Without the DLQ, the message would have been retried forever and someone would eventually have noticed the loop. With an unmonitored DLQ, it fails quietly, waits quietly, and disappears quietly.
The mitigations are unglamorous and effective: alert on age of the oldest DLQ message, not depth, so a single message triggers it; set DLQ retention to the maximum the broker allows; and for anything money-adjacent, copy dead-lettered messages into durable storage you control, where the retention decision is yours rather than the broker’s. Then close the loop with Reconciliation Is a Component, Not a Cleanup Script against the source of truth, which is the only check that catches a message the DLQ itself lost.
Key points
- The DLQ solves liveness immediately and correctness only if alert, inspect, fix, and an explicit replay-or-discard decision all exist.
- The broker does not store why the message failed. Capture the exception, the consumer version and the correlation id yourself, or triage is guesswork.
- Replay re-enters a live stream out of order and after partial side effects; it requires version-aware, idempotent handlers and a rate limit.
- DLQs have retention. An untriaged message expires silently, which turns a loud failure into a quiet one on a timer.
- Alert on the age of the oldest DLQ message, not depth — a depth threshold hides the first failure, which is the one worth catching.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A message exceeds its retry policy — attempt count, elapsed deadline, or an immediate permanent-failure classification.
- • The consumer or the broker routes it to a dead-letter destination, and the main queue or partition resumes.
- • A structured failure record is written alongside, capturing the reason the broker cannot know.
- • An alert fires on the age of the oldest dead-lettered message and reaches an owning team.
- • A human inspects the payload and the reason, identifies the root cause, and ships a fix.
- • A replay tool re-publishes the affected messages through the normal handler, rate-limited, with dedup active — or the batch is explicitly discarded and the reason recorded.
- • The DLQ has no alert and accumulates unnoticed until retention expires.
- • The failure reason was never captured, so triage cannot proceed and the batch is discarded by default.
- • Replay duplicates side effects that partially committed before the original failure.
- • Replay applies stale data over newer state because handlers are not version-aware.
- • A high-volume replay saturates the pipeline or a downstream dependency, causing a second incident during recovery.
- • The DLQ itself becomes poisoned — its consumer fails on the same messages — and there is no DLQ for the DLQ.
- • Silent data loss by expiry: the operator eventually finds that 40,000 messages were dead-lettered over three months and are now gone. There is no record of them anywhere, and no alert ever fired, because depth was checked against a threshold of 1,000.
- • Untriageable DLQ: the queue contains payloads with no failure reasons. The team cannot tell which are safe to replay, so the standing decision becomes "purge", made informally and permanently.
- • Recovery-induced incident: a 200,000-message replay is released at full rate and takes down the same dependency that caused the original failures. The operator sees the outage repeat during the fix.
- • Stale overwrite after replay: customer records revert to older values with correct-looking audit trails. The operator sees data "going backwards" with no failing component anywhere.
- • DLQ redelivery loop: the DLQ consumer fails on the same messages and they cycle within the DLQ, generating log volume and cost while appearing to be under active processing.
- • Ownership gap: the alert fires into a channel belonging to a team that was reorganised. It is acknowledged every week and never actioned; the DLQ depth graph is a straight rising line for a year.
- • None inside the messaging layer — dead-lettering is a local decision by one consumer about one message.
- • The coordination is organisational: a named owner, an alerting route, a triage SLO, and a documented replay procedure. This is the part that decays silently as teams change, and it is the part that determines whether the mechanism works.
- • Replay coordination is real: rate limits, dedup state, and often a decision about whether to pause live traffic. Treat it as a change with a plan, not an operation someone runs ad hoc.
- • The main pipeline keeps moving; that guarantee is unconditional and is the DLQ’s core value.
- • Dead-lettered messages are durable only for the DLQ’s retention window. Beyond it, the guarantee is nothing.
- • Any invariant depending on the dead-lettered message remains broken until replay or compensation, and nothing in the system detects that except reconciliation.
- • Detect: alert on the age of the oldest message in every DLQ, with an owner per queue and a triage SLO measured in hours.
- • Contain: if arrivals are spiking, stop the source — a bad deploy, a schema change, a dependency outage — before triaging individual messages.
- • Recover: fix the cause, then replay through the normal handler, rate-limited, with deduplication active and a dry-run count first.
- • Reconcile: compare the source of truth against derived state for the affected window; replay covers what the DLQ still holds, and reconciliation covers what it lost.
- • Verify: DLQ empty or every remaining record carrying an explicit recorded decision, and the reconciliation delta back to zero.
- • Age of the oldest message per DLQ — the single most valuable metric here, and rarely the one configured.
- • DLQ arrival rate split by failure class, so a spike is attributable to a deploy, a producer, or a dependency without reading payloads.
- • Time from DLQ arrival to decision (the triage SLO), which measures whether the workflow exists rather than whether the queue does.
- • Count of records with no recorded decision older than the SLO — the direct measure of the unowned-DLQ failure.
- • Replay outcomes: how many replayed messages succeeded, failed again, or were suppressed as duplicates.
- • Any pipeline where a single bad message could otherwise block or consume the consumers — which is every pipeline with external input.
- • Debugging: a DLQ with captured failure reasons is one of the highest-signal bug reports available, because it contains the exact input that broke the code.
- • Deploy safety: a spike in DLQ arrivals is often the fastest signal that a release broke a consumer.
- • When it exists without an owner. It converts a loud repeating failure into a silent expiring one, which is strictly worse than having no DLQ at all.
- • When it is used as a substitute for fixing the producer, so the DLQ becomes a steady-state destination for a known-broken message class.
- • When replay tooling does not exist, so the DLQ can only ever be read and purged. That is an audit log, and it should be called one.
- • Fail loudly at the producer: schema validation and a schema registry stop malformed messages entering the system, so the DLQ never sees them.
- • A retry topic with tiered delays, which handles slow-to-resolve transient failures without involving a human at all — reserve the DLQ for genuinely undecidable cases.
- • Park failures in your own database table rather than a broker DLQ, when you need retention, queryability and joins with business data for triage. More work, far better triage.
- • For truly discardable streams, drop and count. Explicitly deciding that loss is acceptable is a legitimate design, and it is much better than an unmonitored DLQ pretending otherwise.
A dead-letter queue is a workflow, not a bin
What people believe, and what is true
We have a DLQ, so no messages are lost.
Messages are retained for the DLQ’s retention period. Without an alert and a triage workflow, the DLQ is a delayed, silent deletion.
The DLQ is where broken messages go.
It is where *undecidable* messages go. During a dependency outage it fills with perfectly valid messages, which is why "just purge it" is a dangerous habit.
Replaying is just re-enqueueing.
It publishes old messages into a live stream, out of order, possibly after partial side effects, at a rate the pipeline was not sized for. Each of those is a distinct way to make things worse.
Alert on DLQ depth > 100.
The first message is the interesting one. Alert on the age of the oldest message so a single dead letter is visible.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
When a message cannot be processed, move it out of the way so the pipeline continues — then make sure someone is told, can see why it failed, and decides explicitly whether to replay it or drop it.
Practical
Per DLQ: an owner, an alert on oldest-message age, a captured failure record with exception and consumer version, a replay tool that runs the normal handler with dedup and a rate limit, and a triage SLO. Set DLQ retention to the maximum, and copy money-adjacent dead letters into storage you control.
Advanced
The DLQ is where an undecidable question — is this failure permanent? — is escalated to the only oracle available, a human. That framing explains its properties: it cannot be automated away without re-introducing the misclassification it exists to avoid; its value is bounded by the latency and reliability of the human loop attached to it; and an unattached DLQ has negative value, because it removes the loud symptom (an infinite retry loop) that would otherwise have summoned the oracle by itself. Designing a DLQ is therefore mostly designing an escalation path, and only incidentally configuring a queue.
Apply it
- 🔧 Audit every DLQ in your estate for: an owner, an oldest-message-age alert, a captured failure reason, and replay tooling. Most estates fail on at least two.
- 🔧 Build a replay that is safe to run twice, and prove it by running it twice against a DLQ containing messages that partially committed.
- ⚡ A quarterly report is wrong. The cause is 12,000 messages that were dead-lettered and expired two months ago. Design the three controls that would each independently have caught it.
- ⚡ Replaying a DLQ overwrites newer customer data with older values. Explain the mechanism and the handler change that fixes it.
- 💬 A message lands in your DLQ at 3am on a Saturday. Walk me through what happens next in your system.
- 💬 You have 200,000 messages in a DLQ after a dependency outage. How do you replay them?
- 💬 Why might a DLQ be worse than no DLQ at all?