At-Least-Once Delivery
Queues can guarantee a message is delivered at least once; only your consumer can guarantee the business effect happens once.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
The queue advertises exactly-once. Does that mean my consumer runs once?
Every order placed must result in exactly one confirmation email and exactly one warehouse instruction, using a queue between the API and the workers.
The broker documentation says exactly-once delivery, so the consumer can do whatever it likes. Read the message, send the email, acknowledge.
The consumer sends the email, then crashes before acknowledging. The broker's visibility timeout expires, the message is redelivered, and a second email is sent. Nothing about the broker was violated.
- The consumer sends the email, then crashes before acknowledging. The broker's visibility timeout expires, the message is redelivered, and a second email is sent. Nothing about the broker was violated.
- The consumer is slow — a downstream API is degraded — and exceeds the visibility timeout while still working. The broker redelivers to a second consumer, and now two consumers are processing the same message concurrently.
- The broker's exactly-once guarantee turns out to apply within its own boundary: a deduplication window on the producer side, or exactly-once between the broker and a specific sink it controls. It does not cover the side effects your consumer performs.
- The producer retries an enqueue after a timeout, so the same logical message enters the queue twice with different message ids. Broker-side deduplication keyed on message id sees two distinct messages.
- A rebalance moves a partition to another consumer between processing and offset commit, and the new owner reprocesses from the last committed offset.
What is actually happening
- There are three delivery semantics, and they are properties of the channel: at-most-once (acknowledge before processing — messages can be lost), at-least-once (acknowledge after processing — messages can be duplicated), and exactly-once (which requires the acknowledgement and the effect to be atomic).
- Acknowledgement and effect can only be atomic if they share a transaction. That is possible when the effect is a write to the same system as the offset — a transactional broker writing to its own log, or a consumer writing offsets into the same database as its output. It is not possible when the effect is an email, a card charge, or a call to another service.
- So the honest formulation is: the channel can offer at-least-once; end-to-end exactly-once processing is achieved by an at-least-once channel plus an idempotent consumer. The idempotency is what collapses N deliveries into one effect (Duplicate Detection).
- Visibility timeouts are the usual duplication source and are widely misunderstood. The timeout is not "how long the consumer may take"; it is "how long before the broker assumes the consumer died". A consumer that takes longer is not misbehaving — it is being redelivered.
- Producer-side retries create duplicates before the queue even sees them, which broker deduplication on message id cannot catch because the two enqueues have different ids. A producer-assigned business key is what makes them recognisable (Idempotency Keys).
- Kafka's transactional producer and read-process-write pattern genuinely give exactly-once within Kafka: consume from a topic, produce to a topic, commit offsets, all in one transaction. The moment the effect leaves that boundary — an HTTP call, a row in Postgres — the guarantee stops at the boundary (The Transactional Outbox).
Two guarantees that share a name
When a broker says "exactly-once", it is making a claim about a boundary it controls. When an engineer hears it, they usually understand a claim about their business effect. Those are different statements, and the gap between them is where duplicate charges live.
Being precise about it is not pedantry, because the imprecision has a specific cost: teams skip consumer idempotency on the strength of a vendor page, and then discover the gap through a customer complaint. The useful habit is to ask, of any delivery guarantee, between which two points does it hold?
For a broker, the honest answer is almost always "from the producer's successful publish to the consumer's acknowledgement". Your email send, your card charge, your row in another database — all of that is downstream of the acknowledgement and outside the guarantee.
| Claim | What is actually guaranteed | What it does not cover |
|---|---|---|
| At-most-once delivery | Acknowledged before processing; never delivered twice | The message may never be processed at all |
| At-least-once delivery | Acknowledged after processing; never lost once accepted | The consumer may run more than once |
| Exactly-once delivery | Not achievable over an unreliable channel | Everything — treat the phrase as marketing unless a boundary is named |
| Exactly-once processing | The observable effect happens once | Nothing — but it is built by you, from at-least-once plus idempotency |
| Broker deduplication window | Duplicate publishes within N minutes are collapsed | A producer retry with a new id; a redelivery after the window |
| Transactional read-process-write | Consume, produce and commit atomically within one broker | Any effect outside that broker: HTTP, email, another database |
| Ordered delivery per partition/group | Messages in one scope arrive in order | Ordering across scopes; ordering after a consumer-side retry |
The visibility timeout is a liveness assumption
Most real duplicates do not come from broker failures. They come from a consumer that took longer than the broker expected, because a downstream dependency was slow. The broker cannot distinguish a slow consumer from a dead one, so after the visibility timeout it assumes the worst and gives the message to someone else.
That makes the timeout a tuning parameter with correctness consequences. Set it from the p99 of processing time, not the mean, and remember that the p99 moves when a dependency degrades — which is precisely the moment you least want duplicate processing.
The deeper point is that no timeout setting removes the duplicate; it only changes how often it happens. Slow consumers, crashes, rebalances and deploys all produce redelivery. The consumer must be idempotent regardless, and the timeout tuning is about how much duplicate work you do, not about whether the effect is safe.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Downstream API degrades, processing slows | Duplicate rate rises with latency | Processing exceeded the visibility timeout | Idempotent consumer; extend visibility via heartbeat; alert when p99 approaches the timeout |
| Consumer crashes after the effect, before the ack | Effect applied twice on redelivery | Effect and acknowledgement are not atomic | Deduplicate on the business key inside the same transaction as the effect |
| Producer retries a timed-out enqueue | Two messages, different ids, one intent | Broker dedupe keys on message id | Producer-assigned business key in the body (Idempotency Keys) |
| Deploy restarts consumers mid-message | A burst of duplicates on every release | In-flight messages redelivered on shutdown | Graceful shutdown that finishes or explicitly releases in-flight work (Graceful Shutdown) |
| Consumer group rebalance | Reprocessing from the last committed offset | Partition reassigned between effect and commit | Idempotent consumer; commit offsets promptly |
| Batch acked as a unit | Whole batch reprocessed after a mid-batch crash | Acknowledgement granularity coarser than processing | Per-message dedupe keys, so reprocessing is a no-op for the ones already done |
Building exactly-once processing out of at-least-once delivery
ON CONFLICT DO NOTHING claim and its rowcount are Postgres. MySQL needs an explicit duplicate-key catch or INSERT IGNORE with attention to its affected-row semantics; a document store needs a unique index and an insert that fails rather than upserts. The requirement is only that the claim is one atomic operation, which every serious store can express somehow.The construction is short and it is the payoff of the whole module. Take an at-least-once channel. Give every message a producer-assigned business key. In the consumer, claim that key atomically in the same transaction as the effect. Acknowledge afterwards.
Redelivery then costs a failed insert and an acknowledgement. Concurrent redelivery costs the same, because the unique constraint admits exactly one claimant. A crash before the acknowledgement redelivers a message whose key is already claimed, and the consumer correctly does nothing.
What this does not give you is atomicity with external effects. When the consumer charges a card, the claim can commit while the charge is in flight, and there is no transaction spanning both. There the answer is the same as everywhere else in this module: pass an idempotency key outward, and let the external system deduplicate on its side.
1def handle(message) -> None:2 # The business key comes from the producer and is stable across a3 # producer retry AND across a broker redelivery. The broker's own4 # message id is neither.5 key = message.body["order_id"]6 7 with db.transaction() as tx:8 claimed = tx.execute(9 """INSERT INTO processed_messages (consumer, business_key, at)10 VALUES (%s, %s, now())11 ON CONFLICT (consumer, business_key) DO NOTHING""",12 ("order-confirmation", key),13 ).rowcount14 15 if claimed == 0:16 metrics.increment("consumer.duplicate_suppressed")17 return # ack outside; effect already applied18 19 # Local effect: same transaction as the claim, so a rollback20 # undoes both and a crash leaves neither.21 tx.execute("UPDATE orders SET confirmed_at = now() WHERE id = %s", (key,))22 23 # External effect: cannot join the transaction. Push idempotency24 # outward with a deterministic key so the provider absorbs a repeat.25 mailer.send(26 template="order-confirmation",27 to=order.email,28 idempotency_key=f"order-confirmation:{key}",29 )Two boundaries, two mechanisms. Inside the database, the claim and the effect are atomic. Outside it, atomicity is unavailable and a deterministic key delegates the deduplication to whoever owns the effect.
How to build it
Most important first.
- Design every consumer as if it will be invoked more than once with the same message, because it will. That single assumption is the whole discipline.
- Deduplicate on a producer-assigned business key carried in the message body — an order id, an event id — not on the broker's message id, which changes when the producer retries (Job Idempotency).
- Make the effect and the dedupe record atomic where both are local: one transaction containing the business write and the processed-key insert (Where the Transaction Boundary Goes).
- Set the visibility timeout from the observed p99 of processing, and extend it explicitly for long tasks if the broker supports heartbeats. A timeout below your real processing time guarantees duplicates.
- Acknowledge after the effect, not before, unless you have decided that losing messages is preferable to duplicating them — a legitimate choice for telemetry and never for payments.
- Where the effect is external, pass an idempotency key derived from the business key so the external system absorbs the repeat (Idempotency Keys).
- Give the consumer a dead-letter path and a retry ceiling, so a permanently failing message cannot be redelivered indefinitely (Dead-Letter Queues).
What can go wrong
- Acknowledging before processing to "avoid duplicates", which converts duplication into silent message loss — usually discovered weeks later as missing records.
- Deduplicating on the broker's message id, which does not survive a producer retry.
- A visibility timeout tuned to the average rather than the tail, so the slowest 1% of messages are always processed twice.
- Consumer-side deduplication in memory, which fails across a consumer restart or a scale-out (Stateless Services).
- Trusting a managed broker's deduplication window without checking its length; a redelivery outside the window is not deduplicated at all.
- Batch consumers that acknowledge the whole batch after processing, so a mid-batch crash reprocesses messages already handled.
- Believing the Kafka exactly-once configuration covers a consumer that writes to an external database, which it does not.
- A slow consumer and its redelivered copy processing the same message simultaneously — the visibility-timeout race, and the most common source of concurrent duplicate processing (Backend Races).
- A consumer rebalance handing a partition to a new owner between the effect and the offset commit.
- Two consumers claiming the same business key concurrently, resolved only by an atomic claim on it, not by a check (Duplicate Detection).
- A producer retry racing the original enqueue, so both land and neither is a redelivery.
- Deduplication-state expiry racing a very late redelivery.
- A replayed message is a valid message. If processing a message grants something — credits, access, a refund — a duplicate is an amplification vector, and deduplication is a security control (Replay Attacks).
- Do not use a client-controllable field as the deduplication key without validation; an attacker choosing the key can suppress a legitimate message by pre-claiming it.
- Messages in a queue are data at rest with the sensitivity of their payload. Prefer ids over full records in message bodies.
- "Our queue is exactly-once, so we do not need idempotency." The most consequential misreading in this domain. Delivery semantics and processing semantics are different guarantees over different boundaries, and the vendor is describing theirs, not yours.
- "Exactly-once is impossible." Exactly-once *delivery* over an unreliable network is impossible; exactly-once *processing* is routine and is built from at-least-once delivery plus idempotent effects.
- "Acknowledging first avoids duplicates." It does, by losing messages instead. That is a different guarantee, not a better one.
- "The visibility timeout is how long the consumer is allowed to take." It is how long before the broker concludes the consumer is dead and gives the message to someone else.
- "Deduplication on message id is enough." Not against a producer retry, which is where a large share of real duplicates come from.
Operating it
- Redelivery count per message, if the broker exposes it. A rising mean is the earliest signal that processing time is approaching the visibility timeout.
- A
duplicate_suppressedcounter in the consumer. Zero forever usually means the deduplication is not wired in, not that duplicates never happen. - Processing duration p99 plotted against the visibility timeout on the same chart. The crossing point is where duplicates begin.
- Dead-letter depth and age, with an alert on both. A message that has exhausted retries is a message no one will ever process again (Queue Backlog).
- Compare messages consumed against distinct business keys processed. The gap is your real duplicate rate, and it is the only number that reflects what customers experience.
- Duplicate rate rises with load, because load raises processing time toward the visibility timeout. The consumer that never duplicated at low volume duplicates continuously at high volume.
- At 10x, the deduplication store becomes a per-message write on the hot path and needs the same design attention as the idempotency key store (Idempotency Storage).
- At 100x, deduplication state retention becomes the constraint: it must outlive the maximum redelivery window, and that product against message rate is the storage bill.
- More consumers means more concurrent processing of the same partition after a rebalance, so the window in which two consumers hold one message widens with the size of the consumer group.
- At-least-once plus idempotent consumers is the workable design and costs a deduplication write per message plus the storage to retain it.
- At-most-once is cheaper and loses messages. For metrics and telemetry that is often correct; the mistake is applying the same reasoning to a payment.
- Transactional exactly-once within a single broker is real and constrains you to effects inside that broker, which most business logic is not.
- A longer visibility timeout reduces duplicates and delays recovery when a consumer genuinely dies, because the broker waits that long before redelivering.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe separation between channel delivery semantics and consumer processing semantics holds for every broker.
- PROTOCOL-SPECIFICKafka offers transactional read-process-write giving exactly-once *within Kafka*, with offsets committed in the same transaction as the produced records; SQS FIFO offers producer-side deduplication over a fixed window and a per-message-group ordering guarantee; RabbitMQ offers at-least-once with acknowledgements and no deduplication at all. Their guarantees are not interchangeable, and none of them covers an HTTP call your consumer makes.
- SIMPLIFIEDThis lesson treats a broker as one channel with a visibility timeout. Partitioned logs with consumer groups and offset commits have a materially different redelivery model — the duplicate window is bounded by the last committed offset rather than by a per-message timer — and the conclusion for consumer design is the same.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — the impossibility of exactly-once delivery, and why every practical system builds exactly-once processing instead.
- — Testing & Reliability Engineering — how to test a consumer for idempotency: replay the same message twice and diff the resulting state.