The question this answers
Why are there exactly two delivery semantics, and what decides which one I have?
At-most-once: every message is processed zero or one times; no duplicates, possible silent loss. At-least-once: every message is processed one or more times; no loss, guaranteed duplicates under failure. Exactly-once-within-a-scope: one *effect* per message for outputs inside a defined transactional boundary, achieved by making the ack and the output one atomic commit — and at-least-once for everything outside that boundary (Exactly-Once Is a Scope, Not a Guarantee).
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A consumer knows which messages it has received and which it has acknowledged. It does not know whether an ack it sent arrived, whether the broker recorded it before crashing, or whether the message it is about to process has already been processed by a previous incarnation of itself. The broker knows what it delivered and what was acked; it does not know what was *processed*, because processing happens in a system it cannot see.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Two actions, one crash, two orders
A consumer does two things with a message: it processes it, and it tells the broker it is done. These are separate actions on separate machines. A crash can occur between them, and the *order* you chose determines what the crash costs you. That is the entire content of delivery semantics.
Acknowledge first, then process. The broker removes the message immediately. If the consumer crashes during processing, nothing redelivers it — the message is gone and the work never happened. This is at-most-once: no duplicates, silent loss.
Process first, then acknowledge. If the consumer crashes after processing and before the ack lands, the broker still holds the message and redelivers it. The work happens again. This is at-least-once: no loss, duplicates.
There is no third position, because there is no point *inside* an atomic step at which to put the ack — unless you make the ack and the effect the same commit, which requires them to live in the same transactional store. That is the only construction that produces exactly-once behaviour, it is available only for outputs inside that store, and it is a special case of at-least-once with the duplicate collapsed at commit time rather than an escape from the dichotomy.
The trade-off, made concrete
Both semantics have a cost, and the costs have completely different shapes. At-most-once loses work silently — there is no error, no retry, no log line, only a message that no longer exists and an effect that never happened. At-least-once duplicates work noisily but detectably — the duplicate is visible at the sink and can be collapsed if the effect is idempotent.
That asymmetry is why at-least-once is the default everywhere that matters. A duplicate is a problem you can solve with a mechanism you already need. A silent loss is a problem you cannot solve at all, because you never learn it happened.
The duplicate rate under at-least-once is not a rare theoretical event either, and it helps to know where it actually comes from. It is roughly the rate of: consumer crashes, deploys (every rolling restart interrupts in-flight messages), visibility-timeout expiries on slow messages, consumer-group rebalances, and network failures on the ack path. Every deploy produces a burst of duplicates. If your consumer is not idempotent, your deploy cadence is your duplicate cadence, and the two will correlate perfectly on any dashboard that plots them together.
| At-most-once | At-least-once | Exactly-once-in-scope | |
|---|---|---|---|
| Ack placementprotocol | Before processing | After processing | Same commit as the output |
| On consumer crashprotocol | Message lost | Message redelivered | Transaction aborts, redelivered, no duplicate effect |
| Duplicatesprotocol | Never | Yes, routinely | Not for outputs inside the scope |
| Lossprotocol | Yes, silently | Never | Never |
| Consumer must beassumption | Nothing special | Idempotent | Transactional with the sink |
| Costtypical | Cheapest | Dedup state or idempotent effects | Coordination + latency + a constrained sink |
| Fitstypical | Telemetry, sampled metrics, live video | Almost everything | Stream processing into the same store |
Where at-most-once is genuinely right
It is a real choice, not a degraded one, and treating it as always-wrong leads teams to pay for durability they do not need on very high-volume paths.
The conditions are specific: the value of any individual message is negligible, the aggregate is what matters, and a duplicate would actively corrupt that aggregate. High-frequency telemetry samples fit exactly — losing one CPU sample in ten thousand is invisible, while counting one twice skews an average that someone will page on. Live video and audio frames fit: a dropped frame is a glitch, a duplicated frame is a stutter, and neither is worth a round trip. Presence and heartbeat signals fit, because the next one is a few seconds away and supersedes it (From Alive-or-Dead to a Suspicion Level).
The condition that disqualifies it is the one people miss: any message that carries state rather than a sample. A cache invalidation is not a sample — losing it leaves stale data forever. An "order created" event is not a sample. A balance update is not a sample. If the next message does not supersede this one, at-most-once is silently discarding business state.
The visibility timeout: where duplicates come from without a crash
Queue systems that hide a message rather than deleting it — SQS, and the same mechanism under many others — introduce a duplicate source that has nothing to do with crashes. The broker delivers the message and starts a timer. If the ack does not arrive before the timer expires, the message becomes visible again and is delivered to another consumer.
So a message that simply takes longer to process than the visibility timeout is redelivered while the first consumer is still working on it. Two consumers now process the same message concurrently, and both may commit. This is a duplicate produced by a healthy system doing exactly what it was configured to do.
The operator signature is distinctive and easy to misread: duplicate work with *overlapping* timestamps, rather than the sequential pattern a crash produces. It appears when processing latency rises — a slow dependency, a large payload, a cold cache — so it correlates with load rather than with failures. The fixes are to set the timeout above the p99.9 of processing time, to extend the timeout mid-processing via a heartbeat for genuinely long work, or to move long work out of the message handler entirely (Visibility Timeout: The Message Is Hidden, Not Yours).
# Crash duplicate — sequential, gap = redelivery delay
msg=a91f consumer=c-7 start=10:04:01.220 end=— (crashed)
msg=a91f consumer=c-2 start=10:04:31.882 end=10:04:32.104 ok
# Visibility-timeout duplicate — OVERLAPPING, both commit
msg=b30c consumer=c-4 start=10:11:02.010 end=10:11:47.551 ok
msg=b30c consumer=c-9 start=10:11:32.418 end=10:12:05.883 ok
^ redelivered 30s in, while c-4 still working
# vis_timeout=30s, p99 processing=44s. Every slow message duplicates.The practical model, and why it is not a compromise
The shape you meet in every mature system is at-least-once delivery plus idempotent processing. It is worth being clear that this is not a resigned compromise arrived at because exactly-once was too expensive. It is the correct answer, and it has a property the alternatives lack: it is *composable*.
An idempotent consumer stays correct when you add a second consumer, change brokers, replay from a dead-letter queue, re-drive a batch by hand, or migrate to a different region. It does not depend on a transactional boundary holding, on a particular broker feature being enabled, or on nobody ever touching the message by another path. It survives operations, which is where guarantees actually go to die.
The practical consequence for design: choose at-least-once by default, then spend your effort on the idempotence rather than on chasing a delivery guarantee. And be honest in documentation about which it is — "we use exactly-once delivery" written in a design doc is how a downstream team justifies not deduplicating, which is how the duplicate becomes a customer-visible bug two systems away.
- Default to at-least-once — the loss failure mode is unrecoverable, the duplicate one is not.
- Make the consumer idempotent — by natural key upsert where possible, by a claimed message id where not.
- Set the visibility timeout above p99.9 processing time — or extend it with a heartbeat.
- Expect a duplicate burst on every deploy — and make sure it is boring.
- Choose at-most-once deliberately, only for samples — and write down why.
- Never promise exactly-once without naming the scope (Exactly-Once Is a Scope, Not a Guarantee).
Key points
- Ack before processing gives at-most-once; ack after gives at-least-once. There is no third placement.
- The two semantics fail in opposite directions: silent loss versus detectable duplication.
- At-least-once is the default because a duplicate is solvable and a silent loss is not.
- Duplicates come from crashes, deploys, rebalances and visibility-timeout expiries — every deploy produces a burst.
- A visibility timeout shorter than processing time produces concurrent duplicate processing in a perfectly healthy system.
- At-most-once is correct for samples that the next message supersedes, and wrong for anything carrying state.
- At-least-once plus idempotent processing is the composable answer, not a compromise.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The broker delivers a message and either deletes it, hides it for a visibility timeout, or tracks a consumer offset.
- • The consumer chooses when to acknowledge relative to performing the effect.
- • Acking first tells the broker to forget the message before the effect is durable.
- • Acking after leaves the broker holding the message until the effect is durable, so any failure in between causes redelivery.
- • Redelivery may go to a different consumer instance, and may overlap with the original attempt.
- • The consumer collapses duplicates by making the effect idempotent, or accepts them.
- • A message that repeatedly fails is eventually routed to a dead-letter queue rather than redelivered forever (Poison Messages: The One That Fails Every Time, Forever).
- • The consumer crashes between processing and acking, causing redelivery.
- • The ack is lost in the network, so the broker redelivers a message that was processed.
- • Processing exceeds the visibility timeout and the message is redelivered concurrently.
- • A rebalance reassigns a partition mid-batch and the new owner reprocesses from the last committed offset (Rebalancing: Everyone Stops So the Partitions Can Move).
- • The broker crashes after acking and before persisting the ack, redelivering acknowledged messages.
- • An at-most-once consumer crashes after acking, losing the message with no signal.
- • A message fails permanently and is redelivered indefinitely, blocking a partition.
- • Duplicate bursts synchronised to deploys: the operator sees duplicate processing spike at every rolling restart and at no other time. Harmless with idempotent consumers, customer-visible without.
- • Concurrent duplicate processing under load: two consumers process the same message with overlapping timestamps whenever p99 latency crosses the visibility timeout. The rate tracks load, not errors, which makes it look like a capacity problem.
- • Silent gap under at-most-once: a downstream aggregate is slightly low and nobody can say why. There is no error, no DLQ entry and no redelivery — the message simply does not exist any more.
- • Redelivery storm after a broker restart: acks that were in flight are lost, and a large batch of already-processed messages is redelivered at once. Consumers see a spike of dedup hits, or of duplicate effects.
- • Stuck partition: one poison message is redelivered forever at the head of an ordered partition, and every message behind it stops. Consumer lag climbs on one partition only while the others look fine (Poison Messages: The One That Fails Every Time, Forever).
- • Acknowledgement is coordination between consumer and broker about what has been done, and it is unreliable in exactly the same way every other message is.
- • At-least-once needs no additional coordination — the redelivery is free and the cost moves to the consumer’s idempotence.
- • Exactly-once-in-scope requires the ack and the output to be one atomic commit, which forces both into the same transactional store and is the coordination that makes it possible (Exactly-Once Is a Scope, Not a Guarantee).
- • Ordering guarantees are a separate coordination axis: a broker that preserves order within a partition still gives no order across partitions, so a consumer cannot assume causality without carrying it (Happens-Before: The Only Ordering You Actually Have).
- • At-least-once preserves every message across any single failure; the cost is that some are processed more than once.
- • At-most-once preserves the no-duplicates property across any failure; the cost is that some messages are never processed and nothing records it.
- • The broker’s durability is independent of the consumer’s: messages survive consumer failure, and consumer effects survive broker failure.
- • Under a partition, an unacked message will be redelivered when connectivity returns, so a network failure translates into duplicates rather than loss.
- • Detect: measure redelivery rate and duplicate-processing rate separately, and correlate with deploys and with processing latency to distinguish the two duplicate sources.
- • Contain: route repeatedly failing messages to a dead-letter queue so one poison message cannot block a partition (A Dead-Letter Queue Is a Workflow, Not a Bin).
- • Recover: replay from the log or drain the DLQ, which is safe precisely because consumers are idempotent — and dangerous when they are not, since a replay is a mass duplicate event.
- • Reconcile: compare messages produced against effects at the sink by natural key; the difference reveals both loss and duplication.
- • Verify: after any broker or consumer incident, confirm the duplicate rate returns to baseline and no partition retains growing lag.
- • Redelivery rate per consumer group — the direct measure of how often at-least-once is doing its job.
- • Duplicate-processing rate, split by whether the attempts overlapped in time (visibility timeout) or were sequential (crash or rebalance).
- • Processing latency p99.9 against the visibility timeout, plotted together. Their crossing point is where duplicates start.
- • Consumer lag per partition, since a single stuck partition is invisible in an aggregate.
- • DLQ depth and arrival rate, and the age of the oldest DLQ message.
- • Ack failure rate — acks that errored or timed out, each of which becomes a future redelivery.
- • Every message-driven system: this choice is being made whether or not anyone made it deliberately, and naming it is what turns a surprise into a design.
- • Where a duplicate is cheap and a loss is expensive — at-least-once with idempotent consumers, which is most business processing.
- • Where the volume is enormous and each message is a sample — at-most-once, saving real cost on the ack path.
- • When diagnosing duplicates: knowing the two sources (crash-sequential and timeout-overlapping) identifies the fix immediately.
- • At-least-once with a non-idempotent consumer, which produces customer-visible duplicates on a schedule set by your deploy cadence.
- • At-most-once applied to state-carrying messages, where the loss is permanent and silent.
- • Very short visibility timeouts chosen for "faster recovery", which manufacture concurrent duplicates under normal load.
- • Treating a broker’s exactly-once mode as a licence to remove idempotence from consumers (Exactly-Once Is a Scope, Not a Guarantee).
- • Make the effect naturally idempotent so the semantic choice stops mattering for correctness (Idempotent Is a Property of the Whole Effect, Not the Write).
- • Upsert by natural key at the sink, which collapses duplicates without any dedup state (Deduplication: Bounded Memory Against an Unbounded Stream).
- • Commit the ack and the output atomically in one store, giving exactly-once effect within that store (Exactly-Once Is a Scope, Not a Guarantee).
- • Use a request/response call with a client-generated key instead of a queue, when the caller needs the answer anyway (The Retry Is a Decision, Not a Reflex).
- • Batch and reconcile: accept both duplicates and gaps in the stream, and derive the truth from a periodic full comparison against the source (Reconciliation Is a Component, Not a Cleanup Script).
Same crash, two ack placements, opposite damage
What people believe, and what is true
At-least-once and at-most-once are configuration options; exactly-once is the premium one.
The first two are the two possible ack placements. The third is not a placement — it is a construction that requires ack and output to commit atomically, and it only covers outputs inside that commit.
Duplicates only happen when something goes wrong.
Every deploy causes a rebalance and a burst of redeliveries, and a message slower than the visibility timeout duplicates during normal operation. Duplicates are routine, not exceptional.
A duplicate means the message was processed twice in sequence.
Under a visibility timeout the two attempts overlap. The dedup check must therefore be atomic, not merely present.
At-most-once is just a worse at-least-once.
For high-volume samples it is the correct choice: a duplicate corrupts the aggregate while a loss is invisible. It is wrong only for messages that carry state.
If the broker says it delivered the message, it was processed.
The broker knows about delivery and acks. Processing happens in a system it cannot observe, and the two can disagree in both directions.
Increasing the visibility timeout has no downside.
It also increases the time before a genuinely crashed consumer’s message is retried, so recovery from a real failure slows by the same amount. The right value is derived from processing latency, not from either extreme.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Acknowledge before you process and you can lose work; acknowledge after and you can repeat it. Pick the one whose failure you can live with — almost always the second.
Practical
Use at-least-once and make consumers idempotent. Set the visibility timeout above p99.9 processing time or extend it with a heartbeat. Chart redelivery rate against deploys and duplicate rate against latency, because the two duplicate sources have different fixes. Send poison messages to a DLQ so one bad message cannot stall a partition.
Advanced
The dichotomy is a consequence of the ack and the effect being two commits in two systems. Collapsing them into one commit is the only escape, and it constrains the output to the store that also holds the offset — which is why exactly-once systems are always described relative to a scope. Everything outside that store is at-least-once again, so the composable design is to keep consumers idempotent even when a transactional mode is available.
Apply it
- 🔧 Implement both ack orders against the same broker and kill the consumer mid-processing. Confirm the loss in one and the duplicate in the other.
- 🔧 Set the visibility timeout below your p99 processing time and measure the resulting concurrent-duplicate rate. Then plot duplicate rate against the ratio of timeout to latency.
- 🔧 Correlate your duplicate-processing metric with your deploy markers for the last month.
- ⚡ A team reports duplicate emails only during business hours. Message volume is flat across the day but processing latency is higher when the cache is cold. Explain.
- ⚡ After migrating brokers, a downstream aggregate is consistently 0.03% low. No errors anywhere. What changed?
- 💬 Why are there only two delivery semantics? Derive them from first principles.
- 💬 Name four sources of duplicates in an at-least-once system that are not crashes.
- 💬 When would you deliberately choose at-most-once?
- 💬 Your duplicate rate spikes every Tuesday at 14:00. What is your first hypothesis?
- 💬 How do you tell a visibility-timeout duplicate from a crash duplicate by looking at logs?