The question this answers
When exactly should my consumer tell the broker it is done — and what does the answer cost me?
Acknowledging after processing gives at-least-once: every message is processed one or more times, and none is lost to a consumer crash. Acknowledging before processing gives at-most-once: no message is processed twice, and a crash loses the message permanently. The broker cannot offer a third option, because the ack and the processing are not in the same transaction.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The broker knows whether it has received an acknowledgement. It does not know whether the work was done — an ack is a claim by the consumer, and a consumer that crashed after doing the work but before acking made a true claim that never arrived. The consumer, symmetrically, does not know whether its ack was received, so it cannot know whether this delivery is the first or the second.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Three steps, two orderings, and no third choice
The consumer does three things: it receives the message, it does the work, it acknowledges. Only the position of the third relative to the second is under your control, and there are exactly two positions.
Ack after processing means a crash anywhere before the ack results in redelivery. Nothing is lost. Duplicates are guaranteed to occur eventually, because a crash between the side effect and the ack is not an exotic scenario — it is what a rolling deploy does several times a week. This is the default for anything that matters.
Ack before processing — including the widespread "auto-ack on delivery" setting — means a crash after the ack loses the message with no trace anywhere. The broker recorded a success. The work never happened. There is no log line, no error, no retry, and nothing to reconcile against unless you built it separately. Auto-ack is fine for a metrics sample and catastrophic for a payment.
The instinct at this point is to look for a way to make the ack and the side effect atomic. They are in different systems, so that requires Two-Phase Commit: Buying Atomicity With a Promise across your database and your broker, which almost nobody runs. The practical resolution is the one this domain reaches over and over: accept at-least-once and make the processing Idempotent Is a Property of the Whole Effect, Not the Write.
A lost ack is indistinguishable from no ack
Even a consumer that never crashes produces duplicates. It processes the message, sends the ack, and the ack is lost in the network. The broker waits, times out, redelivers. The consumer sees the same message again with no way to tell that its previous ack was sent successfully — this is A Timeout Tells You Nothing About Whether It Happened pointed in the opposite direction.
This matters because it removes the last escape route. "We will just make the consumer very reliable" does not help: the failure is in the channel, not the consumer. Any protocol that requires an acknowledgement to be delivered over an unreliable network inherits the possibility of a duplicate, permanently.
The same logic disposes of the negative acknowledgement. A nack tells the broker "I failed, return this now" rather than waiting for a timeout, which is a valuable latency optimisation and not a semantic change. A lost nack degrades into a timeout, and the message comes back anyway.
| Placement | Delivery semantics | Failure signature | Required of the handler |
|---|---|---|---|
| Auto-ack on deliveryprotocol | At-most-once | Silent permanent loss on crash | Nothing — and nothing is what you get |
| Ack before side effectprotocol | At-most-once | Message counted as done, work missing | Nothing |
| Ack after side effectprotocol | At-least-once | Duplicate side effects on crash or lost ack | Idempotence |
| Ack after side effect, dedup table in the same transactionassumption | Effectively-once *for that store* | Duplicate suppressed; dedup table is now a dependency | A stable message identity |
| Batch ack (ack N at once)typical | At-least-once, coarser | A crash replays the whole batch | Idempotence, and tolerance for large replays |
Effectively-once: what you can actually build
You cannot have exactly-once *delivery*. You can have effectively-once *processing*, and the difference is where the guarantee lives. Delivery is a property of the network, which cannot provide it. Processing is a property of your handler, which can — if the side effect and the record of having done it commit together.
Concretely: write the dedup record and the business effect in one local database transaction, keyed by a stable message identity the producer chose (not the broker’s delivery id, which may differ per delivery). On redelivery, the insert conflicts, the handler recognises the duplicate, skips the effect, and acks. The duplicate still arrives — you have simply made it a no-op.
This works exactly as far as your transaction boundary reaches, which is the caveat that gets forgotten. If the side effect is an HTTP call to a third party, there is no shared transaction, and the best available answer is an idempotency key that the third party honours. If they do not offer one, effectively-once is not achievable and you are choosing between duplicate and loss on their behalf. Say so explicitly rather than hoping.
1async function handle(msg: Message) {2 // messageId is chosen by the PRODUCER and is stable across redeliveries.3 // A broker-assigned delivery id is not — it can differ per attempt.4 await db.transaction(async (tx) => {5 const claimed = await tx.insertIfAbsent('processed_messages', {6 message_id: msg.messageId,7 handler: 'billing.v1', // scope: two handlers must not share a row8 processed_at: new Date(),9 })10 if (!claimed) return // duplicate: effect already committed once11 12 await tx.insert('ledger_entries', toEntry(msg)) // same transaction13 })14 15 // Outside the transaction there is no atomicity left. If this call16 // succeeds and the process dies before ack, the message is redelivered,17 // the dedup row blocks the ledger write, and the email is NOT resent —18 // which is correct only because the ledger row is the source of truth.19 await ack(msg)20}Operational consequences people discover late
Unacked messages are invisible in most queue-depth metrics. A consumer holding a thousand messages it will never ack — because it is deadlocked, not dead — shows an empty-looking queue and zero throughput, and the broker will not release those messages until the lease expires. In-flight count is the metric that reveals this, and it is rarely on the default dashboard.
Acknowledgement also interacts badly with in-process concurrency. If a consumer receives messages onto an internal work pool and acks on receipt so it can keep pulling, it has silently converted the pipeline to at-most-once — every message buffered in memory is lost on shutdown. This is a frequent accidental regression when someone adds a thread pool for throughput.
Finally: graceful shutdown is a correctness feature, not politeness. On SIGTERM, stop fetching, finish in-flight messages, ack them, then exit. A consumer that does this turns a routine deploy from "a burst of duplicates and a visibility-timeout stall" into a clean handover. This is one of the highest-value hours of engineering available in a messaging system.
available : eligible for delivery to any consumer in-flight : delivered, lease running, not yet acked <-- invisible to depth metrics acked : deleted (queue) or offset advanced (log) nacked : returned early by the consumer, back to available dead-lettered: exceeded max attempts, moved out of the main queue A "healthy, empty" queue with 4,000 in-flight and zero acks per minute is a stalled consumer, not an idle one.
Key points
- Ack after processing = at-least-once (duplicates, no loss). Ack before = at-most-once (loss, no duplicates). There is no third arrangement.
- Duplicates are inevitable even with a perfect consumer, because an ack can be lost in the network and is then indistinguishable from no ack.
- Effectively-once processing is achievable by committing the dedup record and the side effect in one transaction — and only as far as that transaction reaches.
- Auto-ack is at-most-once with no error path. It is a data-loss setting, appropriate only where loss is genuinely free.
- In-flight messages are invisible to queue-depth metrics; a stalled consumer looks like an idle system.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The broker delivers a message and marks it in-flight, starting a lease.
- • The consumer executes the handler, producing whatever side effects the message implies.
- • The consumer sends an acknowledgement. Delivery of that acknowledgement is itself unreliable.
- • On receiving the ack, the broker removes the message from the queue (or, in a log, allows the offset to advance).
- • If no ack arrives before the lease expires, the message becomes available again and the attempt counter increments.
- • After a configured number of attempts the broker routes the message to a dead-letter destination instead of redelivering.
- • The consumer crashes between side effect and ack — the canonical duplicate.
- • The ack is sent and lost in the network; the broker redelivers a message that was correctly processed.
- • The consumer acks first for throughput and then crashes, losing the message with no record.
- • The handler is slow, the lease expires, and the ack arrives referring to a lease that no longer exists — usually an error, sometimes silently ignored.
- • A batch ack covers messages whose individual outcomes differed, marking failures as successes.
- • Duplicate side effects clustered around deploys: the operator sees two of everything in a five-minute window each afternoon, correlating exactly with the rolling restart. Nothing in the logs is an error.
- • Silent data loss under auto-ack: the operator sees consumer throughput matching publish rate perfectly, and a downstream table that is missing a fraction of records after every crash. Broker metrics report 100% delivery success.
- • Stalled consumer masquerading as idle: queue depth 0, throughput 0, in-flight 4,000, no alerts firing. Discovered when someone asks why a report is empty.
- • Redelivery storm from an expired-lease loop: the same messages are delivered repeatedly, attempt counters climb, and throughput collapses while CPU stays high — every worker is repeatedly starting work that another worker is also doing.
- • Batch-ack loss: a batch of 500 is acked after 499 succeeded and 1 threw, and the failure is invisible because the ack was issued at batch granularity.
- • The ack is the minimal coordination between consumer and broker: a one-way claim, unverifiable by the receiver.
- • Effectively-once requires coordination between the ack decision and the side-effect store — achieved by co-locating them in one transaction, which is coordination avoidance, not coordination.
- • True exactly-once delivery would require an atomic commit spanning broker and application store. That is Two-Phase Commit: Buying Atomicity With a Promise, with its blocking failure mode, and it is why the industry chose idempotence instead.
- • With ack-after-processing, no acknowledged-by-the-broker message is lost to a consumer crash; the cost is bounded duplicate processing.
- • With ack-before-processing, no message is processed twice; the cost is unbounded silent loss, proportional to crash frequency.
- • A dedup record committed with the effect keeps the *effect* exactly-once even though delivery remains at-least-once — the guarantee moves from the transport to the handler.
- • Detect: alert on in-flight count with zero ack rate, and on redelivery-attempt distribution shifting upward.
- • Contain: implement graceful shutdown so planned restarts stop generating duplicates; that removes the majority of them in most systems.
- • Recover: after an incident involving killed consumers, expect and permit a duplicate wave rather than trying to suppress it at the broker.
- • Reconcile: verify the dedup table against the effect table — a dedup row with no corresponding effect means the transaction boundary is wrong.
- • Verify: confirm redelivery rate returns to its normal non-zero baseline. Zero is not the target and would suggest ack-before-processing.
- • Redelivery / attempt-count distribution per queue: the p99 tells you whether leases are too short for real processing times.
- • Ack latency (delivery to ack), which must sit comfortably under the visibility timeout.
- • In-flight message count, separately from queue depth.
- • Duplicate-suppression hit rate on the dedup table — a rising rate is the earliest signal that something upstream is retrying more.
- • Correlation of duplicate rate with deploy events, which usually shows graceful shutdown is missing.
- • Always — every consumer has an ack policy whether or not anyone chose it deliberately. Choosing it is free; inheriting it is expensive.
- • Ack-after-processing plus idempotence is the correct default for anything with a side effect that a person would notice twice.
- • Extreme-throughput telemetry where each message is worth less than the dedup lookup that would protect it. Auto-ack is the right call, stated explicitly.
- • Building a dedup store for effects that are already naturally idempotent — an upsert of a full state does not need protecting.
- • Very long-running handlers, where ack-after-processing forces very long leases and makes crash recovery slow. Split the work instead.
- • Naturally idempotent handlers (upsert full state rather than apply a delta), which make the duplicate free and delete the dedup store entirely.
- • A conditional write in the effect store itself — a unique constraint on a business key — which is a dedup table you were going to need anyway.
- • At-most-once with auto-ack where loss is genuinely acceptable, chosen and documented rather than defaulted into.
- • A A Topic Is Not One Log: Ordering Lives Inside a Partition with offset commits, which changes the granularity of the same trade from per message to per partition — see Commit Before or After: There Is No Third Option.
Acknowledgement: the two-line protocol that decides your delivery semantics
What people believe, and what is true
Exactly-once delivery is a feature some brokers offer.
What is offered is exactly-once *processing* within the broker’s own boundary — its reads and its writes. The moment your handler touches an external system, at-least-once is what you have.
If we never crash, we will not get duplicates.
A lost ack produces a duplicate with a perfectly healthy consumer. The failure is in the channel.
Acking early is a performance optimisation.
It is a change of delivery semantics from at-least-once to at-most-once. The throughput gain is real and the guarantee you sold to get it is usually not yours to sell.
A nack means the message failed.
A nack means the consumer *reported* failure. The side effect may already have committed — a nack after a partial success is the same ambiguity, sent deliberately.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Ack after you have done the work: you may process a message twice, and you will never lose one. Ack before, and you may lose one silently. Pick the first and make the handler safe to run twice.
Practical
Ack after processing, always, unless loss is genuinely free. Give every message a producer-chosen id. Commit a dedup row and the side effect in one transaction. Implement graceful shutdown on SIGTERM. Alert on in-flight count and on redelivery-attempt p99, not on queue depth.
Advanced
The acknowledgement is a one-bit message over an unreliable channel, and its delivery cannot be confirmed without another acknowledgement — the Two Generals regress again. So the system has exactly two consistent designs: the sender retries until acknowledged (at-least-once, duplicates possible) or it does not retry (at-most-once, loss possible). Every real messaging system is one of these two wearing different vocabulary, and the "exactly-once" ones are at-least-once transport plus deduplication inside a boundary they control. Knowing where that boundary ends is the whole skill.
Apply it
- 🔧 Write a consumer with auto-ack, kill it mid-handler, and measure exactly how many messages disappeared with no error anywhere.
- 🔧 Add a dedup row inside the effect transaction and prove that a forced redelivery produces no second effect while the message is still delivered twice.
- ⚡ Duplicate emails appear every afternoon at 14:05. Deploys run at 14:00. Diagnose without looking at any code.
- ⚡ A queue shows depth 0, throughput 0 and 4,000 in-flight. Explain what is happening and what will happen in ten minutes.
- 💬 Where do you put the ack, and what does each choice give up?
- 💬 Your consumer never crashes. Can you still get duplicates? Show me how.
- 💬 What would exactly-once actually require, and why does nobody build it that way?