Exactly-Once Is a Scope, Not a Guarantee

Almost every disagreement about exactly-once resolves once these are separated, because people arguing about it are usually discussing different ones.

Exactly-once delivery — each message arrives at the consumer exactly once. This is impossible in general. The sender cannot know whether its message arrived (A Timeout Tells You Nothing About Whether It Happened), so it must either resend on silence (at-least-once) or not (at-most-once). No exchange of acknowledgements removes this, because the acknowledgement can also be lost — the Two Generals result. Anything claiming exactly-once delivery is either wrong or is describing one of the other two.

Exactly-once processing — the handler body runs exactly once. Also not achievable: a consumer can crash mid-handler after doing half the work, and the redelivery runs the handler again. You can bound *what the reruns do*, but you cannot bound how many times the code executes.

Exactly-once effect — the observable outcome happens once, regardless of how many times the message was delivered or the handler ran. This one is achievable, and it is what every real system means. It is achieved by making duplicate work converge on a single outcome, either through idempotence or through an atomic commit that discards the duplicate.

The construction is always the same, whatever the product name. Commit the record of "I have consumed this input" and the output of processing it in one atomic transaction, in one store. Then a crash either commits both or neither. On redelivery, the consumer sees its committed input position, skips the already-processed message, and no second output exists.

Everything follows from where you can find a single store that holds both. In Kafka’s transactional mode the trick is that consumer offsets are themselves written to a Kafka topic, so "commit the offset" and "produce the output records" are both writes to Kafka and can be one transaction. In a database-backed worker, you store the processed message id in a table in the same transaction as the business write. In Flink, a checkpoint captures operator state and, at a transactional sink, uses a two-phase commit so state and sink output advance together.

Notice what all three have in common: the output must go somewhere that participates in the same transaction as the input position. That is the scope. It is not a setting; it is a structural property of where your data lands. An output to a system that cannot join that transaction is outside the guarantee no matter what mode is enabled.

Kafka’s EOS is real engineering and worth understanding precisely, because it is the claim most often over-read. It combines three things. The idempotent producer gives each producer a producer id and attaches a monotonic sequence number per partition, so the broker discards a retried duplicate within that producer session — this removes duplicates caused by producer retries, per partition. Transactions let a producer atomically write to several partitions and, critically, write consumer offsets into the internal offsets topic as part of the same transaction. Read-committed consumers refuse to surface records from transactions that have not committed.

Put together, these give exactly-once for a consume-transform-produce loop where both the input and the output are Kafka topics in the same cluster. That is a genuinely useful and genuinely hard guarantee, and Kafka Streams builds on it.

What falls outside it is the part that matters for most applications. Any side effect that is not a Kafka write is not in the transaction: a database write to an external store, an HTTP call, an email, a file upload, a metric. A downstream consumer configured read_uncommitted sees aborted records. A consumer that commits offsets outside the transaction loses the atomicity. Zombie fencing depends on transactional.id being stable across restarts — reuse or regenerate it and the fencing does not apply. Producer id state expires (transactional.id.expiration.ms), and after expiry the dedup window is gone. And the whole thing is per-cluster: mirroring to another cluster is a new at-least-once boundary.

So the honest sentence is: Kafka gives exactly-once effect for Kafka-to-Kafka pipelines within one cluster, when configured correctly, and at-least-once for everything else. A team that enables EOS and then deletes its consumers’ idempotence has usually just moved the duplicates to the first external side effect, where they are harder to see.

At-least-once delivery plus idempotent processing. That is the design that works everywhere, composes with everything, and survives operational reality.

It is worth saying why it is preferable to a transactional mode even when one is available. It does not depend on a configuration remaining correct across every service that touches the data. It survives a manual DLQ replay, a re-drive from an admin console, a migration between brokers, and someone adding an HTTP call to a handler. It does not degrade quietly when a downstream team sets read_uncommitted for debugging and forgets. Its correctness is local to the consumer and visible in the consumer’s own code, which is the property that makes guarantees survive time and staff turnover.

Use exactly-once machinery when the pipeline genuinely is store-to-store within one system and the volume makes per-message idempotence expensive — stream aggregations are the canonical fit. Keep idempotence anyway at every boundary the transaction does not cover. And in documentation, never write "exactly-once" without the scope attached, because a downstream team will read the phrase and stop deduplicating.

When a system or a colleague claims exactly-once, three questions settle it quickly, and they work regardless of the technology.

Where does the record of "I processed this" live, and where does the output live? If the answer is not "the same store, the same transaction", the claim is about something else. What happens to a side effect that is not in that store? If the answer is "we retry it", the guarantee is at-least-once at that point. What happens when the dedup or transaction state expires or is reset? Every implementation has a bound — a retention, an expiry, a producer-id lifetime — and the guarantee ends there.

These questions also work on your own design, and asking them early is usually what reveals that the pipeline has three external effects nobody had classified.