Idempotency vs Deduplication
Idempotency makes a repeated request produce the same outcome and hands that outcome back. Deduplication detects that a message was already seen and drops it. Related, frequently confused — and each one fails when asked to do the other's job.
Frame the contract
API design starts with a consumer, a design question and a guarantee — never with a URL.
Two problems wearing the same trench coat
Both start from the same fact — at-least-once is the only delivery guarantee retries can buy, so repeats will arrive. But the repeat means different things in the two settings. In request/response, a caller is *waiting for an answer*: the correct handling of a repeat is to hand back the original outcome, because the caller's whole problem is that it never learned it (Idempotency Keys: The Mechanism). In messaging, nobody waits: the correct handling of a repeated event is to recognize it and do nothing, because the effect already happened.
The confusion costs real bugs in both directions. A webhook consumer that builds full response-replay machinery is carrying storage and complexity for answers nobody will ever read. Worse, an API team that implements "dedup" on a payments endpoint — detect the repeat, drop it, return a generic 200 — has built a trap: the retrying client receives a 200 with no payment id for a request the server silently discarded, and now believes a payment exists that it cannot reference.
| Idempotency (request/response) | Deduplication (messages/events) | |
|---|---|---|
| The repeat is | a retry of an intent whose outcome the caller never learned | a redelivery of a message whose effect already happened |
| Someone is waiting | yes — the caller needs the outcome | no — the producer already moved on |
| Correct handling | replay the stored outcome (success or failure) | acknowledge and discard |
| Identity comes from | client-minted key per intent (Idempotency Keys: The Mechanism) | producer-assigned event_id / message id |
| State required | key → full outcome, for the retry window | seen-set of ids, for the redelivery window |
| Canonical home | POST /payments | webhook and queue consumers (Consumer-Side Idempotency) |
Dedup mechanics: the seen-set and its window
Deduplication needs less than idempotency: an id per message assigned by the producer, a store of processed ids on the consumer, and an atomic "claim id, then process" step — the same check-then-act race from Idempotency Keys: The Mechanism applies, just without the response storage. The consumer that marks the id as seen only *after* processing will double-process on a crash between the two; the consumer that marks before processing will *drop* a message on the same crash. Which failure you prefer is a real decision: mark-first loses, process-first duplicates, and only making the mark and the effect one transaction avoids the choice.
The seen-set has a horizon, and the horizon is a contract clause. A queue that can redeliver for up to 7 days needs 7 days of ids; an id that ages out early turns a late redelivery into a fresh event. Producers help by making ids meaningful — a delivery_id distinct from event_id, timestamps, sequence numbers — so consumers can reason about repeats they see (Webhook Delivery: States, Retries, Redrive).
What neither mechanism delivers is exactly-once delivery. Delivery is at-least-once or at-most-once; "exactly-once" in real systems means at-least-once delivery plus an idempotent or deduplicating consumer — exactly-once *processing effect*, engineered on the receiving side. Any contract that promises exactly-once delivery over a network is describing a system that will eventually surprise its users.
Where each belongs in your contract
As an API provider you usually owe both, on different surfaces. Synchronous mutating endpoints owe idempotency, documented per Idempotency Keys: The Mechanism. Your outbound webhooks owe honest at-least-once framing: a unique event_id on every event, an explicit statement that consumers must deduplicate, and no ordering promises you cannot keep (Webhook Ordering: Assume None). The two clauses are duals — one is you handling your callers' repeats, the other is you giving your consumers what they need to handle yours.
The design smell to catch in review is a mechanism on the wrong surface: response-replay storage inside a queue consumer, or drop-and-generic-200 "dedup" on a request/response endpoint. Ask "is anyone waiting for the outcome of this specific attempt?" — the answer sorts every case.
- Sync mutating endpoint → idempotency keys with outcome replay; the caller must learn what happened.
- Outbound events/webhooks → stable
event_idper event, at-least-once documented, consumer dedup expected (Consumer-Side Idempotency). - Your own queue consumers → seen-set dedup, or naturally idempotent handlers (absolute writes, state-machine transitions) that make the seen-set unnecessary.
- Batch ingestion APIs → per-item ids so a re-submitted batch upserts instead of duplicating (Batch APIs and Partial Failure).
Key points
- Idempotency answers a waiting caller with the original outcome; deduplication silently discards a repeat nobody is waiting on.
- Dedup on a request/response endpoint (drop + generic 200) strands the caller with no outcome — the mechanisms are not interchangeable.
- Both need an atomic claim step; dedup additionally chooses its crash failure mode — mark-first drops, process-first duplicates, transactional avoids both.
- The seen-set horizon must cover the redelivery window, and the window is a documented clause, not an implementation detail.
- "Exactly-once delivery" is not purchasable; exactly-once *effect* is built on the consumer side from at-least-once plus idempotency or dedup.
- Providers owe both: key-based idempotency on sync writes, and event ids + at-least-once honesty on outbound events.
Follow the failure
How the contract fails or gets misused, hop by hop — and what it costs when it completes.
- 1Team → endpoint: implements "idempotency" as detect-and-drop; repeats get an empty
200with no resource id. - 2Client → endpoint: a retried checkout receives the empty
200, has no order id, and shows the user a success page with nothing behind it. - 3Same team → webhook consumer: assumes the queue is exactly-once because "we have idempotency now"; no seen-set is built.
- 4Queue → consumer: a redelivery after a consumer crash re-runs a fulfillment side effect; a customer gets two shipments.
- 5Team → architecture review: both incidents trace to one root: nobody asked "is anyone waiting for this outcome?" per surface.
- Callers of drop-style endpoints operate on phantom successes — client state references resources that were never created.
- Consumers without dedup double-execute effects on every redelivery, at exactly the moments (crashes, slowness) redeliveries cluster.
- Teams that believe they bought exactly-once stop writing reconciliation, so the duplicates that do occur go undetected for months.
Design, observe, evolve
A contract decision is incomplete until you know how you would notice it failing and how it changes later.
- • Sort every surface with one question — is a caller waiting for this attempt's outcome? — and apply outcome-replay idempotency or seen-set dedup accordingly.
- • Make the dedup claim and the business effect one transaction where the store allows it; otherwise choose and document the crash failure mode deliberately.
- • Stamp every outbound event with a stable `event_id` and document at-least-once delivery explicitly, so consumers build dedup instead of discovering the need.
- • Size and document the seen-set horizon from the real redelivery window of your queue or webhook retry schedule.
- • Measure duplicate-detection hits on both surfaces: replay rate on endpoints, discard rate on consumers — both are your delivery-ambiguity rate made visible.
- • Reconcile effects against intents (shipments vs orders, emails vs notifications) to catch the duplicates neither mechanism caught.
- • Alert on seen-set evictions younger than the redelivery window — that is the horizon failing before it fails a customer.
- • Handlers can migrate from seen-set dedup to natural idempotency (absolute writes, convergent transitions) without any contract change — the guarantee is what is promised, not the mechanism.
- • Adding `event_id` to an existing webhook stream is additive; consumers adopt dedup at their own pace. Changing id semantics (reusing ids, new format) is the breaking direction.
- • Running both mechanisms means two stores with two retention policies — teams often under-invest in the second one they build.
- • Transactional claim-plus-effect couples the dedup store to the domain store; using the same database simplifies correctness and concentrates load.
- • Naturally idempotent handlers avoid the seen-set but constrain handler design — every effect must be expressible as a convergent write.