Reliabilityidempotencydeduplicationexactly-oncemessagingevents

Idempotency vs Deduplication

Idempotency makes a repeated request produce the same outcome and hands that outcome back. Deduplication detects that a message was already seen and drops it. Related, frequently confused — and each one fails when asked to do the other's job.

Follow the failure

Frame the contract

API design starts with a consumer, a design question and a guarantee — never with a URL.

Design question
Do I need to answer a repeated request with its original outcome, or silently discard a repeated message?
Consumers
Two different audiences: API clients retrying synchronous calls and needing an answer back, and event/webhook/queue consumers processing an at-least-once stream where nobody is waiting for a per-message response.
The promise
The contract names which guarantee each surface provides: request/response endpoints replay outcomes to callers; event consumers detect and discard repeats. Neither surface claims "exactly-once delivery", because the network does not sell it.
RequirementConsumersResource ModelStyleContractValidationAuthorizationErrorsIdempotencyPaginationVersioningObservabilityEvolutionTrade-offs

Two problems wearing the same trench coat

Both start from the same fact — at-least-once is the only delivery guarantee retries can buy, so repeats will arrive. But the repeat means different things in the two settings. In request/response, a caller is *waiting for an answer*: the correct handling of a repeat is to hand back the original outcome, because the caller's whole problem is that it never learned it (Idempotency Keys: The Mechanism). In messaging, nobody waits: the correct handling of a repeated event is to recognize it and do nothing, because the effect already happened.

The confusion costs real bugs in both directions. A webhook consumer that builds full response-replay machinery is carrying storage and complexity for answers nobody will ever read. Worse, an API team that implements "dedup" on a payments endpoint — detect the repeat, drop it, return a generic 200 — has built a trap: the retrying client receives a 200 with no payment id for a request the server silently discarded, and now believes a payment exists that it cannot reference.

The same repeat, two different correct answers
Idempotency (request/response)Deduplication (messages/events)
The repeat isa retry of an intent whose outcome the caller never learneda redelivery of a message whose effect already happened
Someone is waitingyes — the caller needs the outcomeno — the producer already moved on
Correct handlingreplay the stored outcome (success or failure)acknowledge and discard
Identity comes fromclient-minted key per intent (Idempotency Keys: The Mechanism)producer-assigned event_id / message id
State requiredkey → full outcome, for the retry windowseen-set of ids, for the redelivery window
Canonical homePOST /paymentswebhook and queue consumers (Consumer-Side Idempotency)

Dedup mechanics: the seen-set and its window

Deduplication needs less than idempotency: an id per message assigned by the producer, a store of processed ids on the consumer, and an atomic "claim id, then process" step — the same check-then-act race from Idempotency Keys: The Mechanism applies, just without the response storage. The consumer that marks the id as seen only *after* processing will double-process on a crash between the two; the consumer that marks before processing will *drop* a message on the same crash. Which failure you prefer is a real decision: mark-first loses, process-first duplicates, and only making the mark and the effect one transaction avoids the choice.

The seen-set has a horizon, and the horizon is a contract clause. A queue that can redeliver for up to 7 days needs 7 days of ids; an id that ages out early turns a late redelivery into a fresh event. Producers help by making ids meaningful — a delivery_id distinct from event_id, timestamps, sequence numbers — so consumers can reason about repeats they see (Webhook Delivery: States, Retries, Redrive).

What neither mechanism delivers is exactly-once delivery. Delivery is at-least-once or at-most-once; "exactly-once" in real systems means at-least-once delivery plus an idempotent or deduplicating consumer — exactly-once *processing effect*, engineered on the receiving side. Any contract that promises exactly-once delivery over a network is describing a system that will eventually surprise its users.

event_id evt_91delivered ×2claim id atomicallynewseenProducerQueue / webhook retries · at-least-onceConsumerSeen-set: event idsEffect: onceAck & discard
UserLLMAgentToolDataDecisionHumanGuardrail

Where each belongs in your contract

As an API provider you usually owe both, on different surfaces. Synchronous mutating endpoints owe idempotency, documented per Idempotency Keys: The Mechanism. Your outbound webhooks owe honest at-least-once framing: a unique event_id on every event, an explicit statement that consumers must deduplicate, and no ordering promises you cannot keep (Webhook Ordering: Assume None). The two clauses are duals — one is you handling your callers' repeats, the other is you giving your consumers what they need to handle yours.

The design smell to catch in review is a mechanism on the wrong surface: response-replay storage inside a queue consumer, or drop-and-generic-200 "dedup" on a request/response endpoint. Ask "is anyone waiting for the outcome of this specific attempt?" — the answer sorts every case.

  • Sync mutating endpoint → idempotency keys with outcome replay; the caller must learn what happened.
  • Outbound events/webhooks → stable event_id per event, at-least-once documented, consumer dedup expected (Consumer-Side Idempotency).
  • Your own queue consumers → seen-set dedup, or naturally idempotent handlers (absolute writes, state-machine transitions) that make the seen-set unnecessary.
  • Batch ingestion APIs → per-item ids so a re-submitted batch upserts instead of duplicating (Batch APIs and Partial Failure).

Key points

  • Idempotency answers a waiting caller with the original outcome; deduplication silently discards a repeat nobody is waiting on.
  • Dedup on a request/response endpoint (drop + generic 200) strands the caller with no outcome — the mechanisms are not interchangeable.
  • Both need an atomic claim step; dedup additionally chooses its crash failure mode — mark-first drops, process-first duplicates, transactional avoids both.
  • The seen-set horizon must cover the redelivery window, and the window is a documented clause, not an implementation detail.
  • "Exactly-once delivery" is not purchasable; exactly-once *effect* is built on the consumer side from at-least-once plus idempotency or dedup.
  • Providers owe both: key-based idempotency on sync writes, and event ids + at-least-once honesty on outbound events.

Follow the failure

How the contract fails or gets misused, hop by hop — and what it costs when it completes.

  1. 1
    Team → endpoint: implements "idempotency" as detect-and-drop; repeats get an empty 200 with no resource id.
  2. 2
    Client → endpoint: a retried checkout receives the empty 200, has no order id, and shows the user a success page with nothing behind it.
  3. 3
    Same team → webhook consumer: assumes the queue is exactly-once because "we have idempotency now"; no seen-set is built.
  4. 4
    Queue → consumer: a redelivery after a consumer crash re-runs a fulfillment side effect; a customer gets two shipments.
  5. 5
    Team → architecture review: both incidents trace to one root: nobody asked "is anyone waiting for this outcome?" per surface.
What breaks
  • Callers of drop-style endpoints operate on phantom successes — client state references resources that were never created.
  • Consumers without dedup double-execute effects on every redelivery, at exactly the moments (crashes, slowness) redeliveries cluster.
  • Teams that believe they bought exactly-once stop writing reconciliation, so the duplicates that do occur go undetected for months.

Design, observe, evolve

A contract decision is incomplete until you know how you would notice it failing and how it changes later.

Design the contract
  • • Sort every surface with one question — is a caller waiting for this attempt's outcome? — and apply outcome-replay idempotency or seen-set dedup accordingly.
  • • Make the dedup claim and the business effect one transaction where the store allows it; otherwise choose and document the crash failure mode deliberately.
  • • Stamp every outbound event with a stable `event_id` and document at-least-once delivery explicitly, so consumers build dedup instead of discovering the need.
  • • Size and document the seen-set horizon from the real redelivery window of your queue or webhook retry schedule.
Observe in production
  • • Measure duplicate-detection hits on both surfaces: replay rate on endpoints, discard rate on consumers — both are your delivery-ambiguity rate made visible.
  • • Reconcile effects against intents (shipments vs orders, emails vs notifications) to catch the duplicates neither mechanism caught.
  • • Alert on seen-set evictions younger than the redelivery window — that is the horizon failing before it fails a customer.
Evolve without breaking
  • • Handlers can migrate from seen-set dedup to natural idempotency (absolute writes, convergent transitions) without any contract change — the guarantee is what is promised, not the mechanism.
  • • Adding `event_id` to an existing webhook stream is additive; consumers adopt dedup at their own pace. Changing id semantics (reusing ids, new format) is the breaking direction.
What it costs
  • • Running both mechanisms means two stores with two retention policies — teams often under-invest in the second one they build.
  • • Transactional claim-plus-effect couples the dedup store to the domain store; using the same database simplifies correctness and concentrates load.
  • • Naturally idempotent handlers avoid the seen-set but constrain handler design — every effect must be expressible as a convergent write.

Misconceptions

Claim
“Idempotency and deduplication are the same thing at different layers.”
Reality
They share the at-least-once premise and the atomic-claim mechanics, but differ in the observable contract: one returns the original outcome to a waiting caller, the other discards silently. Swapping them produces bugs in both directions.
Claim
“Our message broker has exactly-once mode, so consumers need nothing.”
Reality
Broker "exactly-once" features are transactional guarantees within the broker's own boundaries; the moment your consumer touches an external system (your database, an email API), you are back to needing idempotent or deduplicating effects at that boundary.

Apply it