Webhook Idempotency
Providers retry, so duplicate delivery is the normal case — deduplicate on the provider event id, atomically.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
The provider sent the same event three times. How do I make sure the customer is charged, emailed and shipped once?
When a payment succeeds we credit the account and send a receipt. The customer must not receive three receipts because the provider retried.
Before processing, check whether we have already seen this event id. If not, process it and record the id. A SELECT then an INSERT.
Two deliveries of the same event arrive concurrently — which is exactly what happens when the first attempt is slow and the provider retries. Both SELECTs find nothing, both process, both insert. The check-then-act is a race with no lock (Backend Races).
- Two deliveries of the same event arrive concurrently — which is exactly what happens when the first attempt is slow and the provider retries. Both
SELECTs find nothing, both process, both insert. The check-then-act is a race with no lock (Backend Races). - The id is recorded after processing succeeds, so a crash between the side effect and the insert leaves the effect applied and unrecorded. The next delivery does it again.
- The id is recorded before processing, so a crash during processing leaves the event marked handled and the work never done — silently, because there is no error anywhere.
- Deduplication keyed on something that is not stable across retries — a receipt timestamp, a hash of the payload including a
delivery_attemptfield — so retries look like new events. - The dedupe table is unbounded and becomes the largest table in the database, and the uniqueness index on it becomes the slowest insert in the hot path (Idempotency Storage).
What is actually happening
- Providers assign each event a stable id that is identical across all delivery attempts, and often a separate delivery id that differs per attempt. Deduplicating on the delivery id deduplicates nothing.
- The database's unique constraint is the only cheap thing in your stack that is genuinely atomic across concurrent connections.
INSERT ... ON CONFLICT DO NOTHINGeither creates the row or does not, with no window in between (Database Constraints). - The insert-first pattern turns "have I seen this?" from a question into an outcome: you attempt the insert, and the number of rows affected tells you whether you are the first delivery or a duplicate.
- Recording receipt and applying the effect are two different facts. Keeping them in one transaction is what makes the pair atomic; splitting them across a transaction boundary is what creates the crash windows.
- Where the effect is external — an email, a charge, a fulfilment API call — it cannot join your transaction. The event row then needs a state (
received,processing,done) so a resumed attempt knows where it stopped (The Dual Write Problem).
Insert first — the check and the claim must be one operation
The instinct is to ask "have I seen this?" and then act. That is a check-then-act, and check-then-act under concurrency is the oldest race there is: between your read and your write, another request does the same thing. Two deliveries arriving milliseconds apart both read "no" and both proceed.
The fix is not a lock around the check. It is to stop asking the question and instead make a claim the database can adjudicate. INSERT ... ON CONFLICT DO NOTHING is a single statement; the unique index decides a winner; the loser learns it lost from the row count. There is no window because there is no gap between reading and writing.
That gives you a clean three-way outcome from one round trip: inserted means you are the first delivery and should process; conflicted means a duplicate and you should return 200 without acting; an error means the database is unhappy and you should return 500 so the provider retries.
const seen = await db.query(
'SELECT 1 FROM webhook_events WHERE event_id = $1', [id])
if (seen.rowCount > 0) return res.sendStatus(200)
await applyEvent(payload)
await db.query('INSERT INTO webhook_events (event_id) VALUES ($1)', [id])
res.sendStatus(200)const claimed = await db.query(
`INSERT INTO webhook_events (provider, event_id, type, payload, state)
VALUES ($1, $2, $3, $4, 'received')
ON CONFLICT (provider, event_id) DO NOTHING
RETURNING id`,
[provider, id, type, payload])
if (claimed.rowCount === 0) {
metrics.increment('webhook.duplicate', { provider, type })
return res.sendStatus(200) // already claimed by another delivery
}
res.sendStatus(200) // acknowledge, then work
await enqueue('process-webhook', { rowId: claimed.rows[0].id })The left version has a window between the SELECT and the INSERT during which a concurrent delivery sees the same empty result. The right version has no window: the unique index on (provider, event_id) picks exactly one winner, and the row count reports the outcome. It also keys on the provider, so two integrations cannot collide on an id.
The effect and the record must commit together — or have a state
Deduplication is only half the guarantee. The other half is that the record of having handled the event and the handling itself are atomic with respect to each other. If they are not, there is a crash window, and a crash window in a retried system is a bug that will occur.
When the effect is a write to the same database, this is easy and should be taken: one transaction containing both the event insert and the domain change. A duplicate that reaches the insert fails the constraint, aborts, and applies nothing.
When the effect is external — charging a card, sending an email — it cannot join the transaction, and no amount of ordering makes the pair atomic. What you get instead is a state machine that makes the ambiguity explicit and recoverable: the row records that you were about to act, so a resumed attempt can ask the external system what happened rather than guessing.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Crash after external call, before marking done | Row stuck in processing; retry may charge twice | Two systems, no shared transaction | Send our own idempotency key to the provider so their side absorbs the repeat (Idempotency Keys) |
| Crash after marking done, before commit | Effect applied, event looks unhandled | Effect outside the transaction that records it | Recovery must be able to ask the external system, not assume |
| Dedupe row committed, domain write in a second transaction | Event marked handled, nothing happened | Two transactions where one was needed | One transaction when both writes are local (One Transaction or Two) |
Duplicate arrives while first is processing | Two workers on the same event | State checked but not claimed atomically | UPDATE ... WHERE state = 'received' and branch on rows affected (Atomic Operations) |
| Retention job deletes a row inside the retry window | Very old redelivery processed again | Retention shorter than the provider's retry horizon | Retention = documented max retry window + margin |
Idempotent at every hop, not just the first
Deduplicating at the edge protects against the provider's retries. It does nothing about the retries that happen inside your own system: the job runner that re-runs a failed task, the queue that redelivers after a visibility timeout expires, the consumer that crashes mid-batch. Each is another at-least-once boundary, and each needs its own answer.
The rule that generalises: every hop where a message can be redelivered needs an idempotent receiver. The event id carries cleanly through your own pipeline, so the downstream job can key on the same value the edge did — which is a good reason to pass the event id rather than a freshly generated job id.
The last hop is the external one. Your charge call to the payment provider can also be duplicated by your own retry, and there your protection is the key *you* send *them* — the same mechanism, pointed outward.
| Hop | Who retries | What deduplicates |
|---|---|---|
| Provider to your endpoint | The provider, on non-2xx or timeout | Unique constraint on (provider, event_id) |
| Endpoint to your queue | Your enqueue retry | Queue-level dedupe, or the job checking event state |
| Queue to worker | Visibility timeout expiry, worker crash | Job idempotency on the same event id (Job Idempotency) |
| Worker to your database | Job retry | The event row's state, updated atomically |
| Worker to payment provider | Your HTTP retry | An idempotency key you generate and send (Idempotency Keys) |
| Worker to email provider | Your HTTP retry | Provider-side dedupe key, or accept the rare double send |
How to build it
Most important first.
- Deduplicate on the provider's event id, and confirm in their documentation that it is stable across retries. Store it with a unique constraint.
- Insert first, do not check first:
INSERT INTO webhook_events (event_id, ...) VALUES (...) ON CONFLICT (event_id) DO NOTHING. Zero rows affected means a duplicate; return 200 and stop. - Put the insert and the domain change in the same transaction where both are in your database. Then a duplicate cannot apply the effect, because the insert that would have permitted it failed (Where the Transaction Boundary Goes).
- When the effect is external, commit the event row as
processing, do the call, then mark itdone. On a duplicate that findsprocessing, prefer returning 200 and letting the job's own retry finish it over racing it. - Make the downstream operations idempotent independently — pass your own idempotency key to the payment or email provider, so even a genuine double-execution is absorbed one layer down (Idempotency Keys).
- Give the table a retention policy: an index on
received_atand a job that deletes rows older than the provider's maximum retry window plus a margin.
What can go wrong
- Deduplicating in application memory — a
Setof seen ids — which works on one instance and fails the moment there are two (Stateless Services). - Deduplicating in a cache with a TTL shorter than the provider's retry window, so a redelivery on day two is treated as new.
- Unique constraint present but the code catches the violation and falls through to processing anyway, because the exception handler was written for a different case.
- Deduplication working perfectly while the *effect* is applied by a downstream consumer that also retries, so the duplicate reappears one hop later (Job Idempotency).
- The retention job deleting rows still inside the provider's retry window, reopening the duplicate window for old events.
- Two different providers whose event ids collide because the table keys on
event_idalone rather than(provider, event_id).
- Two concurrent deliveries of the same event — the central race this lesson exists to close. Only an atomic insert closes it; a
SELECTfollowed by anINSERTdoes not (Duplicate Detection). - A duplicate arriving while the first is in the
processingstate, so both see a row that is neither absent nor complete. - The retention delete racing a very late redelivery of the row it is deleting.
- Two events for the same object arriving at once, each idempotent on its own id and still interleaving their writes to the object (Optimistic Concurrency).
- The event id comes from the payload, which is attacker-controlled unless the signature verified first. Verify, then deduplicate — in that order (Webhook Signature Verification).
- An attacker who can predict event ids and submit them before the provider does can suppress a real event by pre-consuming its id. Signature verification is what makes that impossible, which is why the ordering matters.
- Do not use a user-supplied field as the dedupe key. The provider's id is the only value both parties agree on.
- "We return 200 quickly, so retries will not happen." Retries also happen when the provider's side of the connection fails, when a proxy times out, and when they replay after an incident.
- "The event id is in the payload, so checking it is enough." Checking it is enough only if the check and the write are one atomic operation.
- "Idempotency is the queue's job." The queue can guarantee delivery; only your handler can guarantee the effect happens once (At-Least-Once Delivery).
- "A duplicate is an error." It is the documented behaviour of every major provider. An endpoint that logs duplicates as errors trains everyone to ignore its errors.
Operating it
- Count duplicates as a first-class metric, not an error:
webhook_events_total{result="new"|"duplicate"}. A rising duplicate ratio means your handler is getting slower than the provider's timeout. - Alert on rows stuck in
processingbeyond a threshold — those are the crash-window cases, and they are invisible in error logs because nothing threw. - Log the event id on every path, so a customer report of a double receipt can be traced to two deliveries or one delivery processed twice — different bugs, different fixes.
- Track unique-violation counts on the events table directly; the database counts them whether or not your code reports them.
- The dedupe table takes one insert per delivery including duplicates, so it is the highest-write table in the webhook path. Its index is on the critical path of every delivery.
- At 100x, retention is the dominant concern: without expiry the unique index outgrows memory, and an index that no longer fits changes insert cost qualitatively (Should I Add an Index?).
- Partitioning by day or by month makes deletion a partition drop instead of a mass delete — the difference between a bounded operation and a long-running one holding locks.
- Insert-first means writing a row for every duplicate, which costs storage and write throughput. It is the price of not having a check-then-act race.
- A single transaction spanning receipt and effect gives you atomicity and forces both into the same database, which constrains where the effect can live.
- A state machine on the event row handles external effects correctly and adds a stuck-state problem you must monitor and recover from.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALEvery provider that retries needs this, which is every provider.
- DATABASE-SPECIFICPostgres offers
INSERT ... ON CONFLICT DO NOTHINGand returns the affected-row count; MySQL hasINSERT IGNOREandON DUPLICATE KEY UPDATE, whose affected-row semantics differ (anON DUPLICATE KEY UPDATEthat changes nothing reports 0, one that changes a row reports 2). Code that branches on rows-affected is not portable between them without care. - SCALE-SPECIFICBelow a few events per second, a plain table with a unique index and a nightly delete is entirely adequate. Partitioning and separate storage become worth their complexity when the index stops fitting in memory.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — at-least-once delivery as a consequence of the two-generals problem, and why exactly-once processing is achievable while exactly-once delivery is not.