Webhook Retries and Ordering
The provider decides when to retry and does not promise order, so your handler must be correct for events that arrive late, twice, or backwards.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What must my handler assume about when webhooks arrive and in what order?
A subscription moves from trialing to active to past_due. Our database should reflect the customer's real status, whatever order the notifications turn up in.
Handle each event as a transition from the previous state: on subscription.updated, set the status to whatever the payload says. Events arrive in the order they happened.
past_due was dispatched after active, but active was retried twice and landed later. The final write is active and the customer keeps a service they have not paid for.
past_duewas dispatched afteractive, butactivewas retried twice and landed later. The final write isactiveand the customer keeps a service they have not paid for.- A
customer.deletedarrives before thecustomer.createdyour handler needs, so the handler throws, returns 500, and the provider retries it — forever, because the ordering will never improve on its own. - The provider's backoff means a failed delivery returns hours later. Your handler applies an hours-old snapshot over current state.
- A provider incident ends and the backlog arrives at once. Events from a four-hour window land in seconds, in an order determined by their retry schedule rather than by causality.
- The handler is written as a state machine that rejects invalid transitions, so a duplicate
activeafteractiveis treated as an error and retried rather than absorbed.
What is actually happening
- Providers dispatch deliveries from a pool of workers. Two events created a millisecond apart can be picked up by different workers, hit different network paths, and complete out of order even when nothing failed.
- Retry schedules are exponential and per-delivery. A delivery that fails once is minutes behind; one that fails five times is hours behind. Retry turns a small ordering perturbation into a large one.
- Some providers offer ordering guarantees within a scope — per object, per subscription — usually by refusing to dispatch the next event until the previous is acknowledged. That guarantee costs head-of-line blocking: one stuck event stops the stream for that scope (Head-of-Line Blocking).
- Every event payload carries the provider's own notion of time or sequence: a
createdtimestamp, an objectversion, or a monotonically increasing sequence per object. That field, not arrival order, is what tells you which of two events is newer. - Retries and ordering interact: without idempotency, a retry duplicates an effect; without ordering-tolerance, a retry applies a stale state. Both are consequences of the same at-least-once, no-order delivery model (At-Least-Once Delivery).
Arrival order is not causal order
WHERE clause on the DO UPDATE branch of an upsert, which is what makes the conditional form above a single statement. MySQL's ON DUPLICATE KEY UPDATE has no WHERE; the equivalent is a conditional expression per column, such as status = IF(VALUES(provider_version) > provider_version, VALUES(status), status), which is more verbose and easier to get subtly wrong.The intuition that events arrive in the order they happened comes from watching a healthy system, where they usually do. Retries break it decisively. A delivery that fails its first attempt is scheduled minutes later; one that fails repeatedly returns hours later. Meanwhile the events created after it were delivered on the first try.
So the handler must not treat arrival as evidence of recency. The provider's own version or timestamp is the only ordering signal that survives retries, because it was assigned when the event was created rather than when it was delivered.
This makes the correct handler shape a conditional write rather than an assignment: apply this event only if it is newer than what we have already applied. That single comparison makes the handler tolerant of duplicates, reordering and arbitrary delay at once — which is why it is worth more than a state machine.
1-- The guard is `provider_version < $2`, not the absence of a row.2-- A stale event updates zero rows and is discarded; a duplicate of the3-- version we already applied also updates zero rows. Both are correct.4 5UPDATE subscriptions6 SET status = $3,7 current_period_end = $4,8 provider_version = $2,9 updated_at = now()10 WHERE provider_id = $111 AND provider_version < $2;12 13-- rows affected = 1 -> applied14-- rows affected = 0 -> stale or duplicate; log it, return 200, do nothing15 16-- If the row may not exist yet (event arrived before creation):17INSERT INTO subscriptions (provider_id, status, provider_version, updated_at)18VALUES ($1, $3, $2, now())19ON CONFLICT (provider_id) DO UPDATE20 SET status = EXCLUDED.status,21 provider_version = EXCLUDED.provider_version,22 updated_at = now()23 WHERE subscriptions.provider_version < EXCLUDED.provider_version;The WHERE on the DO UPDATE branch is the part people omit. Without it, an upsert of a stale event overwrites newer state — the exact bug the version column was added to prevent.
Late, missing and unprocessable are three different problems
Grouping every delivery anomaly under "retries handle it" hides that the three cases need different machinery. A late event needs staleness filtering. A missing event — one the provider gave up on — needs reconciliation, because no retry is coming. An unprocessable event needs a decision, because retrying it changes nothing and returning 500 forever is not a decision.
The last of these is where most handlers are wrong. A payload referencing a customer you have never seen is not a transient error, and treating it as one burns the provider's retry budget until they give up and disable the endpoint — a real behaviour of several providers, and one that then breaks every other event type too.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Event arrives hours late after retries | Newer state overwritten by older payload | Applied on arrival order rather than provider version | Conditional update guarded on provider_version < |
| Event arrives before its predecessor | Handler throws on a missing parent row | Assumed causal order | Upsert, or record and defer to your own retry — not the provider's |
| Provider exhausted its retries | Your state permanently diverges, silently | No path exists to learn about the event now | Scheduled reconciliation against the provider's API |
| Payload is malformed or references a foreign tenant | Retried for days, then endpoint auto-disabled | Returned 5xx for a permanent condition | Return 200, store as unprocessable, alert |
| Backlog replay after a provider incident | Pool exhaustion, user requests time out | Webhook path unbounded and sharing resources | Dedicated concurrency limit for the webhook route (Bulkheads) |
| Your endpoint 500s under load | Load increases | Provider retries add to the arriving rate | Shed load with 429 where the provider honours it; fix the acknowledgement path |
Reconciliation is the only thing that closes the loop
Every mechanism in this lesson is best-effort. Signature verification, deduplication and staleness filtering all make individual deliveries safe; none of them can tell you about an event that never arrived. And events do fail to arrive: the provider exhausts retries during an outage of yours, a deploy drops in-flight requests, someone deletes a row by hand.
A reconciliation job — periodically listing the provider's objects modified since your last run and comparing them to yours — is what converts "we might be wrong forever" into "we are wrong for at most one interval". It is unglamorous and it is the difference between a payments integration that can be trusted and one that cannot.
It also gives you a metric worth alerting on. The count of discrepancies per run should be near zero; when it is not, you have found a class of event your handler is mishandling, before finance does.
What is the recovery path when a delivery is permanently lost?
when The provider offers a list endpoint filtered by modification time.
cost A job to build and monitor; bounded staleness equal to the interval; API quota consumed on every run.
when The provider exposes an events list or a resend control.
cost Manual or scripted, and only usable within their retention window.
when Reads are rare and freshness matters more than latency.
cost An external call in the request path, with its timeout and failure mode (Calling Something You Do Not Control).
when The data is advisory — a display-only status, an analytics counter.
cost Nothing operationally, provided nobody downstream treats it as authoritative. That assumption decays.
How to build it
Most important first.
- Write handlers that are commutative or last-write-wins by provider version, not handlers that assume a previous state. Store the version or timestamp you applied and ignore events older than it.
- For events asserting current state, re-fetch the object from the provider at processing time. That converts an ordering problem into a single read of the truth (Inbound Webhooks).
- Never return 500 because a referenced object does not exist locally yet. Record the event and either retry it on your own schedule or reconcile later — a provider retry is the wrong tool because you control neither its timing nor its ceiling.
- Absorb no-op transitions silently.
activeapplied to an already-activerecord is a duplicate, not an invalid transition. - Size the webhook path for burst, not for average, because a provider recovery replays a window all at once (Resource Limits).
- Build a reconciliation job that periodically compares your view of the provider's objects against theirs. It is the only thing that repairs an event you dropped (Scheduled Jobs).
What can go wrong
- Last-write-wins on arrival time, which is precisely the wrong clock. Two events reordered by retry produce a final state that never existed.
- Comparing the provider's timestamp against your own clock rather than against the stored one, so clock skew decides business state (Eventual Consistency in Practice).
- Ordering enforced by making the handler wait for a predecessor, turning a webhook endpoint into a blocking operation and guaranteeing the provider times out.
- The reconciliation job itself racing live webhooks and overwriting a newer state with the value it read at the start of its run.
- A dead-letter path that no one monitors, so events the provider gave up on disappear with no alert.
- Retry storms in the other direction: your handler returns 500 under load, the provider retries harder, and the retries are what keep you overloaded (Retry Storms).
- Two events for the same object processed concurrently by different workers — both read, both write, and the one that commits last wins regardless of which is newer (Optimistic Concurrency).
- A reconciliation job and a live webhook writing the same row at the same time.
- A retry of an event whose original delivery is still in flight, so the stale-check and the state write interleave.
- A late redelivery arriving after the object has been deleted locally, resurrecting it if the handler upserts blindly.
- A replayed old delivery is a valid, correctly signed event. Rejecting stale events by timestamp is a security control as well as a correctness one (Replay Attacks).
- An attacker who can delay or drop deliveries — by exhausting your endpoint — can hold your system at a stale state. Bounding webhook resources protects the correctness of the state, not just availability.
- Do not expose event ordering to callers as a guarantee you do not have; downstream consumers will build on it.
- "Events arrive in order" — no major provider promises global ordering, and the ones that promise scoped ordering do so by blocking.
- "Ordering is fixed by processing sequentially." Processing in the order you *received* them serialises the wrong sequence.
- "Retries mean nothing is lost." Retries have a ceiling. After it, the provider stops and the event is gone unless someone reads the dead-letter view.
- "A 500 is a safe default when something looks wrong." It is an instruction to retry. For a permanently unprocessable event it is an instruction to retry forever.
Operating it
- Record
received_at - payload.created_atper delivery. This is the provider's delivery lag, and a rising p99 is the earliest signal of a backlog heading your way. - Count events discarded as stale, split by type. A type with a high stale rate is one whose ordering assumptions are being violated regularly.
- Instrument attempt numbers if the provider sends them. A rising mean attempt count means your endpoint is failing or timing out more than you think.
- Alert on reconciliation-job discrepancies. A non-zero count means events are being lost, and the count is the only place that shows it.
- Provider backlogs arrive as a step function. Capacity planned on the average delivery rate is planned for the wrong number (Load Test Shapes: The Shape Is the Hypothesis).
- Stale-event filtering gets cheaper at scale, not more expensive: it is a comparison against a column you already load.
- Reconciliation cost grows with the number of objects, not the number of events, so it eventually needs to be incremental — scan by last-modified window rather than in full.
- Version-based last-write-wins is simple and discards information: you never learn about the intermediate states you skipped. If your product needs the transitions, you need an event log rather than a status column (Event Sourcing).
- Re-fetching for current state removes the ordering problem and adds an external call per event, plus a rate limit shared with the rest of your integration.
- Providers that guarantee per-object ordering give you a simpler handler and a head-of-line stall whenever one event cannot be processed.
- Reconciliation catches everything and runs on a schedule, so it turns a correctness gap into a bounded staleness window rather than closing it.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALAssume at-least-once, unordered delivery unless the provider documents otherwise in writing.
- PROTOCOL-SPECIFICRetry schedules and ceilings are per-provider: some retry for a few hours, some for several days, some stop after a fixed attempt count and disable the endpoint entirely. Your retention and reconciliation windows must be set from the specific provider's documented ceiling.
- SIMPLIFIEDThis lesson treats the provider as a black box that emits events with a version field. Real providers differ in whether they give you a version, a sequence, only a timestamp, or nothing — and with nothing, re-fetching is the only correct strategy.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — logical clocks, causal ordering, and why a version assigned at the source is the only ordering that survives an unreliable channel.