Webhook Ordering: Assume None
Retries, parallel dispatch and redrives mean events arrive in whatever order the network permits — order.shipped before order.paid is routine. Consumers that apply event payloads as state, in arrival order, corrupt their data; the contract must say so and give them a defense.
Frame the contract
API design starts with a consumer, a design question and a guarantee — never with a URL.
Why order breaks even when nothing fails
Ordering dies of ordinary causes. order.paid (event 1) times out on its first attempt and is scheduled for retry in 30 minutes; order.shipped (event 2) fires two minutes later and delivers immediately. The consumer sees shipped, then — 28 minutes later — paid. No bug anywhere: retry schedules per event are independent by design, so any retry reorders the stream.
Parallel dispatchers do the same at smaller scale (two workers, two events, no ordering between them), and redrives do it at bulk scale (dead-lettered events from Tuesday replayed into Thursday's live stream). Guaranteeing global order would require delivering events one at a time, each waiting for the previous acknowledgment — one slow response would freeze the pipeline for every subsequent event, converting one consumer hiccup into total delivery stall. Providers rightly refuse; the contract must then refuse *explicitly*, because consumers assume ordering unless told otherwise.
Note what this shares with message queues generally: per-key ordering is purchasable (partition by order id, one in-flight delivery per key) at real throughput and head-of-line-blocking cost — the same trade Kafka-Style Logs: Topics, Partitions, Offsets-style systems make. A few webhook providers offer it per object; most do not, and consumers must handle the general case.
t+0s order.paid attempt 1 → consumer timeout (retry in 30m)
t+2m order.shipped attempt 1 → 200 delivered
consumer state: shipped (paid never seen)
t+30m order.paid attempt 2 → 200 delivered
naive consumer state: paid ← moved BACKWARD
provider timeline: paid → shipped
consumer timeline: shipped → paidThe failure: applying events as state
The naive consumer writes local_order.status = event.data.status on every event. Under the reordering above, their order ends the day as "paid" after being "shipped" — and every downstream decision (release inventory? send review email? count conversion?) runs on regressed state. The corruption is silent: no error, no failed delivery, just a database that quietly disagrees with the provider until an audit notices.
The two working defenses both stop trusting arrival order. Version-gate: the provider stamps each event with a per-object sequence (sequence: 12) or the resource's version; the consumer stores the last-applied number per object and discards anything not newer. Fetch-on-event: treat the event purely as a doorbell — "something changed on ord_4211" — and GET current state, which is correct regardless of which doorbell rang last. Fetch-on-event pairs naturally with thin events from Webhooks: The Inverted Contract and buys freshness at one read per event; version-gating keeps fat payloads usable at the cost of the provider committing to a sequence field forever.
1def on_event(event):2 db.update("orders",3 id=event.data.order_id,4 status=event.data.status)5 6# shipped @ t+2m, then paid @ t+30m7# → final local status: "paid"8# → inventory, emails, analytics all wrong1def on_event(event):2 n = db.execute(3 "UPDATE orders SET status = :s, seq = :q\n"4 " WHERE id = :id AND seq < :q",5 s=event.data.status,6 q=event.sequence, id=event.data.order_id)7 if n == 0:8 log.info("stale event skipped", event.id)9 return # still 200 — stale is not an errorThe conditional update makes ordering irrelevant: whatever arrives, only strictly newer state lands, and the database enforces it atomically even when two events race. The bad version is simpler and correct on every whiteboard, because whiteboards deliver in order.
Choosing a strategy, and what the provider owes
Which defense fits depends on what the events drive. State mirroring wants version-gating or fetch-on-event. Append-only consumption (audit logs, analytics counting events) barely cares about order but needs occurred_at to bucket correctly — and must resist using wall-clock timestamps as a version, since two events in the same millisecond, or provider clock skew, make timestamps a tiebreaker without a tie-breaking rule. If the provider offers no sequence and no version field, fetch-on-event is the only sound option; that absence is itself contract information.
Providers: the cheapest high-value clause you can add is a monotonic per-object sequence number, because it converts every consumer's ordering problem into one integer comparison. Document that delivery is unordered in the same sentence — consumers cannot be blamed for assuming order when the docs never mentioned it. And keep Consumer-Side Idempotency in view: dedup and ordering interact, since a version-gated consumer gets duplicate suppression almost free (a replayed event is never newer), while a fetch-on-event consumer still needs dedup to avoid stampeding the API during redrives.
| Strategy | Requires from provider | Cost | Fails when |
|---|---|---|---|
| Version-gate on sequence/version | Monotonic per-object sequence in every event | One column + conditional update per object | Provider reuses or resets sequences across redeploys |
| Fetch state on event | A GET for the resource; event as doorbell only | One API read per event; read burst after redrives | The read itself is stale behind the next event — harmless if applied via version, loop otherwise |
Timestamp comparison (occurred_at) | Trustworthy clocks and unique timestamps | Cheapest to build | Same-millisecond events, clock skew — silent last-writer-wins |
| Per-key ordered delivery | Provider serializes per object | Provider-side head-of-line blocking per key | Rarely offered; still reordered by consumer-side concurrency |
Key points
- Independent retry schedules reorder the stream by construction; parallel dispatch and redrives reorder it further. No failure is required.
- Global ordered delivery would let one slow consumer response stall every subsequent event — providers refuse, so contracts must state "unordered" explicitly.
- Applying event payloads as state in arrival order silently regresses consumer data; the corruption produces no errors, only wrong answers.
- Version-gating (apply only strictly newer) and fetch-on-event (event as doorbell, GET as truth) are the two sound defenses; wall-clock timestamps are not a version.
- A monotonic per-object sequence number is the cheapest clause a provider can add — it reduces every consumer's ordering problem to one integer comparison.
Follow the failure
How the contract fails or gets misused, hop by hop — and what it costs when it completes.
- 1Provider → docs: never mentions ordering; the demo delivers in order, so consumers assume the contract does too.
- 2Consumer → handler: writes
status = event.statuson arrival, mirroring provider state by arrival order. - 3Network → stream: one timeout delays
order.paidpastorder.shipped; the consumer's order regresses to "paid". - 4Downstream → decisions: inventory release, review emails and conversion metrics all read the regressed status.
- 5Audit → weeks later: consumer and provider disagree on thousands of orders; without sequences, nobody can even sort out which side is right per record.
- Silent state corruption in every consumer that mirrors by arrival order — the worst kind of break: no errors, wrong data.
- Downstream automation acts on regressed state (re-sending "payment received" emails, double-releasing inventory).
- Support burden lands on the provider: "your webhooks are out of order" tickets for behavior that was never promised otherwise.
Design, observe, evolve
A contract decision is incomplete until you know how you would notice it failing and how it changes later.
- • Stamp every event with a monotonic per-object sequence (or the resource version) and document delivery as unordered in the same breath.
- • Teach the two consumer patterns in the docs — version-gated apply and fetch-on-event — with the conditional-update snippet included.
- • Keep `occurred_at` in the envelope for bucketing and audit, and explicitly warn against using it as a version.
- • If you offer per-key ordered delivery, price it honestly in the docs: per-object serialization and head-of-line blocking on slow acks.
- • Consumer-side: count stale-event skips (version-gate rejections); a baseline exists always, spikes correlate with retry storms and redrives.
- • Provider-side: measure inter-event delivery inversions per object in the delivery log — it quantifies how unordered your stream really is, which is the number consumers will ask for.
- • Reconciliation jobs comparing sampled consumer state to provider state catch the corruption that neither side's metrics can see alone.
- • Adding a sequence field to the envelope is additive; consumers adopt version-gating at their own pace while old consumers keep their behavior.
- • Removing or resetting sequence semantics is a breaking change of the quiet kind — version-gated consumers start discarding valid events; treat sequence continuity as a compatibility promise like any field meaning.
- • Per-object sequences require the provider to serialize event numbering per object at emission time — cheap in a transactional outbox, awkward in fan-out-first architectures.
- • Fetch-on-event converts ordering risk into read load: a redrive of 40,000 events becomes 40,000 GETs; rate limits and caching must expect the echo.
- • Version-gated consumers discard information: if events carry deltas rather than states, "skip the stale one" loses the delta — delta streams need true ordering or full-state events.