Webhookswebhookseventsat-least-oncecallbacksenvelope

Webhooks: The Inverted Contract

A webhook flips the roles: the provider becomes the client, calling an endpoint the consumer operates. Delivery is asynchronous and at-least-once, so the event envelope — event id, delivery id, type, timestamp — is what makes the stream usable, not the payload.

Follow the failure

Frame the contract

API design starts with a consumer, a design question and a guarantee — never with a URL.

Design question
When the provider calls the consumer instead of the other way around, what must the event contract state for the consumer to build something reliable on it?
Consumers
The team on the receiving end: an integrator who wants to know when a payment settles without polling `GET /payments/{id}` every five seconds, an internal service reacting to signups, a partner syncing order state into their warehouse system.
The promise
A well-designed webhook contract states its delivery semantics honestly — at-least-once, unordered, retried on a published schedule — and gives every event the identity fields (event id, delivery id, versioned type, timestamp) a consumer needs to dedupe, reorder and audit.
Requirement→Consumers→Resource Model→Style→Contract→Validation→Authorization→Errors→Idempotency→Pagination→Versioning→Observability→Evolution→Trade-offs

The roles invert; the obligations do not

In every other lesson the consumer calls you. With webhooks you call the consumer: they register a URL, and your infrastructure POSTs events to it as things happen. That inversion changes who owns which failure. Their endpoint being down is now *your* delivery problem; your retry storm is now *their* load problem. The contract has to allocate these obligations explicitly, because the defaults each side assumes are incompatible — providers assume "we sent it, done", consumers assume "we receive each event once, in order".

Neither assumption survives contact with the network. You POST the event, the consumer processes it, and the 200 is lost to a connection reset — from your side that is indistinguishable from "never arrived", so you retry, and the consumer sees the event twice. The only delivery promise an HTTP-based webhook system can honestly make is at-least-once, and the whole design of the envelope follows from admitting that early. See Webhook Delivery: States, Retries, Redrive for the retry machinery and Consumer-Side Idempotency for the consumer's half of the bargain.

The alternative to webhooks is the consumer polling you — simpler to reason about, wasteful at scale (a consumer polling every 5s for an event that fires twice a day makes ~17,000 empty calls per useful one), and slower to react. Webhooks trade that waste for operational coupling to thousands of servers you do not control. How the Client Learns the Job Finished compares the notification options; this module is what you sign up for when you pick this one.

  • `event_id` — stable identity of the business fact, identical across redeliveries; the consumer's dedup key.
  • `delivery_id` — identity of this delivery attempt, different on every retry; the support-ticket correlation key.
  • `type` — a versioned, documented event name (invoice.paid, not update); consumers route on it, so it is contract surface.
  • `occurred_at` — when the fact happened, not when this attempt was sent; the only timestamp with business meaning.
  • `data` — the payload, thin or fat (next section); shaped by the same compatibility rules as any response body.

Thin events or fat events — decide, then say so

A fat event carries the full resource, so the consumer can act without calling back — attractive until you notice you are now serializing possibly-stale state into a queue and delivering it minutes later after retries. A thin event carries ids and a type and makes the consumer fetch current state via GET — always fresh, one extra round trip, and your API must be able to absorb the read burst that follows a busy event stream.

The failure mode worth designing against is the half-decision: a fat payload with no event identity, documented as "we send you the object". Consumers then treat the payload as truth, apply it blindly, and inherit every staleness and ordering hazard at once. Whichever body style you pick, the envelope fields are non-negotiable — and the payload schema evolves under the same additive rules as responses (see Backward Compatibility: The Real Rules), because consumers parse it with the same brittle code.

A bare object, no identity, no semantics stated
1POST https://consumer.example/hook
2 
3{
4  "order": 4211,
5  "status": "shipped",
6  "customer_email": "a@b.co",
7  ... 40 more fields of possibly stale state
8}
9 
10# No event id → duplicates are invisible
11# No type → consumers parse the body to guess what happened
12# No occurred_at → a delayed retry looks like fresh news
An envelope with identity; payload is a versioned, minimal fact
1POST https://consumer.example/hook
2 
3{
4  "event_id":   "evt_8f2c1a",
5  "type":       "order.shipped",
6  "occurred_at":"2026-08-25T09:14:03Z",
7  "data": {
8    "order_id": "ord_4211",
9    "shipment_id": "shp_77"
10  }
11}
12 
13# Consumer dedupes on event_id, routes on type,
14# fetches GET /orders/ord_4211 when it needs full state

The good version costs the consumer one extra GET when they need full state — and buys them dedup, routing, auditability and freshness. The bad version is easier to demo and impossible to run reliably: every hazard in this module lands on it at once.

Design for the consumer's week two, not their demo

A webhook integration works in an afternoon and fails in week two, when the first retry burst, endpoint outage or replayed event arrives. The contract features that separate a usable webhook product from a demo are all operational: an event catalog documenting every type and its payload schema, a way to send test events against a sandbox endpoint, a delivery log the consumer can inspect (GET /webhook-endpoints/{id}/deliveries), and manual redelivery for events their bug dropped.

Internally, do not fire webhooks inline from request handlers. The moment a consumer endpoint hangs for 30 seconds, your API latency inherits it. Persist the event, let a dispatcher deliver from a queue with its own retry state — the transactional-outbox shape from Message Queues territory. The contract benefit: you can honestly document a retry schedule, because delivery has state that survives your own deploys and crashes.

same transaction as the writePOST, signed, timeout-boundedevery attempt recordedAPI handlerEvent outboxDispatcherDelivery queue + retry stateConsumer endpointDelivery log
UserLLMAgentToolDataDecisionHumanGuardrail

Key points

  • Webhooks invert the client/server roles: the consumer's uptime becomes your delivery problem, and your retries become their load problem — allocate both in the contract.
  • At-least-once is the only honest delivery promise over HTTP; exactly-once delivery is not achievable, only exactly-once *processing* on the consumer side.
  • Every event needs an envelope: event id (stable across retries), delivery id (unique per attempt), versioned type, and occurred_at.
  • Thin events trade one extra GET for freshness; fat events trade staleness risk for fewer calls — the unacceptable option is a fat payload with no identity.
  • The product is operational: event catalog, test events, delivery logs and manual redelivery are contract features, not tooling extras.
  • Deliver from an outbox and queue, never inline from request handlers, or consumer latency becomes your API latency.

Progressive depth

Overview

A webhook is your contract running against someone else's server: you promise to deliver events, they promise to accept them, and the network promises nothing.

Practical

Every delivery carries an event_id, a delivery_id, a type, a timestamp and a signature; consumers acknowledge with 2xx quickly and process asynchronously (The Webhook Security Contract, Consumer-Side Idempotency).

Advanced

Delivery is at-least-once with retries on a backoff schedule, which makes duplicates and reordering normal; consumers dedup on event_id and either use sequence numbers or fetch current state on receipt rather than trusting event order (Webhook Delivery: States, Retries, Redrive, Webhook Ordering: Assume None).

Internals

Behind the endpoint sits an outbox: the event is written in the same transaction as the state change, a dispatcher reads it and attempts delivery, and the attempt log is what the dashboard and the replay button read. Skipping the outbox is how events get lost between commit and send (Event-Driven Architecture, Message Queues).

Follow the failure

How the contract fails or gets misused, hop by hop — and what it costs when it completes.

  1. 1
    Provider → contract: ships webhooks documented as "we POST the object to your URL" — no event id, no delivery semantics stated.
  2. 2
    Consumer → endpoint: integrates against the happy path they observed: one POST per change, in order, exactly once.
  3. 3
    Network → delivery: a lost response triggers a retry; the consumer ships the same order twice and books the revenue twice.
  4. 4
    Consumer → provider: support asks why "the webhook fired twice"; the provider explains at-least-once for the first time, in a ticket.
  5. 5
    Ecosystem → workarounds: every consumer bolts on their own dedup and reordering, each subtly different, and the provider can no longer change any observable behavior.
What breaks
  • Duplicate side effects on the consumer side — double shipments, double emails, double charges — traced back to the provider's brand.
  • Consumer endpoint outages silently drop business events when there is no retry state or redelivery, corrupting downstream data for weeks.
  • Provider API latency and availability couple to thousands of third-party servers when events are fired inline.

Design, observe, evolve

A contract decision is incomplete until you know how you would notice it failing and how it changes later.

Design the contract
  • • State delivery semantics in the first paragraph of the webhook docs: at-least-once, unordered, retried on a published schedule.
  • • Put identity in every event — event_id, delivery_id, type, occurred_at — and version payload schemas like any response contract.
  • • Deliver asynchronously from a persisted outbox so retries, delivery logs and redelivery are possible at all.
  • • Ship the operational surface with v1: event catalog, sandbox test events, per-endpoint delivery log, manual redrive.
Observe in production
  • • Delivery success rate and end-to-end latency per consumer endpoint; a single consumer's failure burning your retry capacity shows up here first.
  • • Redelivery and duplicate-report rates per consumer reveal who has not built dedup — before their incident becomes your escalation.
  • • Queue depth and event age in the outbox: rising age means consumers are learning about business facts minutes late.
Evolve without breaking
  • • New event types are additive: consumers must ignore unknown `type` values, and the contract must say so from day one.
  • • Payload evolution follows response rules — add optional fields freely, never repurpose or remove without a versioned type (`order.shipped.v2`) and a migration window.
  • • Delivery semantics can tighten (faster retries, longer horizons) without breaking anyone; they can never loosen quietly, because consumer dedup windows are sized to them.
What it costs
  • • Push beats polling on latency and waste, but couples the provider operationally to consumer infrastructure it cannot see or fix.
  • • The envelope and outbox machinery is real engineering before the first event flows — polling needs none of it, which is why polling remains right for low-volume, latency-tolerant consumers.
  • • Thin events keep payloads honest but concentrate a read burst on the API after every event spike; capacity planning must include the echo.

Misconceptions

Claim
“Our delivery system is reliable, so consumers will get each event exactly once.”
Reality
Exactly-once *delivery* over HTTP is impossible: a lost response forces you to choose between retrying (duplicate) and not retrying (possible gap). Reliable systems choose duplicates and say so; consumers achieve exactly-once *processing* via Consumer-Side Idempotency.
Claim
“Sending the full object saves consumers an API call, so fat events are strictly better.”
Reality
A fat payload is a snapshot that ages in your retry queue. Consumers who apply it blindly overwrite newer state with older (Webhook Ordering: Assume None). Fat events are fine when the payload is an immutable fact; mutable state is safer fetched fresh.
Claim
“Webhooks replace the API — consumers can build purely event-driven integrations.”
Reality
Consumers always need reconciliation: a GET-based way to list or re-fetch state after they lose events to their own bugs and outages. A webhook product without list endpoints strands every consumer who ever had a bad deploy.

Apply it

📄 Case Study: Notifications API
Webhook Delivery: States, Retries, Redrive →