Inbound Webhooks
A third party calls your API on its schedule, with its retry policy, and treats your endpoint as infrastructure it depends on.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What changes when an external service is the client and your backend is the server it calls?
Stripe should tell us when a payment succeeds so we can mark the order paid, without us polling every few seconds for events that mostly do not happen.
Add POST /webhooks/stripe, parse the JSON, look up the order, mark it paid, return 200. It is just another endpoint.
Anyone on the internet can POST that URL. Without signature verification, marking an order paid is a public operation (Webhook Signature Verification).
- Anyone on the internet can POST that URL. Without signature verification, marking an order paid is a public operation (Webhook Signature Verification).
- The provider retries on any non-2xx and on any timeout, so a slow handler that eventually succeeds produces two, five or ten deliveries of the same event (Webhook Idempotency).
- The handler does real work inline — writes the order, emails the customer, calls the fulfilment API. The provider's delivery timeout is a small number of seconds; you exceed it, it retries, and now the email goes twice.
payment.succeededarrives beforepayment.createdbecause they were dispatched by different workers. Your handler assumes the order exists and throws.- The provider had an outage and now replays four hours of backlog in ninety seconds. Your endpoint is the only one in the system with no rate limiting, and the pool empties (Connection Pools).
What is actually happening
- A webhook is the provider's outbound HTTP client calling your inbound HTTP server. Every property of a flaky client applies — it retries, it times out, it reorders, it duplicates — and none of it is under your control.
- The provider persists the event and the delivery attempt separately. The event is the fact; the delivery is a best-effort attempt to tell you about it. Redelivery of the same fact is normal operation, not an error condition.
- Your HTTP status code is the only channel you have back to the provider. A 2xx means "stop retrying"; anything else, or no answer inside their timeout, means "try again later".
- Because the status code drives the provider's retry loop, what you do before returning it is a design decision, not an implementation detail. Returning 200 after persisting the raw event is a different contract from returning 200 after fully processing it.
- The payload is a snapshot of the provider's state at dispatch time. By the time you read it, the truth may have moved on — which is why fetching the object by id from their API is sometimes the correct reading of a webhook.
Verify, record, acknowledge — then work
The single structural decision in webhook handling is where the 200 goes. Put it after the full business transaction and your handler duration is the sum of everything downstream: the database, the email provider, the fulfilment API. Any of them being slow turns into a provider retry, and the retry re-runs all of it.
Put the 200 immediately after durably recording the raw event, and the handler becomes bounded work you control. The provider stops retrying, and everything that can be slow moves into a job with your own retry semantics, your own backoff, and your own dead-letter queue.
The cost is honest and worth stating: you have taken ownership. Once you answer 200, the provider considers the event delivered. If your job never runs, no one will send it again.
- 1Read raw body
Buffers the exact bytes, before any JSON parsing.
fails by A body-parsing middleware consumed the stream first, leaving only a re-serialized object.
- 2Verify signature
HMAC over the raw bytes plus timestamp, compared in constant time.
fails by Verifying the re-serialized JSON; comparing with
===; ignoring the timestamp so old captures replay. - 3Insert raw event
One row: provider event id (unique), type, payload, received_at.
fails by Unique-violation on a duplicate — which is the success path, not an error.
- 4Return 200
Tells the provider to stop retrying.
fails by Returning it before the insert commits, losing the event on a crash.
- 5Enqueue processing
Hands the event id to a worker.
fails by Enqueueing inside the same transaction as the insert, or before it commits (The Transactional Outbox).
- 6Process
Applies the event to domain state, idempotently.
fails by Not idempotent, so the job's own retry double-applies (Job Idempotency).
The handler owns only the first four steps. Everything that can be unboundedly slow lives after the acknowledgement.
The status code is an instruction, not a report
Most endpoints use status codes to describe what happened. A webhook endpoint uses them to control a retry loop running on someone else's infrastructure. That reframing changes which code is correct in cases that otherwise look ambiguous.
The question to ask for every branch is not "what went wrong" but "do I want this delivery again?" A transient database failure: yes, 500. A payload for a customer that does not exist in our system: probably not — retrying will not create them, so record it and return 200 with an internal alert. A failed signature: never, 400, and it should page someone if it is sustained.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Signature does not verify | Provider retries a forged or misconfigured delivery for hours | Returned 500 instead of 400 — a server-error code invites retries | Return 400. Alert on the rate; it is either key rotation or probing. |
| Unknown event type | Retry loop on an event you will never handle | Threw an exception, framework mapped it to 500 | Return 200. Record it as unhandled; the provider adds event types without asking. |
| Database briefly unreachable | Event silently dropped | Caught the error and returned 200 to "be safe" | Return 500. This is exactly the case the provider's retry exists for. |
| Handler takes longer than the delivery timeout | Duplicate side effects, growing with load | Full processing inline; provider gave up and re-sent | Acknowledge after recording; move processing to a job. |
| Provider replays 4 hours of backlog | Pool exhausted, user-facing requests time out | The webhook path shares the connection pool with everything else | Bound webhook concurrency separately (Bulkheads, Resource Limits). |
The payload is a claim about the past
A webhook body is a snapshot taken when the provider dispatched it. Between then and your handler reading it, the subscription may have been cancelled, the charge refunded, the customer deleted. On a redelivery hours later, the gap can be large.
This gives you a real choice per event type. For events where the payload is the whole fact — "this charge succeeded, for this amount, with this id" — trusting the body is correct and cheap. For events that assert a *current* state — "subscription status is now past_due" — fetching the object from the provider's API gives you the truth at processing time instead of at dispatch time, at the cost of an external call and a rate limit you now share with everything else.
Is this event a fact about a moment, or an assertion about current state?
when Immutable events: a charge succeeded, an invoice was finalised, a message was delivered.
cost Nothing extra — but you must be sure the field genuinely cannot change.
when Mutable state: subscription status, inventory level, order state machine position.
cost An external call per event, inside your processing job, subject to the provider's rate limit (Rate Limiting).
when High event volume where per-event fetching would exceed the provider's rate limit.
cost A scheduled reconciliation job you must build and monitor (Scheduled Jobs).
when Your own state is authoritative and the event only tells you to look.
cost One fetch per event, and you lose the ability to replay from stored payloads.
How to build it
Most important first.
- Split the endpoint into verify, record, acknowledge and do the real work in a background job. The handler becomes: check the signature, insert the raw event with its provider id, return 200 (Background Jobs).
- Verify the signature against the raw bytes before parsing anything, and reject with 4xx if it fails (Webhook Signature Verification).
- Make processing idempotent on the provider's event id. Duplicate delivery is the normal case, not the exception (Webhook Idempotency).
- Never assume ordering. Handle each event as an independent statement about the world, or reconcile against the provider's current state (Webhook Retries and Ordering).
- Return non-2xx only when you want a retry. A malformed payload will never become well-formed — return 400 and alert, do not make the provider retry forever.
- Bound the endpoint: request size limit, a concurrency cap, and a separate pool or worker allocation so a replay burst cannot starve user traffic (Bulkheads).
What can go wrong
- Slow handler, provider timeout, retry — and the work from attempt one is still running when attempt two arrives. Two concurrent executions of the same event.
- Returning 500 on a permanently bad payload, so the provider retries it for days and it eventually lands in their dead-letter view with no one watching.
- Returning 200 before persisting anything, so a crash between acknowledgement and write loses the event permanently — the provider will never send it again.
- The queue you defer to is down, so the handler cannot even record the event. Now you must decide between losing it and returning 500 to trigger a redelivery.
- A replay of historical events triggers side effects that were correct in the past and are wrong now — refund emails for orders closed months ago.
- A retry arriving while the first attempt is still executing — two handlers, same event, same row (Backend Races).
- Two different events for the same object arriving concurrently and interleaving their updates, so the last writer wins regardless of which state is newer.
- The webhook arriving before the API call that caused it has returned to your own code, so the object the event refers to does not exist yet locally.
- The endpoint is unauthenticated in the usual sense: there is no user session. The signature *is* the authentication, and it must be checked before any other work (Webhook Signature Verification).
- Do not trust amounts, statuses or customer ids in the payload as authorization for anything. A verified signature proves the provider sent it, not that the event refers to an object your caller may touch.
- Do not follow URLs contained in a webhook payload. A
receipt_urlfetched by your backend is an SSRF primitive (SSRF — When the Backend Fetches a URL). - Log the event id and type, never the whole payload — provider payloads carry card metadata, emails and addresses (Secrets in Logs).
- "It is just a POST endpoint." It is an endpoint whose client retries aggressively, cannot be told to stop, and is operated by someone else.
- "HTTPS means it is authenticated." TLS proves you are talking to a server with a certificate for your domain. It says nothing about who is talking to you.
- "We get each event once." Every serious provider documents at-least-once delivery. Assuming otherwise is assuming the failure mode away (At-Least-Once Delivery).
- "A 200 means we handled it." It means you told the provider to stop retrying. What it implies about your own state is entirely up to your design.
Operating it
- Count deliveries received, split by event type and by outcome: verified/rejected, new/duplicate, processed/failed. The duplicate rate is a health signal, not noise.
- Track handler duration against the provider's documented delivery timeout. Requests approaching it are future retries.
- Alert on signature verification failures. A sustained non-zero rate means either a key rotation you missed or someone probing the endpoint.
- Watch the age between the provider's
createdtimestamp in the payload and your receipt time — that gap is the provider's backlog, and it grows before it recovers.
- Webhook volume is not proportional to your traffic; it is proportional to the provider's event generation, which spikes when they recover from an outage.
- At 10x, the verify-record-acknowledge split stops being tidiness and starts being the only thing keeping the endpoint under the delivery timeout.
- At 100x, the raw event table becomes a retention problem. It is append-only and grows with provider volume, so it needs a partitioning or expiry policy before it needs anything else.
- Acknowledging before processing means a 200 no longer implies success. You have accepted responsibility for the event and now owe it your own retry and dead-letter machinery (Dead-Letter Queues).
- Storing raw payloads makes replay and debugging possible and makes you the custodian of someone else's personal data, with the retention obligations that implies.
- Fetching the object from the provider's API instead of trusting the payload avoids stale data and adds an external call — with its own timeout, rate limit and failure mode (Calling Something You Do Not Control).
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe inversion — they are the client, you are the server — holds for every webhook provider.
- PROTOCOL-SPECIFICDelivery timeouts, retry schedules, signature schemes and header names are per-provider and documented per-provider. Stripe, GitHub, Shopify and Twilio each differ in all four; read the specific provider's docs rather than generalising from one integration.
- SCALE-SPECIFICBelow a few events per minute, inline processing genuinely works and the queue is overhead. The split becomes necessary when a burst can exceed the delivery timeout — which is a property of the provider's replay behaviour, not of your average rate.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — delivery semantics between independently-operated services, and why a message is never simultaneously exactly-once, ordered and available.