Outbound Webhooks
When you are the provider: delivering to endpoints you do not control, without letting a slow customer take down your service.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
How do I deliver events to my customers' endpoints without their failures becoming mine?
Customers want to be notified when an order ships, instead of polling our API every thirty seconds. We need to POST to a URL they configure.
After the order ships, POST the event to the customer's configured URL inside the same request handler. If it fails, log it and move on.
The customer's endpoint takes thirty seconds to respond. Your ship-order request now takes thirty seconds, holding a database connection and a worker the whole time (External Calls Inside a Transaction).
- The customer's endpoint takes thirty seconds to respond. Your ship-order request now takes thirty seconds, holding a database connection and a worker the whole time (External Calls Inside a Transaction).
- One customer's endpoint is down. Their failures consume your workers, and every other customer's requests queue behind them (Bulkheads).
- Logging the failure and moving on means the customer never receives the event. They discover this weeks later, having built on the assumption that a webhook means it happened.
- The customer configures
http://169.254.169.254/latest/meta-data/as their webhook URL, and your backend obediently fetches cloud instance credentials and POSTs your own metadata to them (SSRF — When the Backend Fetches a URL). - Your retries have no backoff, so when a customer's endpoint returns 500, you hammer it — and when they recover, your accumulated backlog knocks them over again (Retry Storms).
What is actually happening
- Outbound webhooks invert the earlier lessons: now you are the unreliable client, and the customer's endpoint is the server with the timeout you cannot see. Everything you wanted from a provider is what you now owe.
- Delivery must be decoupled from the triggering request. The domain event is committed; a separate delivery pipeline reads it and attempts the POST. Otherwise a customer's latency is your latency (The Transactional Outbox).
- Each destination needs its own failure isolation. A per-customer concurrency limit or queue means a dead endpoint consumes a bounded slice of your capacity rather than all of it.
- Signing is the mirror of verification: you compute an HMAC over the exact bytes you send, with a per-customer secret, and include a timestamp so they can bound replay.
- The customer-supplied URL is an SSRF vector by construction — you are being asked to make an HTTP request to an address someone else chose. Resolution and address filtering are mandatory, not hardening.
- Retry policy is a contract. Customers build around "you retry for 24 hours with backoff"; changing it silently changes their correctness.
Delivery is a pipeline, not a function call
The single decision that determines whether an outbound webhook system works is whether delivery happens inside the request that caused the event. Inside, and every customer endpoint's latency is added to your own, and every customer outage consumes your capacity. Outside, and the triggering request commits and returns while delivery becomes a bounded, retryable, observable pipeline.
The pipeline has a specific shape. The event and the intent to deliver it are written in the same transaction as the domain change, so there is no state where the order shipped and nobody will ever be told. A dispatcher then reads unsent rows and hands them to workers that do the signing, the URL validation, the POST and the outcome recording.
The dispatcher is where isolation lives. Claiming rows per destination, with a per-destination in-flight cap, is what stops one dead endpoint from occupying every worker.
- 1Commit event + outbox row
One transaction with the domain change; the intent to deliver is now durable.
fails by Posting inside the transaction — an external call holding a database connection (External Calls Inside a Transaction).
- 2Claim
A worker takes a batch of unsent rows, bounded per destination.
fails by Two workers claiming the same row; unbounded claiming so one destination starves the rest.
- 3Validate destination
Resolve the host, reject private/loopback/link-local/metadata addresses.
fails by Validating at configuration time only; following redirects (SSRF — When the Backend Fetches a URL).
- 4Sign
HMAC over
timestamp.bodywith that endpoint's current secret.fails by Signing bytes other than the ones sent; a rotation invalidating in-flight retries.
- 5POST with timeouts
Separate connect and read deadlines; no redirect following.
fails by No timeout, so one endpoint holds a worker indefinitely (Timeouts).
- 6Record outcome
Status, latency, attempt number, response snippet — visible to the customer.
fails by Recording nothing, so "did you send it?" is unanswerable.
- 7Retry or dead-letter
Exponential backoff with jitter to a published ceiling, then park it.
fails by No jitter (herd on recovery); no ceiling (infinite spend); no customer-visible dead-letter (Dead-Letter Queues).
Every step after the commit is retryable and none of it is in the customer's request path — which is exactly the property the naive version lacks.
A customer-supplied URL is a request you were told to make
169.254.0.0/16 entry matters because that is where AWS, GCP and Azure all expose instance metadata. Outside a cloud VM the same range is merely link-local; inside one it is a credential endpoint. The rest of the list is environment-independent.Outbound webhooks are one of the few features whose entire purpose is to make HTTP requests to addresses chosen by someone outside your organisation. That is the definition of the SSRF setup, and it is not an edge case to harden later — it is the feature.
Two properties make naive validation insufficient. Redirects mean the address you validated is not necessarily the address you connect to, so redirect following must be off. DNS means the address a hostname resolves to can change between validation and connection, so the check that matters is on the resolved address at connect time, not on the string the customer typed.
The payload design is the other half. Because the destination is whatever the customer most recently configured, a payload containing full records is a payload that can be redirected to an attacker by anyone who compromises a customer account. Sending ids and requiring an authenticated fetch means the blast radius of a hijacked destination is "they learn that an event happened", not "they receive the data".
1import ipaddress, socket2 3BLOCKED = [4 ipaddress.ip_network("10.0.0.0/8"),5 ipaddress.ip_network("172.16.0.0/12"),6 ipaddress.ip_network("192.168.0.0/16"),7 ipaddress.ip_network("127.0.0.0/8"),8 ipaddress.ip_network("169.254.0.0/16"), # link-local: cloud metadata9 ipaddress.ip_network("::1/128"),10 ipaddress.ip_network("fc00::/7"),11]12 13def resolve_public(host: str) -> str:14 """Resolve now, check now, and connect to *this* address."""15 infos = socket.getaddrinfo(host, 443, proto=socket.IPPROTO_TCP)16 for *_, sockaddr in infos:17 ip = ipaddress.ip_address(sockaddr[0])18 if any(ip in net for net in BLOCKED) or ip.is_reserved:19 raise ValueError(f"destination resolves to blocked address {ip}")20 return infos[0][4][0]21 22# Deliver: scheme must be https, redirects off, both timeouts set, and the23# connection is pinned to the address we just validated.24ip = resolve_public(url.host)25resp = session.post(26 url.replace(host=ip), headers={"Host": url.host, **signature_headers},27 data=raw_body, timeout=(3.0, 10.0), allow_redirects=False, verify=True,28)Resolving and then connecting to the validated address is what closes the DNS-rebinding window; allow_redirects=False is what closes the redirect bypass. Checking the URL string at save time closes neither.
What you owe the receiver
Every difficulty in the first four lessons of this module is a difficulty you are now creating for someone else. The useful discipline is to write the provider-side contract as the mirror of what you wished your providers had given you.
Most of these cost little and are almost impossible to add later without breaking customers. A stable event id, a documented signing scheme, a published retry policy and a visible delivery log are compatibility surfaces from the first customer onward.
| What the receiver needs | What you must provide | What happens if you do not |
|---|---|---|
| Authenticate the sender | HMAC over raw bytes, per-endpoint secret, documented construction | Every receiver's endpoint is an open write (Webhook Signature Verification) |
| Bound replay | A signed timestamp header | A captured delivery is a permanent forgery |
| Deduplicate | A stable event id, identical across retries | They cannot deduplicate at all (Webhook Idempotency) |
| Order events | A per-object version or sequence in the payload | They must re-fetch on every event, or be wrong |
| Plan their retries | A published schedule and ceiling | They cannot size their reconciliation window |
| Recover a lost event | A delivery log with manual resend, or a list API | A dropped event is unrecoverable for them |
| Rotate secrets safely | Two active secrets during an overlap window | Rotation means a coordinated deploy and dropped deliveries (The Secret Lifecycle) |
| Protect their own data | Ids rather than full records in the payload | A hijacked destination is a data breach, not a nuisance |
How to build it
Most important first.
- Commit the event, then deliver asynchronously. The triggering transaction must not contain an outbound HTTP call (The Transactional Outbox).
- Sign every delivery: HMAC-SHA256 over
timestamp.bodywith a secret unique to that endpoint, in a documented header. Publish the verification algorithm and a code sample; your customers will get it wrong otherwise. - Send a stable event id in the payload and document that it is stable across retries. You are asking your customers to deduplicate — give them the key to do it with.
- Validate the destination URL at configuration time and again at delivery time: HTTPS only, resolve the hostname, reject private, loopback, link-local and metadata addresses, and do not follow redirects (SSRF — When the Backend Fetches a URL).
- Retry with exponential backoff and jitter, to a published ceiling, then dead-letter and surface the failure in the customer's dashboard (Backoff and Jitter).
- Apply a per-endpoint concurrency cap and a circuit breaker: after N consecutive failures, back off that destination entirely rather than continuing to spend workers on it (Circuit Breakers).
- Set aggressive connect and read timeouts. You are calling an endpoint whose performance you cannot influence (Timeouts).
What can go wrong
- A single slow customer saturating the delivery worker pool, so every other customer's events queue — the classic noisy-neighbour failure (Twenty Workers, All Busy, Five Hundred Waiting).
- The outbox growing faster than delivery drains it during a broad incident, turning a delivery problem into a storage problem (Queue Backlog).
- Redirects followed to an internal address, bypassing URL validation that only checked the originally configured host.
- DNS rebinding: the hostname resolves to a public address at validation time and to a private one at delivery time. Validating the resolved address at connect time is the only defence.
- Retries without jitter synchronising across many events, producing a thundering herd against a customer who has just recovered (Thundering Herd).
- Dead-lettered deliveries with no customer-visible surface, so the customer believes they received everything.
- A secret rotation that invalidates every in-flight retry, because the retry re-signs with the new secret while the customer is still verifying with the old.
- Two events for the same object delivered concurrently to the same endpoint, arriving at the customer out of order — the ordering problem, now created by you (Webhook Retries and Ordering).
- A retry in flight while the customer rotates their signing secret, so a delivery signed with one key is verified against another.
- A destination URL updated while deliveries are queued for the old one — deciding whether in-flight events go to the old or new URL is a real design choice, and sending customer data to a stale URL is a security question.
- Outbox rows claimed by two delivery workers simultaneously, producing duplicate deliveries from your own side (Atomic Operations).
- Customer-supplied URLs are the textbook SSRF setup. Enforce HTTPS, resolve the host, and reject private, loopback, link-local and cloud-metadata ranges at connect time — not just at configuration time (SSRF — When the Backend Fetches a URL).
- Do not follow redirects. A 302 to
http://169.254.169.254/defeats validation done on the original URL. - Never include secrets, tokens or full personal records in a webhook payload. Send ids and let the customer fetch what they are authorized to see — the payload goes to whatever endpoint is configured, and configurations get changed.
- Per-endpoint signing secrets, so one customer's leaked secret cannot forge deliveries to another.
- Do not leak your internal error detail in delivery attempts or in the customer-facing delivery log (Not Leaking Your Internals).
- Rate-limit how often a customer can change a destination URL, and re-verify ownership on change, or a compromised customer account becomes an exfiltration channel.
- "We POST the event, so the customer has it." You have evidence of a 2xx from an endpoint. Delivery is the customer's to confirm, which is why event ids matter.
- "Their endpoint being down is their problem." It is their problem and your capacity, until you isolate it.
- "We validate the URL when they save it." Validation at configuration time is defeated by DNS changes and redirects. Validate at connect time.
- "Retries make delivery reliable." They make it more likely. Reliability comes from the customer being able to see and replay what failed.
- "Signing is optional if we use HTTPS." HTTPS proves to *you* who you called. Signing is what lets the *customer* prove who called them.
Operating it
- Per-endpoint delivery success rate, attempt count and latency. The customer dashboard should show the same data you see; support questions collapse when it does.
- Outbox depth and oldest-unsent age. Age is the better alert: depth can be large and draining, age cannot.
- Circuit-breaker state transitions per endpoint, with the reason. "Why did we stop delivering to them?" must be answerable.
- Count deliveries rejected by URL validation, split by rule. A spike in metadata-address rejections is someone probing.
- Track the distribution of customer response times — it is the input to your timeout choice, and it is the only measurement of it you will ever have.
- Fan-out is the scaling axis: one domain event can become thousands of deliveries when many customers subscribe. Delivery volume is subscriptions times events, not events (Fan-Out: Waiting for the Slowest of Seven).
- At 10x, per-endpoint isolation stops being defensive and becomes load-bearing — the probability that at least one destination is degraded approaches one.
- At 100x, the delivery pipeline is a distinct service with its own scaling, its own storage and its own on-call, because its failure modes have nothing to do with the application that produces the events.
- Payload size matters more than it seems: it multiplies by fan-out on egress, and egress is billed.
- Asynchronous delivery is the only workable design and it means you can no longer tell the triggering caller whether delivery succeeded. That has to become a separate, visible surface.
- Aggressive timeouts protect you and fail deliveries to customers whose endpoints are merely slow. There is no setting that is right for everyone, which is why the retry policy has to be generous enough to compensate.
- Strict SSRF filtering will reject some legitimate configurations — a customer on a VPN-reachable host, a staging endpoint on a private range. The exception process is a security decision, not a support decision.
- Publishing a retry policy makes you predictable to customers and makes it a compatibility surface you cannot casually change.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe obligations — sign, retry with backoff, isolate per destination, expose delivery state — hold for any push-notification product.
- CLOUD-SPECIFICThe SSRF danger is sharper on cloud VMs with an instance metadata service on a link-local address; IMDSv2-style session-token requirements reduce but do not eliminate it, and container platforms expose different internal ranges. The address blocklist is environment-specific and must be reviewed per platform.
- SCALE-SPECIFICWith a handful of subscribers, a simple queue and a shared worker pool are fine. Per-endpoint isolation and circuit breaking become necessary when the number of destinations makes a degraded one continuously likely.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — fan-out delivery to independently-failing consumers, and why a delivery log is the only durable answer to "did they get it".