Real-TimepollingwebhooksSSEpushcompletionRetry-After

How the Client Learns the Job Finished

Polling, webhooks, SSE/WebSocket, push notification — four ways to say "done", each with a different latency, infrastructure cost, client requirement and duplicate story. Polling with Retry-After is the documented baseline every client can use; the others are upgrades for specific consumers.

Follow the failure

Frame the contract

API design starts with a consumer, a design question and a guarantee — never with a URL.

Design question
For each kind of consumer, which channel delivers "your job is done" reliably enough — and what does the contract promise when that channel duplicates, reorders or misses a notification?
Consumers
Browser UIs showing a spinner that must turn into a result; backend integrations that started 10,000 jobs and cannot poll each one; mobile apps that may be backgrounded when the job finishes; and agents that block on a result and need it as soon as it exists.
The promise
The contract names the notification channels it supports, states polling as the always-available fallback with explicit cadence guidance, and guarantees that every channel points at the job resource as the source of truth — so a duplicated, reordered or lost notification can never produce a wrong outcome.
Requirement→Consumers→Resource Model→Style→Contract→Validation→Authorization→Errors→Idempotency→Pagination→Versioning→Observability→Evolution→Trade-offs

Four channels, four different consumers

Polling is the client asking GET /jobs/{id} until terminal is true. It needs nothing but the API, works through every firewall, and is trivially retryable. Its costs are latency (bounded below by the interval) and load (every waiting client is a periodic read). The contract makes it civilized with Retry-After on non-terminal responses, a documented backoff, and a rate limit that treats polling as normal rather than as abuse.

Webhooks are the server calling the consumer's URL when the job reaches a terminal state (Webhooks: The Inverted Contract). Zero polling load, near-instant, and the natural fit for server-side integrations — but the consumer must run a public endpoint, verify signatures, and handle at-least-once delivery (Consumer-Side Idempotency, Webhook Ordering: Assume None). Browsers and mobile apps cannot receive them at all.

SSE or WebSocket streams (GET /jobs/{id}/events) push state changes and progress to a connected client (Server-Sent Events, WebSocket Message Contracts). Lowest latency, the only channel that carries progress naturally, and browser-friendly — at the cost of a held connection per waiting client and a reconnect/resume story. Push notifications (APNs/FCM) reach a backgrounded mobile app, but are best-effort, rate-limited by the platform, and carry only a hint ("your export is ready") that the app must follow by fetching the job.

Completion channels compared
ChannelLatencyInfra the consumer needsWorks forDelivery guaranteeDuplicate / order handlingCarries progress?
Polling + Retry-AfterInterval-bounded (s)NoneEveryoneClient-driven; always reaches truthN/A — reads are idempotentYes, per poll
WebhookNear-instantPublic HTTPS endpoint, signature verificationServersAt-least-once with retriesConsumer dedups by event id; fetch job on receiptRarely (only terminal events)
SSE / WebSocketInstant while connectedLong-lived connection, reconnect logicBrowsers, agents, CLIsOnly while connected; resume via Last-Event-IDSequence ids; refetch job on reconnectYes, natively
Push notificationSeconds to minutesPlatform tokens, app handlingMobile appsBest-effort; may be dropped or collapsedHint only; app fetches the jobNo

Polling is the baseline, not the fallback of shame

Every other channel can fail silently: the webhook endpoint is down for an hour, the SSE connection dropped during a deploy, the push was collapsed by the OS. Polling cannot fail silently — a client that polls eventually reads the truth from the job resource. That makes polling the contract's floor: always available, always documented, and the thing every other channel degrades to. A consumer that receives no webhook within its timeout polls; a browser whose stream dropped polls until it reconnects.

The contract makes polling cheap by shaping it. Retry-After on every non-terminal response tells the client when to come back — and lets the server stretch the interval as the job's expected duration grows (5s for the first minute, 30s after). Conditional requests (ETag on the job, 304 when unchanged — Conditional Requests: ETags, 304 and 412) make wasted polls nearly free. Long polling — holding the GET up to N seconds until a change or the deadline — cuts latency without a new protocol, at the cost of held connections; if the contract supports it, ?wait=25 is documented with its maximum.

The failure to avoid is documenting nothing and then rate-limiting polls as abuse. A client with no Retry-After guidance polls every second because the spinner has to move; a 429 in response teaches it nothing except that the API is hostile (The Rate-Limit Contract).

A poll that tells the client exactly when to come back — and costs nothing when nothing changed
Request
GET /jobs/job_5k2
If-None-Match: "v7"
Response
HTTP/1.1 304 Not Modified
ETag: "v7"
Retry-After: 10

# (job unchanged — no body; client waits 10 s)

# Later, when it completes:
HTTP/1.1 200 OK
ETag: "v9"
{ "id": "job_5k2", "status": "succeeded", "terminal": true,
  "result": { "url": "https://…/exp_9.ndjson", "url_expires_at": "2026-09-01T10:00:00Z" } }

Every channel points at the job

The rule that keeps all four channels safe: a notification is a hint to go read the job, never the result itself. A webhook body may include the status for convenience, but the consumer's logic is "on job.completed, GET /jobs/{id}, act on what it says". Then a duplicated webhook re-reads the same terminal state and acts idempotently; a reordered pair (running after succeeded) is harmless because the fetch returns the truth; a lost webhook is covered by the polling floor. The same rule makes SSE reconnects trivial (refetch, then resume from Last-Event-ID) and push notifications safe to collapse.

Choose per consumer, and say so in the docs: servers subscribe to job.completed webhooks and poll after a timeout; browsers open the events stream and poll on disconnect; mobile receives a push and fetches; agents and CLIs use the stream when available, otherwise poll with the documented backoff. Offering all four is a cost — the contract can offer polling plus one push channel and still be complete, as long as polling is the floor.

Completion often triggers the next step in a workflow: fetch the result, then start another job. Consumers chaining jobs need the completion event to carry enough correlation (job.kind, the consumer's own reference echoed back from creation) to route it without a lookup — a metadata object accepted on create and returned on every notification is the cheap version of that.

The notification is the result — and the only copy
1POST https://consumer.example/hooks
2{ "job_id": "job_5k2", "status": "succeeded", "rows": 10000, "download": "https://…" }
3 
4# Consumer:
5on webhook → import(download)
6 
7# Delivered twice → imported twice.
8# Delivered after a "running" event that arrived late → state confusion.
9# Endpoint down for an hour → job finished, nobody ever imports it.
The notification is a pointer; the job is the truth; polling is the floor
1POST https://consumer.example/hooks
2X-Event-Id: evt_881   X-Signature: …
3{ "type": "job.completed", "job": { "id": "job_5k2" }, "metadata": { "reference": "nightly-2026-08-25" } }
4 
5# Consumer:
6on webhook  → if seen(evt_881): ack; else job = GET /jobs/job_5k2; if job.terminal: handle(job) once
7on timeout  → poll GET /jobs/{id} with Retry-After until terminal
8 
9# Duplicate → same job read, handled once.
10# Reordered → the job says what is true now.
11# Lost → polling reaches the same state.

Making the job resource the single source of truth turns every delivery failure mode into a no-op. The channels differ only in how fast the hint arrives — never in what the consumer does with it.

Key points

  • Polling with Retry-After is the always-available floor every other channel degrades to; document it as normal, not as abuse.
  • Webhooks fit servers, SSE/WebSocket fit browsers and agents, push fits backgrounded mobile — choose per consumer and say which.
  • A notification is a hint to read the job, never the result itself; that rule makes duplicates, reordering and loss harmless.
  • Conditional polls (ETag/304) and long polling make the baseline cheap without a new protocol.
  • Echo consumer metadata on every notification so chained workflows route without a lookup.

Follow the failure

How the contract fails or gets misused, hop by hop — and what it costs when it completes.

  1. 1
    Team → API: ships GET /jobs/{id} with no Retry-After and a webhook whose body contains the full result.
  2. 2
    Client → API: polls every second to keep a spinner honest; the API answers with 429 and the client shows an error for a job that is fine.
  3. 3
    Webhook → consumer: delivers job.completed twice during a retry storm; the consumer imports the result twice because the body was the result.
  4. 4
    Consumer → endpoint: takes its webhook receiver down for maintenance; twelve jobs complete; nothing polls; the results expire unread.
  5. 5
    Team → mobile: sends a push containing the download URL; the OS collapses three pushes into one; two exports are never fetched.
What breaks
  • Uncontrolled polling becomes the job store's dominant load and gets rate-limited into false failures.
  • Result-bearing notifications make every duplicate a double side effect and every loss a lost result.
  • Consumers without a documented fallback have no recovery path when their push channel is down.
  • Chained workflows stall or misroute when completion events carry no correlation to the consumer's own references.

Design, observe, evolve

A contract decision is incomplete until you know how you would notice it failing and how it changes later.

Design the contract
  • • Make the job resource the only source of truth; every channel delivers a pointer plus optional convenience fields.
  • • Return `Retry-After` on every non-terminal read and document polling cadence, backoff, and the rate limit that accommodates it.
  • • Support `ETag`/`If-None-Match` on job reads; consider long polling with a documented maximum wait.
  • • Offer at least one push channel matched to the primary consumer (webhook for servers, SSE for browsers) with event ids and signatures.
  • • Accept `metadata` on job creation and echo it on every notification.
Observe in production
  • • Poll interval per caller versus the `Retry-After` you sent — callers ignoring it are the ones to contact before rate-limiting.
  • • Ratio of `304` to `200` on job reads shows whether conditional polling is used and how wasteful the floor is.
  • • Webhook delivery attempts and dead-letter counts per consumer identify who is silently missing completions.
  • • Time from terminal state to first consumer read of the job — the real end-to-end completion latency per channel.
Evolve without breaking
  • • Adding a channel (an events stream beside webhooks) is additive; removing one needs consumer telemetry and a migration window.
  • • Adding event types (`job.progress`, `job.cancelled`) is safe only if consumers were told to ignore unknown types ([[enum-evolution]]).
  • • Changing `Retry-After` policy is a behavior change that well-behaved clients absorb automatically — which is the point of putting it in the response rather than in the docs.
What it costs
  • • Supporting multiple channels multiplies documentation, testing and the code paths that must all point at the job consistently.
  • • Pointer-only notifications cost the consumer an extra fetch per event; for high-volume small jobs that read load is real.
  • • Long polling trades poll count for held connections, which pushes the capacity problem to the connection layer.
  • • Push notifications are cheap to send and unreliable to deliver; offering them invites consumers to treat them as reliable.

Misconceptions

Claim
“Webhooks replace polling.”
Reality
Webhooks reduce polling. Delivery is at-least-once and only while the consumer's endpoint is up; the consumer still needs the job resource and a polling path for the hour its endpoint was down.
Claim
“Putting the result in the notification saves a round trip.”
Reality
It also makes every duplicate a duplicate side effect and every lost notification a lost result. The round trip is what makes the channel safe; it is the cheapest request in the system.
Claim
“Polling is a sign of a badly designed API.”
Reality
Unshaped polling is. Polling with Retry-After, conditional requests and a documented backoff is the most reliable completion channel there is, and the floor every other channel degrades to.

Apply it

📄 Case Study: Notifications API📄 Case Study: File Upload API
← The Async Job PatternSlow Clients and Backpressure →