The question this answers
When a downstream stage cannot keep up, how does that fact travel back to whatever is producing the work — and what does the producer do about it?
An ingestion service accepts webhook deliveries over HTTP, enqueues each for enrichment, and the enrichment stage calls a third-party API capped at 200 requests/second. Deliveries arrive at up to 1 200/second during a partner's batch run.
The queue between the HTTP handler and the enrichment workers, and the third-party API's rate budget — which is shared with every other consumer of that credential, including ones in other services.
The number of enrichment calls in flight never exceeds the downstream budget, and the system's memory and response latency stay bounded regardless of arrival rate.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The chain, and where it can be broken
The failure is mechanical and always the same shape. Arrival rate exceeds service rate. The difference has to go somewhere, and the only place it can go is a buffer. The buffer grows, so memory grows and queueing latency grows with it. Eventually one of three things ends it: memory runs out, a timeout upstream fires and the work becomes useless before it is done, or a human notices. None of those is a design.
Backpressure is a signal travelling the *opposite* direction to the data. The data flows producer → queue → consumer → downstream; the signal has to flow downstream → consumer → queue → producer, and finally out of your system to the caller. It only works if every hop in that chain is capable of transmitting it. A single hop that always accepts — an unbounded queue, a fire-and-forget dispatch, a handler that enqueues and returns 202 unconditionally — breaks the chain and everything upstream of the break is blind.
That is why Bounded vs Unbounded Queues is the same lesson from the other side: bounding the queue is what makes the hop capable of refusing. TCP solves this problem with the receive window, which is worth studying because it is the same idea with the signal built into the protocol rather than bolted on (Flow Control: The Receive Window and What Happens When the Receiver Is Slow in Operating Systems). Architecture treats it as a system-shaped concern in Backpressure.
Watching the chain fail, then hold
The timeline below runs the same 1 200/second arrival through two configurations. In the unbounded lane the handler never blocks and never refuses: every lane looks healthy, throughput is flat at the downstream limit, and the only thing moving is depth. In the bounded lane the queue fills in about four seconds, the handler starts refusing, and the partner's client backs off — which is the entire mechanism working as intended.
The important detail is *where the pain appears*. Unbounded, the pain appears in your process, minutes later, as an OOM. Bounded, the pain appears at the partner's client, immediately, as a 429 with a Retry-After — an outcome the partner's client library already knows how to handle. Moving the pain to the party who can actually respond to it is what backpressure is for. It does not make the work go faster; it makes the overload survivable and legible (The Rate-Limit Contract and Quotas vs Rate Limits in API Design cover what to put in that response).
Note the second stall in the bounded lane: workers blocked on the permit set, not on the queue. That is the correct place for the constraint to live. The 200-in-flight limit is a fact about the third party, so it belongs at the boundary with the third party — a semaphore around the call (Semaphores: Counting Permits as a Resource Limit, Bounding Concurrency) — and everything upstream inherits it automatically. Encoding it as a pool size instead scatters the constraint across a number that also controls unrelated things.
Four ways to say no, and picking one
Blocking, rejecting, shedding and buffering-to-disk are the whole menu, and each is right somewhere. Blocking suits an internal pipeline where the producer is a loop you own and slowing it is harmless. Rejecting suits an entry point with a client that retries. Shedding — dropping low-value work to protect high-value work — suits a system with a priority distinction it can actually make. Spilling to durable storage suits bursts you have committed to not losing.
The one that is never right is "accept it and hope", which is what an unbounded buffer implements. And the one that is *subtly* never right is blocking on an inbound request path without a timeout: it converts a rejection you could have returned in 2 ms into a socket held for 30 seconds, and 1 200 of those per second exhausts the connection pool, the file-descriptor limit, or the thread pool long before memory becomes the issue (Connection Pool Saturation: Waiting in Front of an Idle Database in Performance, File Descriptors in Operating Systems).
The compare below is the smallest real version of the fix. The bad version enqueues unconditionally and returns 202, which is a lie: the system has not accepted responsibility for the work, it has only accepted the bytes. The good version offers with a short timeout and converts a full queue into a 429, which is true.
1async function handleWebhook(req: Request): Promise<Response> {2 // queue is unbounded; push always succeeds3 queue.push(req.body)4 return new Response(null, { status: 202 })5}6// Accepts 1200/s into a stage that drains 200/s.7// Every caller is told "accepted". Nothing upstream can ever learn otherwise.1async function handleWebhook(req: Request): Promise<Response> {2 // bounded queue; offer() waits briefly, then reports failure3 const accepted = await queue.offer(req.body, { timeoutMs: 50 })4 if (!accepted) {5 metrics.shed.inc()6 return new Response(null, {7 status: 429,8 headers: { 'Retry-After': '2' },9 })10 }11 return new Response(null, { status: 202 })12}13// The 50 ms grace absorbs jitter; beyond that the caller is told the truth14// while it still has the context to retry.The difference is not the queue type — it is that one version has a return path for "no". A stage that cannot refuse cannot transmit backpressure, so everything upstream of it is operating on stale information about whether the system is coping.
Key points
- Backpressure is a signal that travels against the direction of data flow; one hop that always accepts breaks the chain for everything upstream of it.
- The chain is mechanical: arrival > service → buffer grows → memory and latency grow → OOM, or upstream timeout, or a human.
- Bounding a queue is what makes a hop capable of refusing; that is why bounded queues and backpressure are the same design decision.
- Put the constraint where the constraint actually is — a semaphore at the third-party boundary, not a pool size that also controls other things.
- Blocking on an inbound request path without a timeout trades a memory failure for a connection and file-descriptor failure, which arrives sooner.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • The downstream stage exposes a finite resource: a permit count, a queue capacity, a rate budget, or simply a blocking call that does not return.
- • When that resource is exhausted the consumer stops draining, so queue depth rises — depth is the first place the signal becomes observable.
- • A bounded queue converts sustained depth into a refusal at the enqueue site: block with a timeout, or return failure immediately.
- • The producer translates that refusal into something its own caller understands — a 429 with Retry-After, a rejected message that stays unacknowledged on the broker, or its own blocking.
- • The caller reduces its rate, and the system settles at the downstream service rate instead of the arrival rate. Nothing went faster; the excess is now visibly refused rather than invisibly buffered.
- • Arrivals 1 200/s, drain 200/s, unbounded queue: depth grows 1 000 every second. There is no schedule in which this recovers, which is the point — the failure needs no unlucky interleaving.
- • Bounded, blocking without timeout: handler thread 1 blocks on a full queue; handlers 2..N do the same; at N = pool size the server stops accepting connections and the failure surfaces as connection refusal, not as queue pressure.
- • Bounded, offer-with-timeout: handler waits 50 ms, a worker completes and frees a slot, the offer succeeds — a burst absorbed. Or the 50 ms expires and a 429 is returned in bounded time regardless of load.
- • Permit-set stall: worker A holds a permit and is waiting on a slow third-party call; workers B..D wait on the permit; queue depth rises; the queue fills; the handler refuses. The signal propagated four hops correctly.
- • The broken hop: the enrichment worker dispatches the third-party call as fire-and-forget and returns immediately. The queue drains at full speed, everything looks healthy, and the third party sees 1 200/s until it bans the credential — a stage that never waits also never signals (Orphaned Tasks).
- • Backpressure guarantees that memory and queueing latency stay bounded under arbitrary arrival rates. That is a survivability guarantee.
- • It does NOT guarantee the work gets done. Refused work is refused; whether it is retried is the caller's business and must be part of the contract.
- • It does NOT guarantee fairness. A well-behaved client that backs off gets less throughput than an aggressive one that hammers through 429s, unless you add per-client accounting.
- • It does NOT reduce latency for accepted work — the queue is still full, so accepted items still wait. Bounding depth bounds that wait; it does not remove it.
- • It does NOT compose automatically across a service boundary. Your 429 is only backpressure if the caller's client library honours it; many retry immediately and make things worse (Retry Storms: The Load You Generated Yourself in Performance).
- • Producers contend on the "not full" condition of the queue, and every one of them is usually holding a connection while it waits.
- • Workers contend on the permit set at the downstream boundary; that contention is intended, but it means a slow downstream converts directly into idle workers.
- • Under refusal, contention moves outside the process to the client's retry behaviour, which is where a thundering herd forms if every rejected caller retries at the same instant (Thundering Herd).
- • Instrumentation is itself a contention point at high refusal rates — a shared counter incremented 1 000 times a second on eight threads is a real cache-line problem (False Sharing: Different Variables, Same Cache Line).
- • Memory exhaustion when the chain has an unbounded hop.
- • Connection and file-descriptor exhaustion when the chain blocks on an inbound path without a timeout.
- • Useless work: items are processed after the caller's own timeout has fired, so the whole pipeline is doing work nobody will read (Deadlines vs Timeouts).
- • Retry storms when refusal is honest but clients retry without backoff or jitter, converting a 20% overload into a 300% one.
- • Silent shedding when a drop policy exists and is not counted.
- • Starvation of low-rate clients when refusal is applied globally rather than per-client.
- • Whenever the arrival rate is controlled by someone else, which is every public endpoint, every broker consumer and every webhook receiver.
- • When there is a hard downstream limit — a third-party quota, a database connection pool, a GPU — because the limit exists whether or not you model it, and modelling it is the only way to fail gracefully.
- • When bursts are shorter than the buffer can absorb: a small bounded queue plus a short offer timeout turns a spike into slightly elevated latency instead of a refusal.
- • When the caller can degrade: a client that can drop a sample, batch harder, or retry later is a client that benefits enormously from being told to.
- • When the "producer" is a human-facing request that cannot be retried — refusing a checkout is worse than queueing it, and the right answer is capacity, not backpressure.
- • When the client ignores your signal. Backpressure against a client that retries immediately is a load amplifier, and you need admission control instead.
- • When the refusal threshold is set so tight that normal bursts are rejected, training operators and clients to treat 429 as noise.
- • When blocking is chosen on a path where a stalled caller holds a scarcer resource than the memory you were protecting.
- • Queue depth and head age at every hop — the *first* hop where age rises is the constrained stage, and it is rarely the one being blamed.
- • Refusal rate as a proportion of arrivals, split by client. A global refusal rate hides the single partner causing all of it.
- • Producer block time and offer-timeout expiries; both convert directly into user-visible latency.
- • Downstream in-flight count against the permit limit — pinned at the limit means the constraint is binding and correctly placed.
- • Client retry behaviour after a refusal: if refusals correlate with an *increase* in arrivals, the signal is being amplified rather than obeyed (Retry Storms: The Load You Generated Yourself).
- • Every enqueue site gains an error branch, and every error branch needs a policy decision the code must express.
- • The refusal must be part of the external contract — status code, headers, documented retry guidance — which makes it an API design change, not an implementation detail.
- • Per-client accounting, if you need fairness, adds state and cardinality that has to be bounded itself.
- • Tuning becomes multi-dimensional: queue capacity, offer timeout, permit count and client backoff interact, and changing one moves the others' correct values.
- • Add capacity. If the downstream can be scaled, backpressure is a stopgap and scaling is the fix — but model the limit anyway, because it moves rather than disappears.
- • Admission control at the edge: a rate limiter in front of the service refuses in one place with one policy, rather than at every internal hop (Rate Limiting in Architecture).
- • Batching: if the downstream cost is per-call rather than per-item, batching 50 items into one call multiplies effective throughput without touching the limit (Batch APIs and Partial Failure in API Design).
- • A durable broker in front of the pipeline, which converts backpressure into a growing but persistent backlog you can drain later — memory becomes disk and the failure mode becomes staleness (Message Queues in Architecture).
The producer is faster than the consumer
Bounding concurrency with permits
| permits | goodput | mean latency | timeouts | failed of 10K |
|---|---|---|---|---|
| 1 | 25/s | 43 ms | 0.00% | 0 |
| 5 | 125/s | 43 ms | 0.00% | 0 |
| 10 | 250/s | 43 ms | 0.00% | 0 |
| 25 | 625/s | 43 ms | 0.00% | 0 |
| 50 | 1000/s | 53 ms | 0.00% | 0 |
| 100 | 1000/s | 103 ms | 0.00% | 0 |
| 200 | 1000/s | 203 ms | 0.00% | 0 |
| 350 | 1000/s | 353 ms | 0.00% | 0 |
| 500 | 0/s | 503 ms | 100.0% | 10K |
What people believe, and what is true
We have backpressure — the queue is bounded.
Only if the enqueue site does something meaningful with the refusal. A bounded queue whose caller catches the exception and retries in a loop has a bound and no backpressure.
Returning 202 quickly is good for the caller.
A 202 for work the system cannot do is worse than a 429. It removes the caller's ability to react while the reaction would still have mattered.
Backpressure slows the system down.
It slows the *producer* down to the rate the system was already achieving. Throughput is unchanged; what changes is whether the excess is refused or buffered until something breaks.