Overload & Backpressure

Cap Retries as a Fraction of Traffic, Not as a Count per Request

"Retry up to three times" sounds like a limit. It is not: it bounds one request while leaving the aggregate unbounded, because the number of requests is not something you control. A budget bounds the thing that actually hurts — total retry traffic as a share of original traffic.

▶ Run the lab

The question this answers

The question

What is the right limit on retries, if "three per request" does not bound anything that matters?

The guarantee — the property claimed, and its scope

Retry traffic to a given dependency stays below a stated fraction of original traffic to that dependency (typically 10–20%), measured over a rolling window, at the cost of some retries being refused during exactly the periods when retries fail most. It bounds *load*, not per-request success — an individual request may get zero retries because the budget was spent elsewhere.

Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.

What a node knows — observation versus inference

A client with a budget knows its own recent request and retry counts for one dependency, from one process. It does not know the fleet-wide totals unless the budget is shared, and it cannot tell whether the failures it is seeing are local (this instance’s network path) or global (the dependency is down) — which matters, because those two cases want opposite responses.

A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.

What guarantee?→What does a node know?→How does it work?→What can fail?→How does it fail?→Where is coordination?→What holds under failure?→How does it recover?→How would you know?→What is the simpler thing?
retriesbudgetstoken bucketoverload

Why the per-request count is the wrong control

A per-request retry count is a bound of the form "each request may generate at most 3 attempts". Multiply it by a quantity you do not control — request rate — and it bounds nothing. At 10,000 requests per second, "3 attempts each" authorises 30,000 requests per second against a dependency you sized for 10,000. The setting names a small number and licenses an unbounded one.

Worse, the number of retries actually issued is a function of the failure rate, and the failure rate is highest when the dependency is least able to absorb them. The control variable and the load it produces are *positively correlated with the fault*, which is the opposite of what a safety limit should do. A limit that relaxes as the system degrades is not a limit.

A budget inverts this. It says: the total retry traffic I send to this dependency may not exceed 10% of the original traffic I send it. Now the ceiling is proportional to healthy demand rather than to failure, and total offered load is bounded by 1.1× no matter how comprehensively the dependency is failing. During a total outage, the per-request policy sends 3×; the budget sends 1.1×. That difference is often the difference between a dependency that recovers when you fix it and one that cannot.

Failure rate pPer-request policy (3 attempts)10% retry budgetRatio
p = 0.001 (healthy)assumption10,020 rps10,010 rps~1.0×
p = 0.1 (degraded)assumption13,300 rps11,000 rps1.2×
p = 0.5 (bad)assumption26,600 rps11,000 rps2.4×
p = 1.0 (down)assumption30,000 rps11,000 rps2.7×
10,000 rps of original traffic, dependency failing at rate p

A token bucket where success pays for failure

The standard implementation is a token bucket in the client, one per dependency. Every request — success or failure — deposits ratio tokens; every retry withdraws one. If the bucket is empty, the retry is refused and the original failure is returned to the caller immediately.

The elegance is in what happens automatically at the extremes. While the dependency is healthy, deposits vastly outnumber withdrawals, the bucket sits full, and every retry that wants to happen happens — the budget is invisible. As the failure rate climbs, withdrawals outpace deposits, the bucket drains in seconds, and retries stop. The budget turns itself off exactly when retrying has stopped working, without anyone measuring the dependency’s health or agreeing on a threshold.

The bucket needs a cap — a maximum token count — or a long healthy period banks enough credit to fund a burst that defeats the whole purpose. A cap of a few seconds’ worth of retries is typical: enough to absorb a transient blip, not enough to fund a storm.

Scope matters as much as the ratio. One budget per dependency, not one global budget: a shared budget lets a single broken downstream drain the credit that healthy dependencies needed, converting one dependency’s outage into a fleet-wide retry outage. This is Bulkheads: Buying Independence by Giving Up Utilisation applied to a budget rather than to a thread pool, and the failure it prevents is the same one.

1class RetryBudget {
2  private tokens: number
3  constructor(
4    private readonly ratio = 0.1,   // retries may be at most 10% of requests
5    private readonly maxTokens = 100, // cap: ~10s of retries at 10% of 100 rps
6    private readonly minTokens = 10,  // floor so low-traffic clients can retry at all
7  ) {
8    this.tokens = minTokens
9  }
10 
11  // Called for EVERY attempt outcome, success or failure.
12  onRequest() {
13    this.tokens = Math.min(this.maxTokens, this.tokens + this.ratio)
14  }
15 
16  // Returns false when the budget is spent — the caller must surface the
17  // original error rather than retrying. This is the load-bearing line.
18  tryWithdraw(): boolean {
19    if (this.tokens < 1) return false
20    this.tokens -= 1
21    return true
22  }
23}
24 
25// One budget per dependency, never one shared budget: a single broken
26// downstream must not be able to spend the credit healthy ones need.
27const budgets = new Map<string, RetryBudget>()
Per-dependency retry budget — the shape used by gRPC and by Finagle-style clients

The budget must be visible, or it becomes a silent failure

A refused retry looks, to the caller, exactly like a request that failed once and was not retried. That is a real behaviour change and it must be observable, or you will spend an incident convinced your retry configuration is broken while it is in fact working perfectly.

So the required signal is a counter: retries refused by budget, per dependency. When it is zero, the budget is not binding and your protection is theoretical. When it is large, the budget is doing its job *and* you now know the dependency is failing broadly — the counter doubles as a high-quality health signal, because it only rises when failures are systemic rather than sporadic.

The other honest consequence is that a budget cannot distinguish "this instance cannot reach the dependency" from "the dependency is down for everyone", and it responds to both by refusing retries. For the local-fault case that is the wrong answer: retrying to a different replica would have worked. Mitigate by scoping the budget per dependency *and* retrying to a different endpoint rather than the same one, so the retry you do spend has the best chance of being the useful kind.

Finally: a budget bounds volume, not timing. Ten percent of traffic arriving as one synchronised burst is still a spike. Budgets and Without Jitter, Every Client That Failed Together Retries Together solve orthogonal halves of the same problem, and you need both.

inventory.requests.rate            9,980/s
inventory.errors.rate             9,850/s   <- dependency is down
inventory.retries.attempted         998/s   <- capped at the 10% ratio
inventory.retries.refused_budget  8,850/s   <- the budget is binding
inventory.offered_load           10,978/s   <- 1.10x, not 3.00x

payments.retries.refused_budget       0/s   <- separate budget, unaffected
What a working budget looks like during a dependency outage

Key points

  • A per-request attempt count bounds one request and leaves aggregate retry load unbounded, because request rate is not yours to control.
  • Retry volume under a count policy scales with the failure rate — the load rises precisely as the dependency’s ability to absorb it falls.
  • A budget expresses the limit as a fraction of original traffic, so total offered load is bounded (e.g. 1.1×) regardless of how badly the dependency is failing.
  • A token bucket funded by every request and drained by every retry disables retries automatically when they stop working, with no health threshold to tune.
  • Budgets must be per dependency, and refusals must be counted — an invisible budget looks identical to a broken retry configuration.

The chain, answered

Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.

How it works
  • • Each client process holds a token bucket per dependency, with a deposit ratio, a cap, and a small floor.
  • • Every attempt outcome deposits ratio tokens, so the bucket fills in proportion to real demand.
  • • A failure that is classified as retryable requests one token before retrying.
  • • If a token is available it is withdrawn and the retry proceeds; if not, the original error is returned to the caller and a refusal is counted.
  • • The cap prevents a long healthy period from banking enough credit to fund a burst when failures begin.
What can fail at the boundary
  • • The budget is shared across dependencies, so one broken downstream starves retries for healthy ones.
  • • The ratio is set high enough (50%, 100%) that the budget never binds and provides no protection.
  • • The cap is missing or huge, so accumulated tokens fund exactly the storm the budget was meant to prevent.
  • • Retries happen at a layer that has no budget — a service mesh, an SDK default, a load balancer — and the budgeted layer’s numbers look fine.
  • • The budget is per process, so fleet-wide retry load is ratio × instances relative to what a single-instance reading suggests.
How it fails — what an operator sees
  • • Silent retry loss: success rate drops for a dependency that has a transient fault, and the retry-attempt counter is flat. Without a refused_by_budget counter this reads as "retries are not configured" and gets misdiagnosed for an hour.
  • • Cross-dependency starvation from a shared budget: payments retries stop working during an inventory outage. The operator sees two unrelated services degrade together with no shared infrastructure between them.
  • • Budget that never binds: during a full dependency outage, offered load still reaches 3× baseline and refused_by_budget is zero. The ratio is too generous, or the retries are happening at an unbudgeted layer.
  • • Burst despite budget: retry volume is within the 10% cap but arrives as a single spike each second, and the dependency sees periodic saturation. The budget bounded volume; nothing bounded timing.
Where coordination is required
  • • A per-process budget requires no coordination and is therefore the default — but the fleet-wide limit is ratio only if you reason per instance and accept that the aggregate is ratio × instances of that instance’s own traffic.
  • • A genuinely global budget needs shared counters on the failure path, which adds a dependency that must work during an outage. Rarely worth it.
  • • Because the budget is proportional to each client’s own traffic, the uncoordinated version composes better than most limits: a client sending 1% of traffic gets 1% of the retry allowance automatically, without anyone computing a share.
What still holds under failure
  • • Individual requests lose their retries first, so per-request success rate degrades before the dependency does — a deliberate trade of individual reliability for collective survival.
  • • Total offered load to the failing dependency stays bounded, preserving its ability to recover once the underlying fault is fixed.
  • • The budget state is local and in-memory, so it resets on restart — a fleet-wide restart during an incident temporarily removes the protection.
How it recovers
  • • Detect: alert on retries_refused_by_budget > 0 sustained. It is a near-zero-noise signal that failures have become systemic.
  • • Contain: the budget contains by construction; the operational lever during an incident is to lower the ratio, not to raise it.
  • • Recover: as the dependency heals, deposits outpace withdrawals and the bucket refills within seconds, restoring retries with no operator action.
  • • Reconcile: requests that lost their retry may need re-driving from a durable source if the work mattered — a budget assumes some requests will simply fail.
  • • Verify: inject a 100% failure rate on a dependency in a load test and confirm offered load stays near 1 + ratio. If it reaches 3×, something retries outside the budget.
How you would know
  • • Retry ratio actually achieved per dependency: retries divided by requests, which should sit at or below the configured ratio at all times.
  • • Retries refused by budget, per dependency — the signal that the budget is binding and that failures are systemic.
  • • Token bucket level as a gauge; it drains in seconds during an incident and its shape tells you whether the cap is right.
  • • Offered load to the dependency as a multiple of original request rate — the number the whole design exists to bound.
When it helps
  • • Any client calling a dependency that can fail broadly rather than sporadically — which is most shared infrastructure.
  • • Fleets large enough that per-request policies produce aggregate loads nobody has computed.
  • • As a complement to a circuit breaker: the budget handles the long tail of partial failure that never trips the breaker’s threshold.
When it hurts
  • • Very low traffic clients, where the ratio yields less than one token and no retry is ever possible — hence the minimum floor, which must be set deliberately.
  • • Dependencies whose failures are genuinely independent per request (a flaky network path to one of many replicas), where retrying nearly always works and a budget refuses useful retries.
  • • When the true problem is one bad replica: the budget suppresses retries globally when the right answer was to retry elsewhere.
Simpler alternatives
  • • A circuit breaker: stops all traffic to a dependency once failures exceed a threshold. Blunter, faster, and it also stops the requests that would have succeeded. Architecture owns the mechanism.
  • • An adaptive concurrency limit, which bounds in-flight work rather than retry share and handles slowness as well as failure. See Decide at the Door Whether the Capacity Exists.
  • • Hedging with a strict budget instead of retrying on failure — the same accounting applied to latency rather than errors. See Send a Second Request After p95 and Take Whichever Answers First.
  • • Simply not retrying: correct for any operation where the caller has a durable source of truth and can re-drive the work later.

Retry budgets: cap the aggregate, not the request

Retry budgets: cap the aggregate, not the request
"Retry up to three times" bounds one request and leaves total retry traffic unbounded, because the number of requests is not something you control. A budget bounds the thing that hurts.
retries the count permits
1,710/s
retries the budget permits
414/s
in the incident
100/s of 1,710/s wanted
extra load on a sick dependency
10%
1,710/s0
retries wanted by "3 attempts"retries the budget allowstokens in the bucketdependency degrading left to right
100%0
dependency failure rate
class RetryBudget {
  tokens = minTokens
  onRequest()   { tokens = Math.min(maxTokens, tokens + 0.1)  }  // every attempt, success or failure
  tryWithdraw() { if (tokens < 1) return false; tokens -= 1; return true }
}
// one budget PER DEPENDENCY — a shared budget lets one broken downstream
// spend the credit the healthy ones needed.
Watch the two extremes and note that nobody configured either of them. While the dependency is healthy, deposits vastly outnumber withdrawals, the bucket sits at its cap, and every retry that wants to happen happens — the budget is invisible. As the failure rate climbs, withdrawals outpace deposits, the bucket drains, and retries stop. The budget turns itself off exactly when retrying has stopped working, with no health check and no threshold to agree on. The cap matters: without one, a long healthy period banks enough credit to fund precisely the storm the budget exists to prevent.
simplifiedThe token bucket is the standard client-side shape — deposit on every attempt, withdraw on every retry. Steps are arbitrary units of time; the crossing point is the lesson, not its clock time.

What people believe, and what is true

Claim

Three retries per request is a bounded retry policy.

Reality

It bounds a request. Aggregate retry load is 3 × request rate × failure rate, and two of those three terms are outside your control.

Claim

A retry budget means some requests do not get retried, so it lowers reliability.

Reality

It lowers reliability for individual requests during broad failures — periods when retries were not going to succeed anyway — in exchange for the dependency being able to recover at all.

Claim

One global retry budget is simpler.

Reality

It couples every dependency together: one broken downstream drains the shared credit and healthy dependencies lose their retries. Per-dependency budgets are the isolation boundary.

Claim

With a budget we do not need backoff.

Reality

A budget bounds how many retries; backoff and jitter bound when. Ten percent of traffic delivered as one synchronised burst still saturates the target.

Go deeper

Only the levels this lesson can honestly fill — a missing level is a claim nobody had.

Overview

Do not limit retries per request. Limit them as a share of your traffic — say 10% — so that even a total outage produces only 1.1× load instead of 3×.

Practical

Give each dependency its own token bucket: every request deposits 0.1 tokens, every retry costs 1, cap it at a few seconds of credit, and set a small floor for low-traffic clients. Count refusals and alert on them. Then load-test with a fully failed dependency and check that offered load lands near 1.1×.

Advanced

The budget is a proportional controller with an automatic shutdown property: because deposits track demand and withdrawals track failure, the bucket empties exactly when the failure rate exceeds the ratio, which is the condition under which retrying has negative expected value. It needs no health model of the dependency, which is why it degrades gracefully where threshold-based breakers oscillate around their trip point.

Apply it

Interview questions
  • 💬 Why is "retry 3 times" not a limit on retry load?
  • 💬 Design a retry budget. What refills it, what drains it, and what happens when it is empty?
  • 💬 Your budget is shared across all dependencies. What incident does that design produce?
← One Retry per Tier Is Not One Retry — It MultipliesWithout Jitter, Every Client That Failed Together Retries Together →