Retry Storms: The Load You Generated Yourself
A dependency gets slower, clients retry, the retries become load, the dependency gets slower still. The feedback loop is what turns a 5% error rate into an outage — and it is the one source of traffic you can switch off yourself.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The loop, and why it accelerates
Retries are usually harmless because failures are usually rare and uncorrelated. The assumption breaks precisely when it matters: during a partial outage, failures are *correlated*, so every client retries at the same time, and the retry traffic arrives while the dependency is at its weakest.
The arithmetic is unforgiving. A policy of "3 retries" means a fully failing dependency receives 4x its normal request volume. Add a second layer — the service retries, and the worker that called the service also retries — and the multiplication compounds to 16x. Three layers is 64x. Nobody designs a 64x amplifier; it emerges because each layer was written by a different team and each one independently decided that retrying was the responsible thing to do.
The loop also outlives its cause. Once the queue behind those retries has filled, recovery of the dependency does not end the incident: there is now a backlog of retry work that immediately re-saturates it. This is why retry storms produce the characteristic "it recovers for thirty seconds and falls over again" pattern, and why draining or shedding the retry backlog is often a required step in recovery rather than an optional one.
Amplification is measurable before it is fatal
The signal that separates "the dependency is failing" from "we are attacking the dependency" is the ratio of total attempts to unique logical operations. Instrument every retryable call with a stable operation id and count both. In steady state the ratio sits just above 1.0. When it climbs past 2, the majority of the traffic you are sending is repeat work, and scaling anything is pouring fuel on the fire.
This is the number that makes the case in an incident channel, because it reframes the question. "The provider is down" invites waiting; "we are sending them 4.2x their normal volume" invites turning something off. The second framing is actionable within seconds and the first is not.
The corollary matters for The Backlog Arithmetic: Four Levers and a Drain Time: when arrival rate rises, always check whether unique operations rose with it. Traffic growth and retry amplification look identical on an arrivals graph and demand opposite responses — scale up for the first, shed and back off for the second.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| dependency error rate | 6% → 31% | Rising, but this is the consequence being reported, not the cause of the collapse. | suspect |
| requests to dependency | 2,000/s → 8,400/s | Volume more than quadrupled with no upstream traffic change. Someone is generating this. | smoking gun |
| unique operations | 2,000/s → 2,050/s | Essentially flat — real demand did not change at all. | normal |
| attempt ratio | 1.02 → 4.10 | Four attempts per logical operation. Three-quarters of the load is self-inflicted. | smoking gun |
| retry queue depth | 400 → 190,000 | The backlog will re-saturate the dependency the moment it recovers. | smoking gun |
| upstream request rate | 2,010/s | Unchanged. This rules out a traffic spike as the trigger. | normal |
Backoff, jitter, budgets, breakers — and what each buys
Exponential backoff spreads retries over time so the second attempt does not arrive while the dependency is still failing. Jitter spreads them over *clients*, which is the part teams skip and the part that matters most: without jitter, a thousand clients that failed at the same instant retry at the same instant, and synchronised backoff is just a slower drumbeat rather than a solution.
A retry budget is the strongest of the four because it bounds amplification globally rather than per-caller. The rule is simple — retries may not exceed some fraction (commonly around 10%) of successful requests over a rolling window; beyond that, failures return immediately. This caps worst-case amplification at 1.1x no matter how many layers independently decided to retry, which is exactly the failure mode per-call policies cannot see.
Circuit breaking stops sending entirely once failure rates cross a threshold, letting the dependency recover unmolested and returning fast failures upstream. Its cost is honesty about degradation: some requests fail immediately that might have succeeded. That is usually the right trade during an outage and the wrong trade during a brief blip, which is why breaker thresholds are worth tuning against real incident data. Depth on the pattern itself lives in Architecture → Circuit Breaker and Reliability Patterns.
| Control | What it bounds | Amplification cap | Cost / failure mode |
|---|---|---|---|
| Exponential backoff | Retry rate per caller over time | Still 4x for a 3-retry policy, just spread out | Longer tail latency for requests that eventually succeed |
| Jitter | Synchronisation across callers | No cap, but removes the thundering herd | Nearly free; the common omission — backoff without it stays synchronised |
| Retry budget | Total retries as a fraction of successes, globally | ~1.1x regardless of layer count | Needs shared state or a per-process approximation; some retryable failures give up early |
| Circuit breaker | Whether calls are sent at all | 1.0x while open | Fails requests that might have succeeded; thresholds need tuning against real incidents |
| Idempotency keys | Not amplification — *correctness* under retry | None | Storage and key lifecycle, but without it retries can double-charge (Idempotency Keys: The Mechanism) |
Key points
- Retries assume failures are rare and uncorrelated; during an outage they are neither, so retry traffic arrives exactly when the dependency is weakest.
- Amplification multiplies across layers: 3 retries per layer is 4x, then 16x, then 64x — nobody designs this, it emerges from independent decisions.
- The diagnostic signal is total attempts ÷ unique logical operations; above ~2 the majority of load is self-inflicted.
- Jitter is the most-skipped and highest-value control: backoff without jitter keeps a thousand clients synchronised.
- Retry budgets cap amplification globally at ~1.1x, which is the only control that sees across independently-written layers.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Dependency → callers: provider latency rises past the client timeout, converting slow responses into timeouts and therefore into failures.
- 2Callers → retry policy: each failure schedules up to three retries with no jitter, so retries from all callers arrive in synchronised waves.
- 3Retries → dependency: request volume against the provider reaches 4.1x normal while unique operations stay flat, confirming the extra load is self-generated.
- 4Load → dependency: the provider, already degraded, now serves four times the traffic and its error rate climbs from 6% to 31%, generating still more retries.
- 5Retry queue → recovery: 190,000 queued retries mean that even after the provider recovers it is immediately re-saturated, producing the recover-and-collapse oscillation.
- • "The provider is down, nothing we can do" — three-quarters of the traffic hitting them is ours; turning it off is entirely within our control.
- • "Traffic spiked, scale up" — upstream request rate never moved. Scaling workers here multiplies the attack.
- • "We added exponential backoff, we are protected" — without jitter, backoff synchronises the herd rather than dispersing it.
- • "Errors dropped when the breaker opened, we are recovered" — the breaker is failing requests fast; user-visible success has not returned.
- • "Retries improve reliability" — retries without backoff, jitter and a budget reduce reliability by converting a partial failure into a total one.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Total attempts and unique logical operations as two counters on one graph, with the ratio published as its own metric.
- • Retry queue depth and its growth rate, because that backlog is what re-saturates the dependency after recovery.
- • Dependency request volume compared to upstream request volume — a widening gap with flat upstream traffic is amplification, not demand.
- • Circuit breaker state as a metric (closed/open/half-open) with transitions annotated on the latency graph.
- • Attempt-number distribution: what fraction of successful calls succeeded on attempt 2, 3, 4 — a heavy tail means the policy is doing real work and needs backoff tuning.
- • Open the circuit breaker (or manually disable retries) to stop the amplification immediately — this is the fastest lever and it is entirely ours to pull.
- • Drain or discard the retry backlog before declaring recovery, otherwise the dependency is re-saturated the moment it returns.
- • Add full jitter to every backoff policy; this is a small change with the largest effect on synchronised herds.
- • Introduce a retry budget capping retries at a fraction of successes, which bounds amplification across layers that cannot see each other.
- • Audit the call chain for stacked retries and retry at exactly one layer — usually the outermost one that can still make a meaningful decision.
- • Ensure retried operations are idempotent, so bounded retries are safe to keep rather than something to be afraid of.
- • Attempt ratio returns to near 1.0 and stays there through the next dependency blip — this is the direct measure of whether the loop is closed.
- • Inject dependency failures in a load test and confirm request volume against it stays within 1.2x of baseline rather than multiplying.
- • Confirm recovery is monotonic rather than oscillating: after the fix, a recovering dependency should not be knocked over again by queued retries.
- • Check the attempt-number distribution: retries should still be succeeding on attempt 2 for transient blips, proving the policy was bounded rather than removed.
- • Circuit breakers fail requests that might have succeeded; during a brief blip that is a self-inflicted error-rate spike, so thresholds need real tuning.
- • Retry budgets need shared state to be exact; per-process approximations are cheaper but drift under uneven load distribution.
- • Backoff with jitter increases the latency of requests that do eventually succeed, which shows up in the tail even when the fix is working correctly.
- • Retrying at exactly one layer means inner layers surface transient failures they previously hid, so error handling upstream has to become more explicit.
- • An alert on attempt ratio > 2 sustained, which catches amplification while it is still small.
- • A chaos or load test that fails a dependency and asserts an amplification ceiling, run in CI before release.
- • A code-review rule that retry policies at more than one layer of a call chain require explicit justification and a documented total amplification factor.
- • Retry budget and jitter defaults set in the shared HTTP/queue client library, so new services inherit the safe policy instead of writing their own.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe 4.1x ratio and 190,000-deep retry queue are constructed to make the loop legible. The multiplication rule (retries+1 per layer, compounding) is general; the specific figures are not.
- ENVIRONMENT-SPECIFICThe commonly-cited ~10% retry budget is a starting point from published SRE practice, not a universal constant — the right fraction depends on your dependency's headroom and your tolerance for early failures.
Misconceptions
Apply it
Where the depth lives
The closed/open/half-open transitions and their thresholds are a design decision taught there; here we only read the state as a signal.