Backoff and Jitter
Waiting longer between attempts stops you hammering a struggling dependency; randomising the wait stops every client from hammering it in unison.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
How long should I wait between retries, and why does adding randomness matter so much?
When the fraud service has a wobble, our retries should help it recover rather than finish it off.
Retry immediately, or sleep a fixed second between attempts. It is simple, predictable and easy to reason about.
Immediate retries send the second attempt while the dependency is in exactly the state that failed the first, so the extra load buys nothing and costs everything.
- Immediate retries send the second attempt while the dependency is in exactly the state that failed the first, so the extra load buys nothing and costs everything.
- A fixed delay makes every client retry at the same offsets. Clients that failed together retry together, and the dependency sees a wall of traffic every second instead of a smooth stream.
- Recovery becomes impossible: the moment the dependency comes back it is hit by every accumulated client at once and immediately falls over again (Thundering Herd in Concurrency).
- Backoff without a cap grows to minutes, so a retry eventually fires long after the user has given up and the request deadline has passed.
- Backoff without a deadline check means a request can spend most of its budget sleeping rather than working (Timeouts).
What is actually happening
- Backoff addresses volume: each successive attempt waits longer, so a client that keeps failing sends progressively less traffic. Exponential growth is the usual shape because it reduces load quickly without requiring you to know the dependency's capacity.
- Jitter addresses correlation, and it is the half people omit. Without randomness, clients that failed at the same moment retry at the same moments forever — the failure synchronises them, and synchronised clients produce spikes rather than load.
- Correlation is created by the incident itself. A dependency that returns errors for thirty seconds hands every client the same starting point, so deterministic backoff makes them a coordinated fleet (Retry Storms).
- Full jitter — sleeping a random duration between zero and the current exponential ceiling — is the common choice because it spreads attempts across the whole interval rather than clustering them near it. Variants exist (equal jitter keeps half the wait deterministic; decorrelated jitter derives the next wait from the previous one) and they trade spread against predictability.
- A cap on the ceiling stops the sequence growing past anything useful, and a deadline check stops the sleep outliving the request that wanted it.
- When the dependency tells you how long to wait, that instruction outranks your formula.
Retry-Afteris the dependency's own statement about its recovery, and ignoring it is how a throttled client stays throttled (Rate Limiting).
What each half of the mechanism actually does
Backoff and jitter solve two different problems and are routinely treated as one. Backoff reduces how much a single client sends. Jitter decorrelates when many clients send. Implementing only the first leaves you with a fleet that has agreed on a schedule.
The comparison below is the whole idea in code. Nothing about the retry logic changes; only the shape of the load arriving at the dependency does.
const wait = base * Math.pow(2, attempt) // attempt 1: 200ms, 2: 400ms, 3: 800ms — for EVERY client // // clients that failed together now retry together, // forever, in waves. The dependency sees spikes, // not load.
const ceiling = Math.min(base * Math.pow(2, attempt), maxBackoff) const wait = Math.random() * ceiling // each client draws its own wait in [0, ceiling) // // the fleet spreads across the whole interval, // and the ceiling still shrinks total volume // as attempts accumulate. if (deadline.remainingMs() <= wait) throw new DeadlineExceeded() await sleep(retryAfterMs(err) ?? wait) // the dependency's instruction wins
The deterministic version reduces one client's rate while keeping the fleet in lockstep, so total arriving load stays spiky and recovery is repeatedly knocked over. Drawing uniformly below the ceiling keeps the volume reduction and destroys the correlation, which is the property that actually lets the dependency recover.
Choosing a jitter strategy
Full jitter is the sensible default and the other options exist for real reasons. The criteria are how much spread you need against how much predictability you want to keep, and whether you can tolerate an attempt firing almost immediately.
Whichever you choose, the parameters that matter more than the strategy are the cap and the deadline check. An unbounded ceiling and an unchecked deadline will produce bad behaviour under any jitter formula.
How should the wait before the next attempt be distributed?
when A single client, or a fleet small enough that synchronisation is irrelevant.
cost Every client retries on the same schedule. Recovery is repeatedly disrupted by synchronised waves.
when The default. Many clients, shared dependency, recovery matters.
cost Highly variable per-request latency; an attempt can fire almost immediately, which is fine in aggregate and surprising in a trace.
when You want a guaranteed minimum wait, for example to let a downstream cache or a failover settle.
cost Half the spread of full jitter, so a larger residual spike.
when Long-running reconnect loops where you want the wait to wander upward rather than reset.
cost Harder to reason about and to bound; the cap becomes essential.
when The dependency told you. Always outranks the formula.
cost You are trusting the dependency's number, which may be far larger than your deadline allows — in which case fail now (Rate Limiting).
Everything that can synchronise, will
Retries are the well-known case, and they are not the only one. Any periodic or triggered behaviour shared across instances will eventually align, and the alignment is usually created by the incident you are recovering from.
The pattern to internalise: whenever many actors compute the same schedule from the same event, add randomness at the point the schedule is computed.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Cache entries populated during an outage | A second traffic spike exactly one TTL after recovery. | Identical TTLs created at the same moment expire at the same moment. | Randomise TTLs within a band; consider refresh-ahead (Cache Stampede). |
| Cron jobs across many instances | Every instance hits the database at the top of the hour. | Everyone shares a clock and a schedule. | Jitter the start; or elect one runner and let it fan out (Scheduled Jobs). |
| Database restart | All instances reconnect simultaneously and the connection storm delays recovery. | Reconnect loops with no jitter. | Jittered reconnect with a cap (Connection Pools). |
| Deploy restarts every instance | A burst of cold caches and simultaneous dependency warmup calls. | Rolling deploy with too little spacing. | Stagger the rollout; jitter startup warmup (Rolling Deployments). |
| Breaker opens fleet-wide | Every instance probes half-open at the same time; the dependency is hit by N probes. | Breaker timers started at the same instant. | Jitter the open duration; bound concurrent probes to one (Circuit Breakers). |
| Token expiry | All instances refresh credentials in the same second and the auth service throttles. | Tokens issued together expire together. | Refresh at a randomised fraction of the remaining lifetime. |
How to build it
Most important first.
- Grow the wait exponentially with the attempt number, cap the ceiling at a value that still fits the request budget, and randomise within it.
- Prefer full jitter — a uniform random draw between zero and the ceiling — unless you have a specific reason to keep a deterministic floor.
- Honour
Retry-Afterwhen present, in both its seconds and HTTP-date forms, and treat it as a lower bound rather than a suggestion. - Check the remaining deadline before sleeping. If the wait does not fit, fail now rather than sleeping into a deadline you will miss (Timeouts).
- Apply the same reasoning to reconnection loops, not just HTTP retries: database reconnects, broker reconnects and websocket reconnects all synchronise after a shared outage (Connection Pools).
- Add jitter to anything else that could synchronise across instances: cache TTLs, scheduled jobs, health-check intervals and cron fan-out (Cache Stampede, Scheduled Jobs).
- Combine with a breaker so that after sustained failure you stop generating attempts at all, rather than backing off forever (Circuit Breakers).
What can go wrong
- Jitter omitted because it "adds nondeterminism", which is precisely its function — and its absence is what turns a partial outage into a total one.
- Backoff applied per attempt but the total elapsed time never checked, so retries blow the caller's deadline while sleeping.
- An uncapped ceiling, producing a retry that fires long after anyone cares.
Retry-Afterignored in favour of the local formula, so a throttled client keeps arriving early and stays throttled (Rate Limiting).- Jitter applied to the delay but not to the initial attempt after recovery, so every client still reconnects in the same instant when the dependency returns.
- Randomness derived from a source seeded identically across instances — the classic case being a fixed seed in a container image, which produces the same "random" sequence everywhere.
- All clients waking from backoff together and racing to reconnect, which is the exact scenario jitter exists to break (Thundering Herd in Concurrency).
- A retry timer firing at the same moment a breaker transitions to half-open, so many probes arrive at once instead of one (Circuit Breakers).
- Cache entries created during an outage sharing a TTL and expiring simultaneously afterwards, producing a second wave with no retry involved (Cache Stampede).
- Deterministic retry timing is a fingerprint. A dependency or an observer can identify your client by its retry schedule, and predictable schedules make timing-based correlation easier.
- Backoff on authentication failures is what separates a client with a stale credential from something indistinguishable from a brute-force attempt (How Passwords Are Actually Attacked in Security covers the attacker side).
- Synchronised retry waves are a self-inflicted denial of service, and an attacker who can trigger a brief dependency failure can use your own fleet as the amplifier (Cascading Failure).
- "Exponential backoff is enough." Backoff alone leaves clients synchronised. The combination is what works; jitter is not a refinement, it is the other half.
- "Jitter is a small optimisation." It is the difference between a smooth recovery curve and a dependency that is knocked over again the instant it comes back.
- "Backoff means the request takes longer." Only failing ones. Successful first attempts are unaffected, which is why the cost is concentrated exactly where you want it.
- "We can back off far enough to avoid overload." Not indefinitely — past a point the correct action is to stop calling, not to call later (Circuit Breakers).
- "Backoff protects us." It protects the dependency. What protects you is a timeout, a breaker and a bulkhead.
Operating it
- A histogram of actual sleep durations. If it is spiky rather than spread, jitter is missing or misconfigured.
- Inbound request rate at the dependency during an incident: sawtooth means synchronised clients; a smooth decline means backoff and jitter are working.
- Attempt-number distribution over time. During recovery it should shift back toward one quickly; if it does not, the ceiling or the budget is wrong (Retries).
- Time spent sleeping as a fraction of total request duration. A request that spends most of its life in backoff is one that should have failed earlier.
- The more clients, the more jitter matters. Ten clients synchronising is noise; ten thousand is an outage on recovery.
- The jitter window needs to be wide enough to spread the fleet: a window far shorter than the number of clients divided by the dependency's capacity still produces a spike, just a shorter one.
- At high instance counts, a shared retry budget matters more than the backoff curve, because per-instance policies multiply (Retries).
- Backoff has a hard limit as a mitigation. Past a certain sustained failure rate the correct behaviour is to stop calling entirely, which is what a breaker provides (Circuit Breakers).
- Longer waits protect the dependency and increase user-visible latency for requests that would have succeeded on a fast second attempt.
- Full jitter maximises spread and makes any individual request's timing unpredictable, which complicates latency debugging and deadline accounting.
- Randomised waits make tests nondeterministic unless the clock and the random source are injectable, which is extra plumbing.
- Backoff spreads load in time and does nothing about load in aggregate. If the dependency is simply undersized, backoff delays the failure rather than preventing it.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALApplies to any repeated attempt against a shared resource: HTTP retries, database reconnects, broker reconnects, lock acquisition, polling loops.
- SCALE-SPECIFICWith a handful of clients, deterministic backoff is fine because there is no meaningful synchronisation to break. Jitter becomes load-bearing as client count rises relative to the dependency's capacity, which is why this problem appears abruptly during growth.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — why independent clients converge on correlated behaviour after a shared failure, and how randomisation restores independence.