Reliability Patterns
Timeouts, retries with backoff and jitter, circuit breakers, bulkheads, rate limits, fallbacks and graceful degradation each exist because of one specific failure — and each, applied without its budget, becomes a new way to turn a slow dependency into a full outage.
A remote call can hang, fail, or succeed slowly; without explicit policy the caller inherits the dependency’s worst behaviour, and a single slow downstream service takes the whole request path with it.
Every pattern is an answer to one failure
A call across a network has four outcomes, not two: success, failure, slow success, and unknown (the request left, no answer came back). Application code written for the first two is written for a function call, not a network. The reliability patterns are the missing handling for the other two. A timeout converts "unknown" into a decision after a bounded wait. A retry turns a transient failure into a success at the cost of a second attempt. A circuit breaker stops attempting when the dependency is clearly down. A bulkhead keeps one slow dependency from consuming the threads or connections every other call needs. A rate limit protects a dependency from callers who are collectively too enthusiastic. A fallback returns something useful when the real answer is unavailable, and graceful degradation is the product-level version: the page still renders without the recommendations strip.
None of these is free. Each one adds a decision the system makes on your behalf, and each one has a well-known way of making the outage worse when that decision is wrong. The skill is not knowing the names; it is knowing which failure each one buys back and which failure it introduces.
The eight patterns, their problem, and their misuse
Read the last column as carefully as the second. Most reliability incidents are not caused by a missing pattern; they are caused by a present pattern configured without regard for its neighbours — a 30 s timeout inside a 10 s caller, three retries at each of three layers, a fallback that quietly served empty carts for four hours while the dashboard stayed green.
| Pattern | The problem it answers | Mechanism | Failure it introduces if misused |
|---|---|---|---|
| Timeout | A call that never returns holds a thread, a connection and the user forever | Abort after a bounded wait; propagate a deadline downstream | Timeout longer than the caller’s: the caller gives up first and retries, and the original work still completes — duplicate side effects |
| Retry | Transient failures (connection reset, 503, leader election) succeed on a second attempt | Re-issue the call, with backoff and jitter, up to a budget | Retry storm: N retries at each of M layers is N^M amplification against a dependency that is slow precisely because it is overloaded |
| Exponential backoff + jitter | Synchronised retries arrive as a wave | Wait base × 2^attempt, randomised over [0, cap] | Backoff without jitter: every client waits the same 2 s and returns as one wave |
| Circuit breaker | Calling a dead dependency wastes the timeout on every request | Track failure rate over a window; open the circuit and fail fast; probe in half-open | One global breaker for many dependencies, or a breaker that trips on 5 calls: one bad user request opens it for everyone |
| Bulkhead | One slow dependency exhausts the shared thread/connection pool | Separate pools or semaphores per dependency; a full pool rejects instead of queueing | Pools sized so small that normal traffic is rejected; or so large that the bulkhead does nothing |
| Rate limit | A dependency (or a tenant) is overwhelmed by aggregate demand | Token bucket / sliding window per client, tenant or route; 429 + Retry-After | Limits below real traffic during a launch; limits keyed on a shared IP behind a NAT |
| Fallback | The real answer is unavailable but a degraded answer is acceptable | Cached value, default, or a queued deferral | A fallback that hides a full outage: empty recommendations, stale prices, or a "success" that was never persisted |
| Graceful degradation | Losing one feature should not lose the page | Rank features by criticality; shed the optional ones first under load | Degrading the critical path (checkout) while protecting the optional one (banners) because nobody ranked them |
Retries: the pattern that makes outages worse
A retry is a bet that the second attempt succeeds. It pays off for transient failures: a connection reset, a 503 from one instance during a rolling deploy, a leader election lasting 200 ms. It loses badly for overload: if the payment provider is slow because it is saturated, every retry adds load to the thing that is failing for lack of capacity. Three retries at the gateway, three at the order service and three at the payment client is 3 × 3 × 3 = 27 attempts per user click. That is not resilience; it is a distributed denial of service against your own dependency, and it is exactly how the challenge retry-storm-took-down-payments unfolds.
Three rules make retries safe. Retry only what is retryable: connection errors, 503, 429 with Retry-After, and idempotent operations. A POST /charge without an idempotency key is never retryable; the idempotency lesson in the async module shows how the key turns a repeated request into a repeated *response*. Retry with a budget, not a count: allow at most, say, 10% of a service’s calls to be retries over a window; when the budget is spent, retries stop and the failure surfaces. Retry at one layer, ideally the one closest to the failure, and let the layers above see the final result.
1export async function withRetry<T>(2 fn: () => Promise<T>,3 opts = { attempts: 3, baseMs: 100, capMs: 2_000 },4 isRetryable: (e: unknown) => boolean = () => true,5): Promise<T> {6 for (let attempt = 0; ; attempt++) {7 try {8 return await fn()9 } catch (err) {10 if (attempt + 1 >= opts.attempts || !isRetryable(err)) throw err11 const ceiling = Math.min(opts.capMs, opts.baseMs * 2 ** attempt)12 const wait = Math.floor(Math.random() * ceiling) // full jitter13 await new Promise((r) => setTimeout(r, wait))14 }15 }16}Timeouts and bulkheads: bounding the blast radius
A timeout is the pattern people most often omit and most often set wrong. Omitted, a slow dependency holds a thread or connection per in-flight request until the pool is empty; the service is then "down" for every endpoint, including the ones that never touch the slow dependency. That is the challenge one-slow-dependency-took-everything: no timeout, one shared pool, and a recommendations service that started answering in 12 s instead of 40 ms. Set wrong — longer than the caller’s own timeout — the caller abandons and retries while the original call is still running, so the work happens twice and the caller never learns which attempt won.
The rule is deadline propagation: the edge sets a total budget (say 2 s), and every hop passes the *remaining* budget downstream, so no inner call can outlive its caller. gRPC carries deadlines natively; over HTTP you pass a header and subtract elapsed time at each hop. The bulkhead is the complementary bound: give each dependency its own connection pool or semaphore sized to its expected concurrency (p99 latency × request rate, by Little’s law), so a full recommendations pool rejects the recommendations call and everything else keeps working.
- Set the timeout from the dependency’s measured p99 plus headroom, not from a default. A 30 s default on a 40 ms call is not a timeout; it is a promise to wait 750× longer than normal.
- A rejected call from a full bulkhead should be a fast, typed failure the caller can degrade on — not an exception that bubbles up as a 500.
- Timeouts, retries and breakers each need their own metric: timeouts per route, retry ratio, breaker state. An invisible pattern is an untuned pattern.
Order of application: timeout → retry (with budget) → breaker → fallback
The patterns compose as layers around the raw call, and the order is not arbitrary. The timeout wraps each individual attempt, so a retry has a fresh, bounded window rather than inheriting a stalled one. The retry wraps the timeout, but consults its budget before every attempt. The breaker wraps the retry, so that when the dependency is down, no attempts are made at all — a breaker inside the retry loop would let each retry re-check and re-trip it. The fallback wraps everything, so it sees the final outcome and can answer with a cached value, a default, or a deferral to a queue. Read from the inside out, the wrapping is also the order in which each failure is *discovered*: attempt slow → attempt failed → dependency failing → request degraded.
- Graceful degradation is a product decision made before the incident: which features are optional, in what order they are shed, and what the user sees. Under load, shed optional features first and protect the critical path.
- A fallback that succeeds must be visible: log it, count it, alert when the fallback rate exceeds a threshold. "The site is up" and "the site is serving cached prices from Tuesday" are different states.
- The simplest reliable configuration is usually: one timeout per hop from measured p99, one retry at one layer for idempotent calls only, one breaker per dependency, and an honest error when all of that fails.
fallback( // 4. answer something useful
breaker( // 3. stop trying when it is clearly down
retryWithBudget( // 2. try again, but only within budget
timeout(2_000 ms, // 1. bound every single attempt
() => paymentClient.charge(order)
)
)
)
)Key points
- A network call has four outcomes — success, failure, slow success, unknown — and the reliability patterns exist to handle the last two.
- Retries are a bet that the second attempt succeeds; against an overloaded dependency they lose, and N retries at M layers is N^M amplification.
- Timeouts must be shorter than the caller’s; propagate a remaining deadline downstream so no inner call outlives its caller.
- Bulkheads isolate the blast radius: one slow dependency fills its own pool, not the one every endpoint shares.
- Compose inside-out: timeout → retry with budget → circuit breaker → fallback, and make every fallback visible in metrics.
Eight reliability patterns, visualised
Compose them in this order on every outbound call: timeout → retry (with budget, only if idempotent) → circuit breaker → fallback the timeout bounds one attempt · the budget bounds all attempts · the breaker stops attempts · the fallback decides what the user sees
How data moves through it
One request or event, hop by hop.
- 1Client → Order Service:
POST /checkout, edge sets a 2 s total deadline in a header. - 2Order Service → breaker: checks the payment breaker state; if open, skips straight to the fallback.
- 3Order Service → Payment Provider: one attempt with a 600 ms timeout (measured p99 400 ms + headroom); on 503 or connection reset, one retry after 0–200 ms of jittered backoff, if the retry budget allows.
- 4Payment Provider → Order Service: success, or a final failure that increments the breaker’s failure window.
- 5Order Service → Queue (fallback): if payment is unavailable, persist the order as
PENDING_PAYMENTand enqueue a retry job; respond 202 to the client with a status URL.
When to use — and when not
- Any synchronous call to a dependency you do not control — a payment provider, a third-party API, another team’s service.
- When one dependency’s latency has, or could, take down endpoints that do not use it.
- When the product can define a degraded but acceptable answer (cached, default, deferred).
- Retries on non-idempotent operations without an idempotency key: you convert a timeout into a duplicate charge.
- Fallbacks for correctness-critical data — a stale balance or a default "in stock" is worse than an honest error.
- Breakers and bulkheads on in-process calls: they add machinery and a failure mode to something that cannot partition.
Tradeoffs
Cheap individually, expensive in interaction: every pattern needs a metric and a tuned threshold, and the interactions (timeout vs retry, retry vs breaker) are where incidents live.
How it fails
- Retry storm: three retries at each of three layers become 27 attempts per request against a dependency that is failing for lack of capacity.
- Inner timeout longer than the outer one: the caller retries while the original call is still executing, and side effects happen twice.
- Fallback masking an outage: the recommendation service is down for hours and nobody notices because the empty-list fallback returns 200.
- Missing bulkhead: a slow, non-critical dependency exhausts the shared connection pool and the checkout path fails with it.
- Breaker with a tiny minimum-call threshold: a handful of failures from one bad request opens the circuit for every tenant.
How it scales
- Retries multiply load: budget them as a percentage of traffic so their cost stays proportional as the fleet grows.
- Bulkhead pools and rate limits are per instance; with autoscaling, the aggregate limit against a dependency scales with instance count — size them from the dependency’s capacity, not from one instance’s comfort.
- Breaker state is per instance, so a fleet of 50 instances discovers an outage 50 times; that is acceptable and simpler than sharing state.
How it interacts with databases, queues, caches, APIs and external systems
- External APIs: the primary target. Timeouts from measured p99, retries only for documented transient codes, an idempotency key on every mutating request.
- Databases: a connection pool is a bulkhead; a query timeout (
statement_timeout) is a timeout; the pool being full is the signal, not a bug. - Queues: the natural fallback for work that can happen later — a payment that cannot be charged now is queued, not dropped.
- Caches: the natural fallback for reads — serve the stale cached product page when the catalogue service is unreachable, and count how often you do.
- Load balancers and gateways: enforce edge timeouts and rate limits there so per-service configuration cannot exceed them; see API Gateway.