ArchitectureGENERALSCALE-SPECIFICSIMPLIFIED

Cascading Failure

The feedback loops that turn a recoverable degradation into a system that cannot come back up.

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

Why does a system that was only slightly overloaded fail completely — and then fail again the moment you restart it?

The requirement

A ten-minute database slowdown produced a two-hour outage. The database recovered after twelve minutes; the platform did not. Someone has to explain the missing hour and forty minutes.

The obvious build

Overload is proportional: more load means more latency, and when load returns to normal, so does the system. Restarting the affected services will clear it.

Why it breaks

Overload is not proportional past saturation. Beyond the knee, queueing means latency rises steeply for a small increase in load, and every queued request is capacity that produces nothing (Queueing: Why Systems Get Slow Before They Get Broken in Observability & Performance).

How it breaks in production

Overload is not proportional past saturation. Beyond the knee, queueing means latency rises steeply for a small increase in load, and every queued request is capacity that produces nothing (Queueing: Why Systems Get Slow Before They Get Broken in Observability & Performance).
Failure creates load. Timeouts trigger retries, so the load *increases* as the success rate falls. That is a positive feedback loop, and it is the defining property of a cascade (Retry Storms).
Restarting makes it worse before better: a cold instance has empty caches and opens a burst of new connections, so it is slower and more expensive exactly when the dependency is least able to serve it (Cache Stampede).
The system can end up in a state where normal traffic is enough to keep it down — recovery requires less than normal traffic, which no automatic mechanism provides unless you built one.

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

A cascade is a feedback loop, not a chain of dominoes. Something increases load, the increase reduces capacity, the reduction increases load again. Any control that breaks the loop stops it; nothing else does.
Loop 1 — retry amplification. A timeout produces a retry, so a dependency at capacity receives more requests than before. With retries at multiple layers the multiplier compounds along the path (Retries).
Loop 2 — capacity removal. Health checks fail because the instance is saturated, the orchestrator removes or restarts it, and its traffic moves to instances that are already saturated (Health Checks: Startup, Readiness, Liveness).
Loop 3 — cold start. Restarted instances have cold caches and cold pools, so they are slower per request and hit the dependency harder while warming (Startup Time & Cold Start in Cloud & Infrastructure).
Loop 4 — connection storms. A dependency restart drops all connections; every client reconnects simultaneously, and the reconnect burst prevents it from staying up (Connection Pools).
Loop 5 — queue and backlog. Work accumulates during the outage, so the moment capacity returns the backlog is processed at full rate and saturates everything again (Queue Backlog).
Loop 6 — autoscaling lag. Scaling reacts on a timescale of minutes to something that saturated in seconds, and often adds instances after the load has gone (Autoscaling Lag: The Gap Where the Outage Lives in Observability & Performance).
The property that makes recovery hard is metastability: the system has a stable degraded state that persists after the trigger is gone, because the load it generates itself is sufficient to keep it there.

Six loops, and the control that breaks each

This is the failures table that matters most in the module: each row is a self-reinforcing loop, and the response column is the specific control that breaks it. Adding capacity is absent from every response, because capacity does not break a feedback loop.

Recognise the shape from the symptom column. The first row's symptom — request rate up, success rate down — is the single clearest indicator that you are in a cascade rather than a simple overload.

Feedback loops and their circuit-breakers

Trigger	Symptom	Cause	Response
Requests time out	Request rate rises while success rate falls	Retry amplification: failure generates load	Retry budget as a percentage of traffic, plus backoff with jitter (Backoff and Jitter)
Saturated instances fail health checks	Instances removed or restarted during peak load	Capacity removal: the check measures load, not health	Probe process liveness only; never let one shared dependency fail every instance (Health Checks: Startup, Readiness, Liveness)
Instances restart	Recovery attempt is slower than the steady state was	Cold caches and cold pools raise per-request cost	Bring instances back in waves; warm caches before full traffic (Caching in Backends)
A dependency restarts	It comes up and immediately falls over again	Connection storm: every client reconnects at once	Jittered reconnect, connection limits, and a global cap on concurrent connections (Connection Pools)
Backlog accumulated during the outage	Capacity returns, then the system saturates again	The backlog is processed at full rate	Paced drain with a rate limit on consumers (Worker Scaling, Queue Backlog)
Autoscaler reacts to the latency spike	New instances arrive late and increase downstream pressure	Scaling lag plus per-instance connection pools	Cap total downstream concurrency across the fleet; scale on a leading signal (Autoscaling a Backend)

Why the system stays down after the trigger is gone

The hardest part of a cascade to explain afterwards is the gap between "the database recovered at 14:12" and "the platform recovered at 15:50". The answer is that the system entered a metastable state: a degraded equilibrium sustained by load the system itself produces.

Retries keep arrival rate above capacity. Cold caches keep per-request cost above normal. The backlog keeps consumers saturated. Each of those is fed by the others, so normal traffic is now enough to hold the system where it is.

Exiting requires reducing offered load below the degraded capacity — shedding aggressively, pausing consumers, disabling non-essential traffic — and then restoring it gradually. That is counterintuitive during an incident, which is why it belongs in a rehearsed procedure rather than in a judgement call at 03:00.

UserLLMAgentToolDataDecisionHumanGuardrail

A retry budget, and why a retry count is not one

SIMPLIFIEDOne process's budget. In a fleet, per-instance budgets still allow fleet-wide amplification proportional to instance count — a shared limiter or a service-mesh policy is what bounds the total, and the ratio has to be chosen per dependency.

Per-call retry limits look like a control and are not: three retries per call means three times the load when everything is failing, which is exactly when the dependency can least afford it. A budget bounds retries as a share of total traffic, so the amplification factor has a ceiling no matter how bad things get.

Pair it with jitter. Without jitter, everything that failed at the same moment retries at the same moment, and the retry burst is sharper than the original traffic that caused the problem.

Budgeted, jittered retry

1// One budget per dependency, shared across all calls in this process.
2// Refills continuously; retries are only permitted while tokens remain.
3const budget = new TokenBucket({
4  ratePerSec: () => successRate1m() * 0.1,   // retries capped at ~10% of successful traffic
5  burst: 20,
6})
7 
8async function callWithRetry<T>(fn: () => Promise<T>, attempts = 3): Promise<T> {
9  let lastErr: unknown
10  for (let i = 0; i < attempts; i++) {
11    if (i > 0 && !budget.tryTake()) throw lastErr      // budget exhausted: fail, do not amplify
12    try {
13      return await fn()
14    } catch (err) {
15      if (!isRetryable(err)) throw err                  // retryable != safe to retry
16      lastErr = err
17      const base = 100 * 2 ** i
18      await sleep(base / 2 + Math.random() * base)      // full jitter
19    }
20  }
21  throw lastErr
22}

Two distinctions do the work. The budget is shared, so global amplification is bounded rather than per-call. And isRetryable must mean "the operation is idempotent and this error indicates it did not happen" — a timeout is retryable only if the write carries an idempotency key (Idempotency Keys). Treating "retryable" as "safe to retry" is how a cascade acquires duplicate charges as well.

How to build it

Most important first.

Retry budgets, not retry counts. Cap retries as a fraction of total requests (a few percent), so retries can help a single failing request and cannot multiply system-wide load (Retries).
Exponential backoff with jitter on every retry and every reconnect. Without jitter, clients synchronise and the burst repeats at a regular interval (Backoff and Jitter).
Circuit breakers on every remote dependency, so once it is clearly unhealthy, calls fail immediately instead of each one occupying a worker for the full timeout (Circuit Breakers).
Load shedding at the edge, before expensive work. Rejecting 30% of requests cheaply keeps 70% working; accepting 100% and failing them all is the alternative (Backpressure, Rate Limiting).
Prioritise when shedding. Not all traffic is equal — shed background sync and analytics before checkout. This requires request classification, which has to exist before the incident.
Bound and pace recovery. Drain backlogs at a limited rate, bring instances back in waves, and let caches warm before full traffic (Worker Scaling).
Decouple health checks from dependency health, so a shared dependency failing cannot remove every instance at once (Health Checks: Startup, Readiness, Liveness).
Cap total concurrency to the dependency, across all instances, so adding instances cannot increase downstream pressure past what it can serve (Unbounded Concurrency).
Rehearse the recovery. The first time you drain a large backlog should not be during an incident, and the pacing control should be one you have used.

What can go wrong

Failure modes

Retries at three layers — client, gateway, service — multiplying into an order-of-magnitude load increase that nobody designed.
A circuit breaker whose half-open probe sends enough traffic to re-saturate the dependency, producing an oscillation instead of a recovery.
Load shedding that runs after authentication and validation, so the rejected requests still consumed most of their cost.
Backlog processed at full speed on recovery, immediately re-saturating the dependency that just came back.
Autoscaling adding instances that each open a full connection pool, so scaling up increases pressure on the bottleneck (Serverless and Database Connections in Cloud & Infrastructure).
Every instance restarting at once because a deploy or a liveness probe fired fleet-wide, resulting in zero warm capacity.
A dependency that fails fast being treated as healthy by a breaker, so the loop never opens.

What can race

Synchronised timeouts: everything queued at saturation expires at nearly the same moment, so all clients retry together. Jitter is the entire mitigation (Thundering Herd in Concurrency & Parallelism).
Simultaneous cache expiry sends every instance to the database for the same key at once (Cache Stampede, Request Coalescing).
Every instance's circuit breaker opening and closing in unison because they observe the same signal, producing oscillating traffic instead of a recovery.

Security

A cascade is what a volumetric attack is trying to induce; the mechanisms are identical, so load shedding and retry budgets are security controls as much as reliability ones (Rate Limiting).
Any unauthenticated endpoint that consumes an expensive shared resource is a lever for triggering the first loop from outside (The Backend Security Checklist).
Controls that fail open under load vanish at the moment of maximum stress. Decide explicitly which controls are allowed to be skipped when saturated — the answer for authorization is never (Defence in Depth).
Degraded modes must preserve authorization and audit. An incident is exactly when shortcuts get taken and exactly when they are least reviewable (Audit Logs for Privileged Actions in Security Engineering).

Misreads

"We were overloaded, so we need more capacity." Capacity was not the trigger. A cascade generates its own load, and adding instances that each open connections often increases pressure on the bottleneck.
"Restarting will fix it." Restarting empties caches, drops connections and produces a reconnect burst. It is often what keeps the system down.
"Retries make us resilient." Retries help against independent transient faults. Against a saturated shared dependency they are the amplifier (Retry Storms).
"The dependency recovered, so we should be fine." If the loop is still running, normal traffic keeps the system in the degraded state. That is metastability, and it needs pacing to exit.
"Autoscaling handles load spikes." It reacts in minutes to something that saturates in seconds, and it can add downstream pressure while doing so (Autoscaling a Backend).
"This is a distributed-systems problem, so it does not apply to our monolith." A single service with a connection pool and a retrying client has loops 1, 2, 3 and 4 available to it.

Operating it

How you see it in production

The signature to recognise: request rate rising while success rate falls. Organic traffic does not behave that way; retries do. This single chart identifies a cascade faster than anything else.
Instrument retries as a distinct metric from requests, per hop. Without it, amplification is invisible — the dependency sees the total and you see your originals.
Track circuit-breaker state transitions, shed-request counts and queue age as first-class signals, and alert on shed rate: it means the protection is working and something is wrong (Depth Is Not an Emergency; Age Is in Observability & Performance).
Keep an incident timeline with deploy markers. Cascades often start with a change, and the trigger is usually simpler than the mechanism ("What Changed?" — Deploy Markers and the Invisible Deploys in Observability & Performance).
Watch for the metastable signature during recovery: capacity restored, traffic normal, latency still pinned. That means a loop is still running and pacing is required.

What changes at 10x and 100x

Larger systems cascade faster and further: more layers means more places retries multiply, and more services means more shared dependencies that can be the common cause.
At 10x, manual response is too slow — saturation to full outage can be under a minute. The controls have to be automatic and always on, not runbook steps.
At 100x, cascades are a design constraint rather than an incident class: request prioritisation, global concurrency limits and paced recovery are ordinary infrastructure, and capacity is planned with the recovery path in mind, not only the steady state (Capacity Planning: Traffic to Machines in Observability & Performance).

What this costs

Load shedding rejects real users while the system is still partly working. That is the trade, and it must be agreed before the incident because it will not be agreeable during one.
Retry budgets mean some transient failures that a retry would have fixed are surfaced as errors. That is the price of not amplifying.
Circuit breakers cut off dependencies that might have recovered, and their tuning is genuinely hard — a badly tuned breaker causes outages on healthy days.
Paced recovery means the outage lasts longer in wall-clock terms and actually ends. Fast recovery attempts frequently restart the cascade.
All of these controls are inert most of the time and must still be tested, which costs effort with no visible return until the day it returns everything.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

GENERALThe loops need only a bounded resource, a retry and a health check. They occur in a single service with one database as readily as in a service graph.
SCALE-SPECIFICBelow a few hundred requests per second the loops usually cannot outrun a human — you have minutes to react, and a runbook is a reasonable control. Above that the response must be automatic, because saturation to total failure is a sub-minute transition.
SIMPLIFIEDPresented as six named loops for teaching. Real incidents run several at once and interact, which is why the timeline afterwards rarely matches anyone's live hypothesis.

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

Architecturecircuit-breaker reliability-patterns backpressure rate-limiting availability-and-slos

Performanceretry-storms queueing queue-age autoscaling-lag capacity-planning incident-debugging deployment-markers

Cloudstartup-time-and-cold-start autoscaling-signals self-healing failure-domains

Concurrencythundering-herd bounding-concurrency singleflight

Domains that do not exist yet

— Distributed Systems — metastable failure states and the load/capacity feedback analysis that formalises this lesson.
— Testing & Reliability Engineering — game days and fault injection are how you find out whether any of these controls actually work.