The question this answers
Why did one slow dependency take down five services that were all healthy a minute ago?
None, and that is the lesson: a system with positive feedback in its failure response has no stable degraded mode. Below a load threshold it is fine; above it, the system moves away from equilibrium rather than back toward it, and it will not recover on its own even after the original trigger is removed.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
Each node knows only that its own calls are slow and its own workers are busy. A cannot distinguish "B is slow" from "the network to B is slow" from "B is fine and my own thread pool is exhausted so my requests never reach B". Crucially, A cannot see that its retries are a significant fraction of B’s load — from A’s perspective it is issuing a reasonable number of requests to an unreasonably slow dependency, and every other caller believes exactly the same thing.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The loop, one step at a time
Cascading failure is not "many things broke". It is one specific mechanism, and it is worth walking through slowly because every step is individually reasonable.
B slows down. Not down — slow. A GC pause, a cold cache, a slow query, one degraded replica. A’s threads wait on B. A synchronous call holds a worker for the whole duration, so B’s latency is converted directly into occupied workers at A. A saturates. With a pool of 200 workers and B taking two seconds instead of fifty milliseconds, A’s capacity has silently dropped by a factor of forty. A times out and returns errors — to requests that never reached B, and to requests unrelated to B. A’s clients retry. Each retry is new load. Load on A rises, which means more workers waiting on B, which means more load on B, which makes B slower.
That last arrow is the whole lesson. The system’s response to the failure — retrying — increases the quantity that caused the failure. This is positive feedback, and a system with positive feedback does not settle into a degraded steady state; it runs away. The trigger and the cause become separate things: B’s GC pause lasted 400ms, and forty minutes later the outage is sustained entirely by retry traffic, with B long since recovered and instantly re-saturated by the backlog every time it comes back.
Latency becomes occupancy, and occupancy is capacity
The step that surprises people is the second one, so it is worth making arithmetic. A service with 200 worker threads and a mean handling time of 50ms serves roughly 4,000 requests per second. The same service with a mean handling time of 2,000ms serves 100. Nothing about A changed. No code was deployed, no instance was lost. A dependency got slower and A’s capacity fell by 97.5%.
This is Little’s Law read in the uncomfortable direction: concurrency equals arrival rate times residence time, so with concurrency fixed by the pool size, an increase in residence time must be paid for by a decrease in throughput. Performance owns the law; what matters here is the consequence — a slow dependency is a capacity loss, not a latency problem, and it hits requests that have nothing to do with that dependency, because they are queued behind workers that do.
This is also why [[crash-vs-slow]] matters operationally. A crashed dependency is easy: calls fail fast, the caller sheds the work, capacity is unaffected. A slow dependency is far more dangerous, because it consumes the caller’s resources while producing nothing. The bad case is not the one that looks worst on a status page.
- Edge / clients — retrying every failure, 3 attempts
- Service A — 200/200 workers blocked on B
- Service C — does not call B — starved behind A
- Service B — the original 400ms GC pause, now buried under retries
- Service D — shares A’s connection pool
- abelieves “B is down”✕ and it is false
- edgebelieves “A is down”✕ and it is false
- bbelieves “traffic has tripled organically”✕ and it is false
- cbelieves “its own dependency is failing”✕ and it is false
Every node above is acting on what it believes. Nothing in the cluster tells the mistaken one that it is mistaken.
Why it does not recover when you fix the trigger
The property that makes cascading failure genuinely hard is metastability: the system has two states, and once it is in the bad one, removing the original trigger does not return it to the good one. B has recovered. Its GC pause ended thirty-nine minutes ago. And the system is still down, because there is now a queue of retries large enough to re-saturate B the instant it accepts traffic.
Restarting A does not fix it either. A restarts, comes up empty and cold, and is immediately handed the entire backlog, which saturates it again in seconds. This is why "we restarted it and it fell over again" is the signature line of a cascade. The system is not failing to recover — it is recovering into a load level that it could not serve even when healthy.
The only way out is to break the loop, which means reducing load below the recovery threshold and holding it there: shed at the edge, cut retry budgets to zero, drain the queue slowly, and bring capacity back gradually. Recovery from a cascade is a load-management operation, not a repair operation. Fixing the trigger is necessary and nowhere near sufficient.
t+00:00 b_gc_pause_ms 412 <- the trigger, over in under a second t+00:20 a_worker_pool_utilisation 1.00 <- capacity gone, no code changed t+00:40 a_5xx_rate 0.34 t+01:00 retry_to_original_ratio 2.9 <- THE signal: 3x the real traffic t+01:30 b_inbound_rps 8400 (organic rps: 2800) t+05:00 b_gc_pause_ms 3 <- trigger fully resolved t+05:00 a_5xx_rate 0.61 <- and it is still getting worse t+39:00 retry_to_original_ratio 3.4 <- the outage is now self-sustaining
Breaking the loop is a design decision made in advance
Every countermeasure is a way of removing one edge from the diagram. Bound the concurrency per dependency so B’s latency cannot consume all of A’s workers — this is the single highest-value change, because it stops step two, and everything downstream of step two disappears with it. Cap total retries with a budget rather than a per-call count, so retries can never exceed a fixed fraction of organic traffic. Shed load at the edge so the arrival rate is bounded by what the system can serve. Fail fast on a dependency already known to be failing, which Architecture owns as circuit-breaker.
The ordering matters when you are already in the loop: shed first, then restore. Restoring capacity into an unshed backlog re-saturates instantly. Bring traffic back in steps — 10%, 30%, 60%, 100% — watching worker occupancy rather than error rate, because occupancy turns before errors do.
And prefer the countermeasures that need no coordination. A per-dependency concurrency limit is a local decision that each service can make alone, which means it still works during a partition. A global retry budget is more effective but needs shared state, and shared state during a cascade is exactly the thing that is unavailable.
- Bounded concurrency per dependency: caps how much of your capacity one slow dependency can consume.
- Retry budget: total retries capped as a fraction of organic traffic, not a per-request attempt count.
- Backoff with jitter: prevents retries synchronising into waves that arrive together.
- Load shedding at the edge: the only mechanism that reduces the arrival rate, and thus the only one that breaks the loop from outside.
- Fail fast on known-failing dependencies, so waiting does not convert into occupancy.
- Gradual capacity restore: 10% → 30% → 60% → 100%, watching worker occupancy, not error rate.
Key points
- A cascade is a positive feedback loop: the response to failure increases the load that caused the failure.
- Latency at a dependency becomes occupancy at the caller, and occupancy is capacity — a slow dependency is a capacity loss.
- The failure spreads to requests that never touch the failing dependency, because they queue behind workers that do.
- Metastability: removing the trigger does not restore the system, because retries now sustain the outage on their own.
- A slow dependency is far more dangerous than a crashed one.
- Recovery is load management — shed first, restore capacity gradually — not repair.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A dependency’s latency rises for any reason: GC, cold cache, a slow query, one degraded replica.
- • Callers holding synchronous connections convert that latency into occupied workers.
- • Caller capacity falls in proportion to the latency increase, with no change to the caller itself.
- • The caller begins timing out, including on requests unrelated to the slow dependency.
- • Clients interpret timeouts as transient and retry, adding load.
- • Added load reaches the dependency, increasing its latency further, and the loop closes.
- • The system leaves the good equilibrium and does not return to it when the trigger is removed.
- • Retries synchronise into waves, so the dependency sees a periodic spike instead of a smooth increase.
- • Connection pools shared between dependencies let a slow one starve calls to healthy ones.
- • Health checks time out under load, so the orchestrator kills healthy-but-busy instances and removes the capacity you had left.
- • Autoscaling adds instances that all connect to the same saturated dependency, increasing load on it.
- • A restart cycles instances into the backlog and they fall over on arrival, producing a crash loop.
- • Unrelated endpoints failing: the operator sees 5xx on endpoints that make no call to the failing dependency, because those requests are queued behind workers blocked on it.
- • Traffic that never happened: the operator sees inbound RPS at the dependency triple with no corresponding rise in real user traffic — the retry-to-original ratio is the signal that names the cause.
- • Recovery that does not stick: the operator restarts the service, watches it come up, and watches it fail again within seconds as the backlog arrives.
- • Trigger already gone: the operator finds the root-cause metric fully recovered thirty minutes before the outage ends, and no new fault to explain the continuing failure.
- • Orchestrator amplification: the operator sees healthy pods being killed and rescheduled during the peak, because liveness probes shared the saturated worker pool and timed out.
- • Scaling made it worse: the operator sees the dependency degrade further immediately after the caller autoscaled, because the new instances added connections to the thing that was already the bottleneck.
- • None of the loop requires coordination — that is why it is fast. Each participant acts locally and reasonably.
- • Breaking the loop with a global retry budget requires shared state, which is a coordination point that is itself under stress during the cascade.
- • Per-dependency concurrency limits are purely local and therefore survive partitions; prefer them as the primary defence for exactly that reason.
- • Coordinated recovery — one operator controlling the ramp rate for the whole tier — beats independent per-service recovery, because independently recovering services collectively re-saturate the shared dependency.
- • Committed data stays committed; a cascade is an availability failure, not usually a durability one.
- • In-flight workflows are left at arbitrary steps across many services, so the reconcile surface after a cascade is unusually large.
- • Any operation retried during the cascade may have executed multiple times, so idempotence assumptions are load-bearing exactly here.
- • Queues and backlogs hold the work; the system’s eventual behaviour depends entirely on whether that backlog is drained gradually or released at once.
- • Detect: watch worker-pool occupancy and the retry-to-original ratio, both of which move before error rate does.
- • Contain: shed at the edge and drop retry budgets to zero. This is the step that breaks the loop; nothing else does.
- • Recover: restore capacity in steps, watching occupancy rather than errors, and drain any backlog at a controlled rate.
- • Reconcile: audit for duplicate effects from retried non-idempotent operations, and for workflows stranded mid-step across services.
- • Verify: confirm retry-to-original ratio is back near its baseline — an error rate at baseline with an elevated ratio means the loop is still primed.
- • Retry-to-original request ratio per dependency — the single most diagnostic signal, and rarely instrumented.
- • Worker-pool and connection-pool occupancy as a fraction of capacity, per dependency, which turns before error rate.
- • Concurrent in-flight requests per dependency, which is the quantity the concurrency limit bounds.
- • Queue depth and drain rate during recovery, to predict the second saturation before it happens.
- • Whether the trigger metric has recovered while the symptom continues — the definitive sign of metastability.
- • Understanding the mechanism helps everywhere there is a synchronous call graph more than one hop deep.
- • It is the highest-value model for a team that keeps having "one service degraded and everything went down" incidents.
- • It is what makes bounded concurrency and retry budgets legible as necessities rather than as tuning knobs.
- • Systems where all inter-service communication is asynchronous through durable queues cascade differently — the loop is damped, and the failure looks like unbounded backlog rather than saturation.
- • Applying cascade countermeasures where there is no feedback path adds latency and complexity for no protection.
- • Aggressive shedding in a system with no cascade risk simply rejects revenue.
- • Asynchronous invocation through a durable queue removes the "latency becomes occupancy" edge entirely: the caller enqueues and returns, and the dependency’s slowness becomes backlog rather than saturation.
- • Bulkheading — a separate pool per dependency — does not stop the cascade but confines it, so a slow dependency cannot consume capacity used by others.
- • Static overload protection: reject above a fixed concurrency rather than queueing, which trades some throughput for a system that cannot enter the bad equilibrium.
- • Reducing the depth of the synchronous call graph is the structural fix; a two-hop graph has far less loop to close than a six-hop one.
One dependency slows down. Watch the failure response become the load.
What people believe, and what is true
The cascade was caused by the service that failed first.
That was the trigger. The cause of the sustained outage is the feedback loop, which is a property of the callers’ retry and concurrency behaviour, not of the trigger.
Once the failing dependency recovers, the system will recover.
Not if the retry backlog is large enough to re-saturate it on arrival. That is metastability, and it is why cascades outlive their triggers by tens of minutes.
A crashed dependency is worse than a slow one.
A crash fails fast and frees the caller’s resources. A slow dependency consumes them while producing nothing, which is what converts one failure into many.
Autoscaling protects against this.
Adding callers adds load to the saturated dependency. Autoscaling into a cascade routinely accelerates it.
Restarting the service clears the problem.
The restarted instance receives the backlog immediately and saturates again. Restart without shedding is how a cascade becomes a crash loop.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
B slows, A’s workers block on B, A saturates, clients retry, load rises, more services fail. The failure response feeds the failure.
Practical
Bound concurrency per dependency so one slow dependency cannot consume all your workers. Budget retries as a fraction of organic traffic. Instrument the retry-to-original ratio and worker occupancy — both move before error rate. In an incident: shed first, restore capacity in steps.
Advanced
Treat it as a control system. The loop gain is roughly (retries per failure) × (fraction of capacity a slow dependency can occupy). Below gain 1 the system damps a perturbation; above it, the system runs away, and the equilibrium it runs to is an outage that no longer needs the trigger. Every countermeasure is an attempt to push the gain under 1: concurrency limits attack the second factor, retry budgets the first. Both factors are usually set by framework defaults nobody chose — the retry count in a client library, the size of a connection pool — which is why the gain of a system is almost never a number anyone has looked at.
Internals
The metastable state persists because the queue has memory. The system has two attractors — a high-throughput one where residence time is low, and a congested one where residence time is high and effective capacity is a fraction of nominal. The barrier between them is asymmetric: a brief load spike is enough to cross into congestion, while leaving it requires holding arrival rate below the *congested* service rate, which is much lower than the nominal one. That asymmetry is precisely why "just turn it back on" fails and why staged restoration works.
Apply it
- 🔧 Find one dependency call in your codebase with a timeout but no concurrency limit, and compute how much of the caller’s capacity that dependency can consume at a 10x latency increase.
- 🔧 Instrument retry-to-original ratio for your busiest dependency and record its normal value, so the abnormal value means something.
- ⚡ During a cascade, the orchestrator begins killing pods because liveness probes are timing out. The pods are healthy but saturated. What do you change, and what is the risk of changing it?
- 💬 One dependency got slower and five services went down. Walk me through the mechanism.
- 💬 The dependency that triggered the outage recovered thirty minutes ago and we are still down. Why, and what do you do?
- 💬 Which is more dangerous to a caller: a dependency that returns errors instantly, or one that takes ten seconds to answer?
- 💬 You have one change to make to prevent cascades. What is it, and why that one?