ArchitectureGENERALRUNTIME-SPECIFICSCALE-SPECIFIC

Failure Propagation

A slow database becomes timeouts, becomes exhausted workers, becomes an outage in endpoints that never touched the database.

What actually happens How to build it

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

How does one degraded dependency take down parts of the system that do not use it?

The requirement

The database got slower for ten minutes. The whole API went down, including endpoints that only read from cache. Nobody can explain the second part.

The obvious build

Each endpoint depends on what it depends on. If the database is slow, database-backed endpoints are slow; the cached ones are unaffected. The blast radius is the dependency graph.

Why it breaks

The dependency graph is not the failure graph. Endpoints that share nothing at the logical level still share the connection pool, the worker set, the event loop and the instance's memory.

How it breaks in production

The dependency graph is not the failure graph. Endpoints that share nothing at the logical level still share the connection pool, the worker set, the event loop and the instance's memory.
Slow is worse than down. A dependency that fails in 1 ms returns capacity immediately; one that answers in 8 seconds holds a worker for 8 seconds. Availability metrics stay green while capacity disappears.
The arithmetic is unforgiving. Little's Law: concurrency = arrival rate × service time. At 200 requests per second, service time moving from 50 ms to 2 s takes required concurrency from 10 to 400. If you have 50 workers, the other 350 requests are queueing, and every endpoint queues behind them.
Then the health check times out because it waits on the same saturated resource, the load balancer removes the instance, and its traffic lands on instances already at their limit (Health Checks: Startup, Readiness, Liveness).

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

Failure propagates through shared bounded resources, not through logical dependencies. The four that matter in a backend: the connection pool, the worker or thread set, the runtime's execution capacity, and memory (Resource Limits).
Step one: a dependency slows. Step two: handlers that call it hold their worker and their pooled connection for the full duration instead of releasing it (Connection Pools).
Step three: with arrival rate unchanged, in-flight concurrency rises until the bounded resource is fully occupied. From that moment every request queues, including ones that would have completed in a millisecond (Queueing: Why Systems Get Slow Before They Get Broken in Observability & Performance).
Step four: queued requests exceed the caller's timeout, so clients retry, adding load to a system that is already past capacity (Cascading Failure).
Step five: readiness checks that touch the same resource fail, instances are pulled, and the remaining instances receive the same total traffic with less capacity. This is the point at which a degradation becomes an outage.
The runtime shapes step three but does not change the story: a thread-per-request server runs out of threads, an event-loop runtime runs out of concurrency budget or lags on CPU-bound work, and a worker-process model runs out of workers (Backend Runtime Models).

The arithmetic of saturation

SIMPLIFIEDOne instance, one uniform service time, no variance. Real traffic has a distribution and the tail saturates earlier than this arithmetic suggests — the model understates the problem rather than overstating it (Percentiles: Which One, and How Many Users Is That? in Observability & Performance).

This is not a metaphor and it is not complicated. Little's Law says the number of requests in flight equals arrival rate times service time. Arrival rate is set by your users; service time is set by your slowest dependency. Concurrency capacity is a number in a configuration file.

Once required concurrency exceeds the bound, the excess queues, and the queue is shared. That is the whole mechanism: the moment a shared bound is reached, every request pays for the slow dependency, including the ones that never call it.

arrival rate      200 rps        (users, unchanged)
service time       50 ms  -> 2 s  (database degrades)

concurrency needed = 200 x 0.05 =  10   healthy
                     200 x 2.0  = 400   degraded

workers available                =  50

  -> 50 in flight, 350 queued and growing
  -> queue is shared: a 1 ms cache-only endpoint waits behind it
  -> queued requests exceed client timeouts -> clients retry
  -> readiness probe (also queued) times out -> instance removed
  -> its traffic moves to instances already at 400-needed / 50-available

The path from slow dependency to total outage

Each hop below is a place a control could have stopped the spread, and the response column names it. Notice that the first two responses are limits rather than fixes: they do not make the database faster, they stop its slowness from becoming yours.

The last row is the one that turns a bad ten minutes into a full outage, and it is the most avoidable. A readiness probe that depends on the degraded dependency removes every instance at once.

One slow database, five hops

Trigger	Symptom	Cause	Response
Database p99 goes from 50 ms to 2 s	Database-backed endpoints slow down	The dependency itself — a lock, a plan change, a failover	Timeout per query, shorter than the request budget (Timeouts)
Handlers hold pooled connections for 2 s	Pool wait time becomes non-zero	Connections are a bounded shared resource	Separate pools per workload; alert on pool wait time, not just utilisation (Connection Pools)
In-flight concurrency exceeds worker count	Cache-only and static endpoints slow down too	Every request now queues for the same workers	Per-dependency concurrency limits so one dependency cannot occupy all workers (Bulkheads)
Queued requests exceed client timeouts	Request rate rises while success rate falls	Clients retry; load amplifies (Retry Storms)	Retry budgets, jitter, and load shedding at the edge (Backoff and Jitter, Backpressure)
Readiness probe queues and times out	All instances removed; total outage	The probe shares the saturated resource	Probe process health, not dependency health; never fail every instance for one shared dependency (Health Checks: Startup, Readiness, Liveness)

Bulkheads: what the shape looks like when it holds

SCALE-SPECIFICPermit counts are the whole design and they are workload-specific: size them from measured healthy concurrency plus headroom, and re-check when traffic shape changes. Copying numbers from another service is how a bulkhead becomes an outage on a normal day.

The fix is not to make dependencies reliable; it is to make their unreliability local. Give each dependency class its own bounded slice of concurrency, and a slow dependency can consume its slice and nothing more.

The cost is honest and worth stating: reserved capacity sits unused when everything is healthy, and the sum of the slices must be sized against what the downstream can actually serve. You are buying containment with utilisation (Defence in Depth is the same trade in a different domain).

UserLLMAgentToolDataDecisionHumanGuardrail

How to build it

Most important first.

Bulkhead the shared resources. A separate connection pool or semaphore per dependency class means the slow one can only consume its own share. This is the single highest-value control here (Bulkheads).
Bound concurrency explicitly per dependency, so "how many requests can be waiting on payments at once" is a number you chose rather than a consequence of how many workers you happen to have (Unbounded Concurrency, Resource Limits).
Timeouts everywhere, and shorter than your caller's. A timeout longer than the upstream's means you hold resources for a client that has already given up (Timeouts).
Fail fast when saturated. Shed load at the edge with 429 or 503 rather than accepting work you cannot do. A rejected request costs almost nothing; a queued one costs a worker (Backpressure).
Circuit-break the dependency so that once it is clearly unhealthy, calls fail immediately instead of each one paying the full timeout (Circuit Breakers).
Separate liveness from readiness, and both from dependency health. An instance that cannot reach the database is not necessarily an instance that should be killed — if all of them go, the outage is total (Health Checks: Startup, Readiness, Liveness, Liveness vs Readiness in Cloud & Infrastructure).
Degrade rather than fail where the product allows it: serve a stale cache entry, omit a section, return a partial response with a flag (TTL and Expiry).
Rehearse it. Make a dependency slow in a staging environment and watch what actually saturates. It is almost never what the team predicted.

What can go wrong

Failure modes

Retries with no budget, amplifying load on a dependency that is failing because of load (Retry Storms).
A timeout set higher than the upstream proxy's, so your handler is doing work for a connection that no longer exists.
Readiness probes that call the database, so a database blip removes every instance simultaneously.
Bulkheads sized by guesswork so that the sum of all pools exceeds what the database can serve, which moves the saturation into the database instead of preventing it.
A circuit breaker whose half-open probe admits enough traffic to re-saturate the recovering dependency immediately.
Load shedding implemented after the expensive work — rejecting a request only once you have already paid for it (Authenticate First, or Rate-Limit First?).
The cache saving you until it does not: a mass expiry during a degradation sends every request to the struggling database at once (Cache Stampede).

What can race

Everything queued at the moment of saturation times out at roughly the same time, so all clients retry together — a synchronised burst on a system that is already over capacity (Thundering Herd in Concurrency & Parallelism).
Circuit breakers across many instances open and close in near-unison because they observe the same dependency, producing traffic that oscillates instead of recovering (Backoff and Jitter).

Security

Saturation is a denial-of-service surface. Any unauthenticated endpoint that consumes a shared resource — a search, an export, an image resize — can be used to exhaust it for every other caller (Rate Limiting).
Controls that fail open under load are the ones that disappear during exactly the incident an attacker created. Decide the direction in advance (Defence in Depth).
Degraded modes must not degrade authorization. Serving a stale cached response is acceptable; serving it without checking who is asking is not (Object-Level Authorization).
Timeout and error behaviour leaks information about internal topology. Keep client-facing errors uniform (Not Leaking Your Internals).

Misreads

"The database was slow, so database endpoints were slow." The mechanism is resource exhaustion, so the affected set is everything sharing that resource — which is normally everything.
"Add more instances." If the bottleneck is downstream, more instances means more concurrent pressure on it. Capacity is not always the fix and is sometimes the accelerant.
"The dependency was up the whole time." Availability metrics measure whether it answered, not how long it held your worker. Slow is the more damaging failure.
"Retries make us resilient." Retries help with independent transient faults. Against a saturated dependency they are load amplification, precisely when load is the problem (Retries).
"Our health checks caught it." If the health check depends on the failing dependency, it removed every healthy instance at once and turned a degradation into an outage.
"Timeouts fixed it." Timeouts bound the damage per request. Without a concurrency limit, a short timeout with retries can produce more load than no timeout at all.

Operating it

How you see it in production

Saturation metrics beat error rates for early warning: pool utilisation and wait time, worker or thread occupancy, event-loop lag, queue depth. All of these move before anything returns an error (USE: Utilization, Saturation, Errors in Observability & Performance).
Pool wait time specifically — how long a handler waits to acquire a connection. Non-zero wait time is the earliest reliable signal that the propagation has begun (Connection Pool Saturation: Waiting in Front of an Idle Database in Observability & Performance).
Per-dependency latency percentiles and in-flight counts, separated from your own service time. The gap identifies the source without guessing.
Watch endpoints that do not use the failing dependency. Their latency rising is the definitive fingerprint of resource-level propagation, and it is the fact that makes the incident explicable.
Alert on client-observed timeouts and on 503s from load shedding as distinct signals. Shedding working correctly should look different from the system falling over.

What changes at 10x and 100x

At 10x arrival rate, the same increase in service time saturates you ten times faster. The window between "a bit slow" and "fully queued" shrinks to nothing, which is why manual response stops being viable and limits must be automatic.
More instances do not help once the shared downstream is the bottleneck — they add concurrent demand to a dependency that is already at its limit, and often make recovery slower (The Bottleneck Moves After Every Fix in Observability & Performance).
At larger scale, propagation crosses services: your saturation becomes your caller's slow dependency, and the same mechanism runs one level up. That is Cascading Failure.

What this costs

Bulkheads reserve capacity that sits idle when everything is healthy. You are buying isolation with utilisation, deliberately.
Aggressive timeouts abandon work that would have completed, converting slow successes into errors. That is usually right and it is not free.
Load shedding means rejecting real users while the system is still partly capable. The alternative is serving nobody, but it will be visible in the numbers and someone will ask.
Circuit breakers add a failure mode of their own: an open circuit that stays open on a recovered dependency, and a probe policy that has to be tuned.
Every one of these controls is a limit someone must size, and a badly sized limit causes an incident on a healthy day.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

GENERALThe mechanism — bounded shared resources plus increased service time — is universal. It applies to a single monolith exactly as much as to a service graph; the monolith just has fewer places to watch.
RUNTIME-SPECIFICWhat saturates differs. Thread-per-request (JVM, Go with a bounded pool, most Python WSGI setups) exhausts threads or workers, and the symptom is a queue at the accept stage. Node exhausts nothing visible — the loop keeps accepting and every in-flight promise waits, so latency rises smoothly with no error until memory or the pool gives out. The second is harder to detect, which is why explicit concurrency limits matter more there (The Node Event Loop).
SCALE-SPECIFICBelow a few requests per second, a slow dependency is just a slow endpoint — there is not enough concurrency to exhaust anything. The propagation behaviour appears when arrival rate multiplied by degraded service time exceeds your bounded resource, which is a number you can compute for your own service.

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

Architecturereliability-patterns circuit-breaker backpressure availability-and-slos

Performancelittles-law queueing saturation connection-pool-saturation use-method worker-pool-saturation tail-latency

Cloudliveness-vs-readiness health-checks failure-domains

Concurrencypool-saturation bounding-concurrency thundering-herd thread-pool-sizing

Domains that do not exist yet

— Distributed Systems — this mechanism generalises to any bounded resource shared across a network, and is where partial failure and backpressure theory live.