The question this answers
One service is failing. Why is the failure spreading, and what actually stops it?
A failure in one component degrades only the functions that depend on it, with a stated maximum blast radius: which user journeys are affected and which explicitly are not. It is a statement about a *specific* failure through a *specific* shared resource, and it holds only for resources you have actually enumerated.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A service knows which of its own dependencies are currently failing and what its own resource pools look like. It does not know its transitive dependency graph, which of its peers share the resource it is about to exhaust, or whether the dependency it is calling is itself waiting on the same database it uses. Blast radius is a global property that no single node can compute, which is why it must be designed rather than observed.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The fault domain is the resource, not the deployment unit
Architecture’s circuit breaker is the mechanism for cutting one bad dependency: how to trip, how to half-open, how to recover. Do not re-derive it. The question here is one level up and applies to a whole system: given forty services and one failure, which of the forty go down with it, and why?
The answer is never "the ones that call it". It is "the ones that share a resource with the ones that call it". Two services that never call each other but run on the same node pool are one fault domain. Twenty services that use different schemas in the same database instance are one fault domain. A service that calls a failing dependency from the same thread pool it uses for everything else has merged its own endpoints into one fault domain.
That last case is the most common and the most surprising in practice. A service has ten endpoints; one of them calls a payment provider that starts taking 30 seconds. Requests to that endpoint occupy threads for 30 seconds each. Within a minute every thread in the pool is parked on the payment provider, and the nine endpoints that never touch payments are returning timeouts. The failure spread through a resource, not through a call.
So containment is an inventory exercise before it is an engineering one: list what is shared — threads, connections, pools, hosts, node groups, databases, caches, network paths, deployment pipelines, on-call attention — and each shared thing is an edge along which failure travels regardless of your architecture diagram.
Every dependency needs a declared criticality and a declared degraded behaviour
Containment requires knowing, in advance, what the system should do when each dependency is gone. That is a per-dependency decision and it has exactly three honest answers: fail (the function cannot proceed and returns an error), degrade (proceed with a reduced or stale result), or defer (accept the request durably and complete it later).
The decision must be made before the incident, because during the incident there is no time and no information. And it must be written where the code can act on it, not in a document — a dependency whose degraded behaviour exists only in a runbook will not degrade at 3 a.m., it will time out.
The classification exposes a common architectural lie. Teams describe recommendations, personalisation, fraud scoring and analytics as "non-critical", and then the checkout code path awaits all four with no fallback. A dependency is optional only if there is code that runs when it is absent. If there is no such code, it is a hard dependency wearing an optional label, and your availability is the product of every dependency’s availability: ten dependencies at 99.9% each gives 99.0%, which is seven hours a month.
The corollary drives the design: reducing the number of dependencies on the critical path buys more availability than making each one more reliable, because the first changes the exponent and the second only changes the base.
| Dependency | Criticality | Behaviour when unavailable | Requires |
|---|---|---|---|
| Payment providertypical | Hard | Fail the checkout with a clear, retryable error | Nothing — but say so explicitly |
| Inventory servicetypical | Degradable | Accept the order, verify stock asynchronously, compensate if short | A compensation path and a customer comms path |
| Recommendationstypical | Optional | Render a static popular-items list | A fallback that is exercised in normal operation |
| Fraud scoringassumption | Policy | Fail open under a value threshold, fail closed above it | A written risk decision, not an engineering one |
Containment across many services: cut the graph, not the edge
A breaker on one call protects one caller from one callee. Containment across a system needs the failure to stop at a boundary that spans many services, and there are only a few shapes that achieve it.
Tiering. Assign every service a tier and forbid a lower tier from being a synchronous dependency of a higher one. Tier-1 (checkout, auth) may not synchronously call tier-3 (analytics, recommendations); if it needs their output, it reads a cached or precomputed value. This is enforceable in CI by inspecting the dependency graph, which makes it one of the few architectural rules that survives contact with a deadline.
Cells. Partition the whole stack — services, data, queues — into independent cells, each serving a slice of users. A failure caused by data or load is confined to one cell, and the blast radius becomes 1/cells by construction rather than by care. The cost is real: cross-cell operations become hard, and you now operate N copies of everything.
Asynchronous boundaries. Replacing a synchronous call with a durable queue converts a dependency outage from an availability failure into a latency increase. This is the single most effective containment move available, and it is why the "is this call synchronous because it has to be?" question is worth asking about every edge on the critical path.
What all three have in common: they change the *shape of the graph*, not the reliability of a node. Containment is a topology property. You cannot buy it by making individual services better.
Failure: recommendation service fully unavailable affected: product page carousel (static fallback), email digest (skipped) unaffected: browse, search, cart, checkout, payments, account mechanism: called async from page render with 80ms budget + static fallback verified: game day 2026-06-11, fault injection, no tier-1 impact Failure: primary user database unavailable affected: EVERYTHING — sign-in, checkout, cart, order history unaffected: static pages, CDN-cached catalogue mechanism: none. This is a single fault domain by design. verified: accepted risk, RTO 12min, reviewed 2026-05-02
Key points
- Failure spreads along shared resources, not along service boundaries — a shared thread pool merges every endpoint that uses it into one fault domain.
- Availability of a synchronous chain is the product of its parts: ten 99.9% dependencies give 99.0%. Removing dependencies beats improving them.
- A dependency is optional only if code exists that runs when it is absent. Labels without fallbacks are hard dependencies.
- Containment is a topology property: tiering, cells and asynchronous boundaries change the graph; better services do not.
- The blast radius of each significant failure should be a written, tested statement — otherwise it is a guess you will test during an incident.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Enumerate shared resources across services: pools, hosts, node groups, datastores, caches, network paths, pipelines.
- • For every dependency on a critical path, declare criticality and the behaviour when it is unavailable: fail, degrade, or defer.
- • Implement that behaviour in code, with a bounded timeout and a fallback that is exercised during normal operation.
- • Cut the graph so that failure cannot cross tiers or cells synchronously, and enforce the rule automatically.
- • Inject each failure deliberately and record the observed blast radius against the intended one.
- • A dependency becomes slow rather than down, so a breaker sized on error rate never trips while every thread parks.
- • A fallback path is never exercised in normal operation and is broken when first needed.
- • The containment boundary is nominal: two "isolated" services share a node pool, a database instance, or a NAT gateway.
- • A transitive dependency reintroduces the coupling you removed — service A avoids B, but calls C, which calls B synchronously.
- • Degraded mode is more expensive than normal mode (cache miss, full scan, extra call) and amplifies the load rather than reducing it.
- • Whole-service brownout from one endpoint: every endpoint times out while only one dependency is unhealthy. In-flight count is pinned at the pool size and the thread dump shows every worker in the same downstream call.
- • Cascading dependency failure: an outage that begins in a tier-3 service ends with checkout unavailable, because a synchronous call was added months ago and nobody re-checked the tier rule.
- • Fallback that fails: the breaker trips correctly, the fallback path executes for the first time in production, and it throws — the error rate does not improve when the breaker opens, which is the diagnostic signature.
- • Shared-infrastructure coupling: two services with no code dependency degrade at the same instant. The correlation points at a shared node pool, a shared database instance, or a shared network path — and no service graph will show it.
- • Containment removes coordination rather than adding it: every synchronous dependency is a coupling that makes two services share a fate.
- • Deferring work to a queue trades coordination for reconciliation — you no longer need both parties available at once, and you now need a path that fixes up state afterwards.
- • Cells reduce coordination sharply within a cell and make anything cross-cell expensive, which is a design constraint, not a detail.
- • Functions that do not depend on the failed component keep their full guarantees — that is the entire claim, and it is only true for resources you actually separated.
- • Degraded functions return results with weaker guarantees (stale, partial, approximate), which must be visible to the caller rather than silently substituted.
- • Deferred work is durably accepted and completes later, so the user-visible failure becomes a delay.
- • Detect: correlate degradation across services that have no code dependency on each other — that correlation is the map of your real fault domains.
- • Contain: separate the pool first. Giving the failing dependency its own bounded pool is usually a smaller change than any architectural fix and stops the bleeding immediately.
- • Recover: let breakers half-open gradually; a full-rate resumption against a recovering dependency reproduces the outage.
- • Reconcile: work that was deferred or degraded must be completed or corrected — reconciliation is part of containment, not a follow-up.
- • Verify: run the failure as a game day and compare observed blast radius against the written one. The gap is the finding.
- • In-flight requests per *dependency*, not per service — this is the number that reveals a pool filling up before the service is unresponsive.
- • Fallback execution rate in normal operation. Zero means untested code on your recovery path.
- • Cross-service degradation correlation, which surfaces shared fault domains that no dependency graph records.
- • Per-dependency error and latency budgets, so a dependency degrading below its declared assumption raises an alert before it takes the caller with it.
- • Systems with many services and shared infrastructure, which is where failures spread furthest and least predictably.
- • Any critical path with more than two or three synchronous dependencies, where the availability product is already the dominant term.
- • Multi-tenant systems where one tenant’s load or data can degrade everyone else’s experience.
- • Small systems where every component genuinely is required: containment machinery adds complexity and contains nothing.
- • When isolation is drawn so finely that utilisation collapses — every pool sized for its own peak means paying for the sum of peaks. See Bulkheads: Buying Independence by Giving Up Utilisation.
- • When degraded modes multiply beyond what anyone can test, and the untested combinations become the incident.
- • Make the dependency reliable enough not to need containment — occasionally right for a single critical dependency, and it does not compose past two or three.
- • Remove the dependency from the critical path entirely, by precomputing or caching what it provides. Strictly better than containing it, where possible.
- • Accept the coupling and state it: a documented shared fault domain with a tested recovery time is more honest than isolation that does not exist.
- • Replicate the dependency per consumer (a copy of the data, a dedicated instance), trading consistency and cost for independence.
Blast radius: which of the forty go down with it
offered: 40/s to /checkout retired: 0.0333/s per thread × 200 threads = 7/s utilisation: 6.00× -> 200 of 200 threads parked every thread is waiting on the provider; the pool is the fault domain
What people believe, and what is true
Microservices give us fault isolation.
They give you deployment isolation. Fault isolation comes from separating resources; forty services on one node pool with one database are one fault domain with forty names.
The recommendation service is non-critical.
Only if checkout runs when it is gone. Awaiting an optional dependency with no fallback makes it a hard dependency regardless of the label.
A circuit breaker contains the failure.
It contains one caller-callee edge. Slow-but-not-failing dependencies often never trip it, and the pool fills anyway. Containment is about the resource, and the breaker is one tool for protecting it.
We can work out the degraded behaviour during the incident.
During the incident you have no information and no time. A degraded path that has never run in production is untested code on your recovery path.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
A failure spreads to everything that shares a resource with it, not to everything that calls it. Contain by separating resources and by having something to do when a dependency is gone.
Practical
For each dependency: bound the timeout, give it its own pool, declare fail/degrade/defer, and implement the fallback so it runs sometimes even when nothing is broken. Then write the blast-radius statement for your top failures and verify it with fault injection.
Advanced
Treat containment as graph surgery. Availability of a synchronous path is the product of its nodes, so the leverage is in shortening the path, not hardening nodes. Tiering forbids edges by policy; cells partition the graph into replicas; asynchronous boundaries convert an availability edge into a latency edge. Each changes the exponent rather than the base — which is why an organisation that keeps adding synchronous hops cannot buy its way back to reliability.
Apply it
- 💬 One endpoint calls a payment provider that slows to 30 seconds. Why do the other nine endpoints start failing?
- 💬 Your critical path has ten dependencies at 99.9% each. What is your availability ceiling, and what is the highest-leverage fix?
- 💬 Two services with no code dependency on each other degrade at exactly the same moment. What do you look for?