Circuit Breakers
After a dependency has failed enough, stop calling it: fail fast, protect your own capacity, and probe carefully for recovery.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
When a dependency has been failing for a while, why is continuing to call it worse than refusing to?
When the fraud service is down, checkout should degrade in milliseconds rather than waiting on a timeout for every request.
Keep calling it. Each request gets its own chance, and when the service comes back everything resumes automatically.
Every request now pays the full timeout before failing, so a dependency outage becomes a latency outage across your whole service (Timeouts).
- Every request now pays the full timeout before failing, so a dependency outage becomes a latency outage across your whole service (Timeouts).
- Those waiting requests each hold a worker, so the pool fills with calls that are certain to fail and endpoints that never touch the dependency start failing too (Connection Pool Exhaustion).
- Your traffic keeps arriving at a dependency that is trying to recover, and its recovery is prevented by the load (Retry Storms).
- When it does come back, every accumulated client hits it at once and it falls over again (Thundering Herd in Concurrency).
- The fallback you wrote is never reached, because the code path always ends in a timeout rather than in a fast failure.
What is actually happening
- A breaker is a small state machine wrapped around one dependency. Closed passes calls through and counts outcomes. Open rejects calls immediately without attempting them. Half-open allows a small number of probes to test whether the dependency has recovered.
- The value is fail fast. When the dependency is definitely unwell, a rejection in microseconds is enormously better than a rejection after a full timeout — because the difference is a worker held or released (Bulkheads).
- Trip on a failure rate over a window with a minimum sample count, not on a raw consecutive-failure count. A count-based trigger fires spuriously on low traffic and far too slowly on high traffic; a rate needs a minimum volume so that two failures out of two do not open the breaker.
- Not every error should count. Timeouts, connection failures and 5xx indicate the dependency; 400s and 404s indicate your request and must not trip the breaker, or one bad client input takes the dependency offline for everyone.
- Half-open must be concurrency-limited. If the open period ends and every waiting request probes at once, you have recreated the thundering herd with extra steps — one or a few probes at a time, with the rest still rejected.
- Breakers and retries cover different regimes: retries handle transient failures lasting one attempt, breakers handle sustained failures lasting many. Together they mean "try again briefly, then stop trying" (Retries).
- Breaker state is per dependency, and often per operation. One global breaker means a failing analytics call blocks payments; a breaker per endpoint means one slow method does not condemn the rest (Failure Propagation).
Three states and the transitions between them
The state machine is simple. The engineering is in the transition conditions, and each of the four arrows below is a decision with a wrong answer that is common in production.
The transition worth the most attention is open to half-open, because it is where a recovering dependency is most fragile. Every instance's timer expiring at once, with unbounded probes, reproduces the exact traffic pattern that caused the outage.
A breaker with the details that matter
state and probesInFlight effectively atomic. On a genuinely multi-threaded runtime the same code has a race on both fields and needs atomics or a lock; and on any runtime, sharing breaker state across instances requires an atomic compare-and-set in the shared store (Atomic Operations).Most breaker bugs are not in the state machine; they are in what counts as a failure, how many probes half-open allows, and whether the transition is atomic. The implementation below is deliberately explicit about all three.
The isDependencyFault classifier is the line that prevents the most embarrassing failure mode. Without it, a client sending malformed requests can open the breaker and deny the dependency to every other user of your service.
1type State = 'closed' | 'open' | 'half'2 3class Breaker {4 private state: State = 'closed'5 private openedAt = 06 private probesInFlight = 07 private consecutiveProbeSuccesses = 08 private readonly window = new RollingWindow() // successes + failures over time9 10 async call<T>(op: () => Promise<T>, fallback: () => Promise<T>): Promise<T> {11 if (this.state === 'open') {12 if (Date.now() < this.openedAt + this.openMs) return fallback()13 this.state = 'half' // time to test14 }15 16 if (this.state === 'half') {17 // bounded probes: everyone else keeps getting the fallback18 if (this.probesInFlight >= this.maxProbes) return fallback()19 this.probesInFlight++20 }21 22 try {23 const result = await op()24 this.onSuccess()25 return result26 } catch (err) {27 // ONLY the dependency's own faults count. A 400 is our bug,28 // and letting it trip the breaker denies the dependency to everyone.29 if (isDependencyFault(err)) this.onFailure()30 throw err31 } finally {32 if (this.state === 'half') this.probesInFlight--33 }34 }35 36 private onFailure() {37 this.window.recordFailure()38 if (this.state === 'half') return this.trip() // probe failed: straight back to open39 // rate over a window, with a minimum sample count so that40 // 2-out-of-2 on a quiet minute does not open the breaker41 if (this.window.count() >= this.minSamples &&42 this.window.failureRate() >= this.threshold) this.trip()43 }44 45 private onSuccess() {46 this.window.recordSuccess()47 if (this.state !== 'half') return48 // require sustained success; one lucky probe is not recovery49 if (++this.consecutiveProbeSuccesses >= this.probesToClose) {50 this.state = 'closed'51 this.consecutiveProbeSuccesses = 052 this.window.reset()53 }54 }55 56 private trip() {57 this.state = 'open'58 // jitter so the whole fleet does not probe in the same instant59 this.openedAt = Date.now() + Math.random() * this.openJitterMs60 this.consecutiveProbeSuccesses = 061 }62}Four decisions carry the design: only dependency faults count, tripping is rate-based with a minimum sample size, half-open probes are bounded, and closing requires sustained success. The state here is per-process, which means each instance learns independently — a deliberate trade of reaction speed for having no shared dependency.
What happens while it is open
A breaker that opens and then throws is only half a design. The open state is where you decide what the product does without this dependency, and that decision is a product decision as much as a technical one.
The options below differ in what the user experiences and in what you owe them afterwards. The one that catches teams out is the cached fallback: serving stale data with no indication of staleness is worse than an error in any domain where the number matters.
What does this request do without the dependency?
when The data changes slowly and staleness is tolerable and disclosable.
cost Silently stale answers. Mark the staleness in the response or you have traded an error for a wrong number (TTL and Expiry).
when The dependency enriches rather than enables — recommendations, fraud scoring, personalisation.
cost A second code path with its own correctness and its own tests. For a fraud check, degrading is a risk decision, not a technical one.
when The work must happen and does not have to happen now: emails, webhooks, syncs.
cost The user gets a promise instead of a result, and you now own a backlog (Background Jobs).
when The dependency is genuinely required — payment at checkout.
cost The user cannot complete the action. It is honest, immediate, and does not hold a worker (Status Codes From the Server's Side).
when One request touches several dependencies and only one is broken.
cost Partial responses need a contract that expresses partiality (Partial Failure: When 3 of 5 Succeed in API Design).
How to build it
Most important first.
- Trip on failure rate over a rolling window with a minimum request count, so the decision is statistically meaningful at both low and high traffic.
- Classify errors explicitly. Only failures attributable to the dependency count: timeouts, connection errors, 5xx, and 429 if it indicates sustained overload rather than your own burst.
- Define the open behaviour deliberately — it is the entire point. Serve a cached value, serve a degraded response, enqueue for later, or return a clear error. "Throw whatever the breaker throws" is not a design (Error Boundaries: Three Translations, Not One).
- Limit half-open probes to a small fixed concurrency, and require several consecutive successes before closing so that one lucky call does not reopen the floodgates.
- Jitter the open duration across instances so the whole fleet does not probe simultaneously (Backoff and Jitter).
- Scope the breaker to the dependency and, where operations differ materially, to the operation.
- Expose the state as a metric and as part of the service's own health reporting, so "why did checkout degrade" has an immediate answer (Health Checks: Startup, Readiness, Liveness).
- Decide per-instance versus shared deliberately. Per-instance state is simple and means each instance learns independently; shared state reacts faster and adds a dependency to the thing protecting you from dependencies.
What can go wrong
- A breaker that trips on client errors, so one malformed request pattern takes a healthy dependency out of service for everyone.
- A threshold so sensitive it opens on normal variance, converting brief blips into self-inflicted outages.
- A threshold so lenient it never opens, which is the same as not having one and is much harder to notice.
- Half-open with unbounded probes, so recovery attempts arrive as a wave and knock the dependency back down.
- A breaker open with no fallback defined, so the fast failure is simply a fast 500 — better for your capacity, no better for the user.
- A shared breaker store that becomes unavailable, so the mechanism protecting you from dependency failure fails with the dependency.
- Per-instance breakers on a large fleet, where each instance must independently learn the dependency is down and each pays its own failure quota to do so.
- A breaker that stays open because the probe path is broken independently of the dependency — for example a probe with an expired credential.
- Many concurrent requests observing the failure threshold cross at the same instant and all attempting the transition to open — the counter and the state change must be atomic or the transition happens repeatedly (Atomic Operations).
- Multiple requests entering half-open simultaneously when the open period expires, producing a probe burst instead of a probe (Unbounded Concurrency).
- A probe succeeding while other in-flight calls are still failing, so the breaker closes on partial evidence and immediately reopens.
- Shared breaker state read and written non-atomically across instances, so two instances disagree about the current state for a window (Optimistic Concurrency).
- An open breaker is a fail-open/fail-closed decision. For an authorization or fraud dependency, failing open silently is a security decision made by an exception handler (Fail Open vs Fail Closed in Security).
- If a user can reliably cause failures in a dependency, they can force a breaker open and reach a degraded path with weaker checks. Breaker state should not be steerable by request content.
- Breaker state exposed in a public response reveals internal topology and dependency health to anyone probing (Not Leaking Your Internals).
- "A breaker makes the system more available." It makes the *rest* of your system more available while the dependency is down. Calls to that dependency get worse — instantly and by design.
- "Breakers replace retries." They are for different durations of failure. Retries handle the blip, breakers handle the outage, and neither substitutes for the other (Retries).
- "Open the breaker on N consecutive failures." Consecutive counts behave badly at both traffic extremes. Use a failure rate with a minimum sample size.
- "The dependency being down should trip the breaker" — only if the errors are the dependency's. Counting your own 400s means a bad request pattern can take a healthy dependency offline.
- "Once it is closed again we are fine." Closing after a single successful probe is how you reopen a wound. Require sustained success.
- "A breaker protects the dependency." Its primary job is protecting *you*. The dependency benefits as a side effect, which is real but secondary.
Operating it
- Breaker state per dependency as a time series, plus a counter of transitions. Flapping between states is a signal that the thresholds are wrong.
- Count of calls rejected by an open breaker, separate from calls that failed. These are different user experiences and different capacity stories.
- Time spent open per dependency, aggregated per day. It is a direct measure of how much of your degradation is caused by others.
- Half-open probe outcomes, so you can tell "the dependency is still down" from "our probes are broken".
- Correlate breaker opens with deploys, on both sides. A breaker opening minutes after their release is the fastest available diagnosis (Deploys Are the First Suspect).
- On a small fleet, per-instance breakers learn slowly — each instance must fail its own quota before opening. On a large fleet, that same independence means most instances have already stopped calling before any one of them has done much damage.
- At high request rates the window fills quickly and the breaker becomes responsive; at low rates the minimum-sample requirement means it may effectively never trip, which is why timeouts remain the primary defence for low-traffic paths.
- Shared breaker state reacts fleet-wide and introduces coordination cost and a new dependency, which is a trade worth making only when per-instance learning is demonstrably too slow.
- With many dependencies, per-dependency breakers plus per-dependency concurrency limits become the main structural defence against one failure becoming all failures (Bulkheads).
- A breaker deliberately rejects calls that might have succeeded. During recovery it is wrong on purpose, and that is the cost of protecting your own capacity.
- Thresholds are a tuning problem with no universally correct answer, and they need revisiting as traffic shape changes.
- A breaker adds state, and state adds failure modes — including a stuck breaker, which is an outage caused entirely by your own machinery.
- The fallback is where the real work is. A breaker without a designed degraded path just changes how quickly you return an error.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe state machine and the reasons for it are stack-independent.
- SCALE-SPECIFICBelow a certain request rate a rate-based breaker rarely accumulates enough samples to trip, so timeouts and concurrency limits do the protecting instead. Breakers earn their complexity when a dependency is called often enough that a window fills in seconds.
- FRAMEWORK-SPECIFICLibrary implementations differ in ways that change behaviour materially: whether the window is time-based or count-based, whether half-open probe concurrency is bounded, and whether state is per-process or shared. Read the implementation rather than assuming the canonical three states behave as described here.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — local failure detection, why it is necessarily imperfect, and what a service can conclude from its own observations alone.