The question this answers
How long should I wait before deciding a node is dead — and why is that the wrong question?
A heartbeat detector is eventually strongly complete: a crashed node is eventually suspected by every correct node. It is never accurate — it can suspect a live node at any moment, and no timeout removes that. A phi-accrual detector adds a calibrated statement about *how likely* a suspicion is to be wrong, given the observed message history.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A knows the timestamps at which messages from B arrived, and therefore how long it has been since the last one. That is the entire input. "B is dead" is an inference from silence, and silence is produced by a crash, a network drop, a GC pause, a saturated queue, a slow disk and a paused VM alike.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The timeout you cannot pick
Set the timeout at one second and a routine 1.2-second garbage-collection pause gets a healthy node evicted, a leader failed over, and a rebalance started. Set it at thirty seconds and a genuinely crashed node keeps receiving traffic for half a minute, with every request to it hanging until the client’s own deadline.
The reflex is to search for the right number. There is no right number, and the reason is structural: several different consumers read the same detector, and each has a different cost of being wrong.
A router deciding whether to try another replica pays almost nothing for a false positive — one wasted attempt. It should react in a hundred milliseconds. A leader election pays enormously for a false positive — a term change, a period of unavailability, in-flight work abandoned, possibly a fencing incident. It should wait many seconds. A rebalancer pays even more, because acting wrongly moves terabytes (Rebalancing: A Load Spike You Schedule for Yourself). One timeout cannot serve all three, and whichever value you pick is wrong for at least two of them.
So the fix is not a better timeout. It is to stop making the *detector* decide, and instead have it report a graded signal that each consumer thresholds for itself.
| Consumer | Cost of a false positive | Cost of a slow detection | Sensible threshold |
|---|---|---|---|
| Route this request elsewheretypical | One wasted attempt | A few slow requests | Very low — react fast, be wrong often |
| Eject from the connection pooltypical | Reduced capacity for a moment | Continued errors to a dead peer | Low |
| Trigger leader electiontypical | Term change, brief unavailability, abandoned work | Cluster stalled with no leader | High |
| Re-replicate the node’s datatypical | Terabytes moved for nothing, plus the load spike | Extended time under-replicated | Very high, plus a mandatory delay |
| Page a humantypical | A wasted night | Nobody notices a real outage | High, with sustained confirmation |
Phi accrual: report a number, not a verdict
The accrual failure detector, in the form popularised by Hayashibara and used in Cassandra and Akka, changes the interface. Instead of returning a boolean, it returns a continuously-updated suspicion level.
The mechanism is straightforward. Maintain a sliding window of recent inter-arrival times for heartbeats from B and fit a distribution to it — normal or exponential, both are used. Then, given that it has been Δt since the last heartbeat, compute the probability that a heartbeat would still arrive later than Δt under that distribution, and take:
φ = −log₁₀ P(arrival later than Δt)
The scale is directly interpretable, which is the whole point. φ = 1 means about a 10% chance you are wrong. φ = 2 means 1%. φ = 8 means one in a hundred million. A router can act at φ = 1 and be wrong a tenth of the time, cheerfully, because a wasted retry is nothing. An election can wait for φ = 12. Both read the same detector.
The second benefit is adaptivity, and in practice it matters as much as the grading. Because the distribution is fitted to observed arrivals, a link with 200 ms of jitter naturally develops a wider tolerance than a 1 ms LAN link — without anyone tuning per-link timeouts. During a period of general network slowness, inter-arrival times lengthen, the fitted distribution widens, and φ rises more slowly for everyone. The detector becomes automatically more forgiving exactly when the network is degraded, which is exactly when a fixed timeout would produce a storm of false positives and trigger the failover cascades that turn a slow network into an outage.
t (s) since last beat φ detector says 0.0 0.2s 0.1 healthy 1.0 0.2s 0.1 healthy 2.0 1.4s 0.9 slightly late — router may try elsewhere 2.6 2.0s 2.4 1% chance we're wrong — eject from pool 3.4 2.8s 5.1 GC pause ends here; heartbeat arrives 3.5 0.1s 0.1 recovered; nothing was failed over -- a real crash instead -- 8.0 2.0s 2.4 eject from pool 9.0 3.0s 6.0 stop routing entirely 11.0 5.0s 13.8 one in 10^13 — safe to elect a new leader 15.0 9.0s 31.0 safe to begin re-replication One signal. Each consumer acted where its own cost curve told it to.
What grading does not fix, and what SWIM adds
φ calibrates the inaccuracy; it does not remove it. Two limitations remain, and one of them has a genuinely elegant answer.
Asymmetry. A cannot hear B while B hears A perfectly. A’s φ for B climbs and B’s φ for A stays flat. Nothing in the detector notices, and the resulting disagreement is one of the hardest membership problems to diagnose (Cluster Membership: A Belief, Not a Fact).
The detector cannot tell whose fault the silence is. Is B down, or is the A→B path down? From A’s position these are identical.
SWIM answers the second — and largely the first — with indirect probing. When A’s direct probe to B fails, A does not conclude anything. It asks k other members to probe B on its behalf. If any of them succeeds, A learns that B is alive and the problem is the A–B path specifically. Only if all k indirect probes also fail does A mark B suspect. For a small constant extra cost, this collapses the false-positive rate from a broken link, which is a large fraction of real false positives.
SWIM then adds suspicion with refutation: B is marked *suspect*, not dead, and the suspicion is gossiped. If B is alive and hears about it, B broadcasts a higher incarnation number for itself, superseding the suspicion (Gossip: Epidemic Spread Instead of Everyone Telling Everyone). Only if the suspicion survives a timeout does B become *confirmed dead*. This gives the only party with direct evidence of B’s liveness — B — a chance to speak, and it turns a false positive into a one-round correction instead of an eviction, a rejoin and a rebalance.
The two ideas compose well and are worth stating as a single pattern: before acting on silence, ask someone else, and give the accused a chance to answer.
A heartbeat is not health
Everything above concerns whether messages arrive. That is a weaker claim than "this process can serve requests", and the gap is where a whole family of production incidents lives.
A process can answer heartbeats while being useless: its thread pool is exhausted but a dedicated health-check thread still responds; its disk has gone read-only; its downstream dependency is gone and every real request 500s; its connection pool is saturated but ICMP still works. A liveness signal that only proves the network stack is running is a lie told confidently. Load balancers keep routing to it, membership keeps counting it, and every request it receives fails (Discovering Services: The Registry Is a Distributed System Too).
The obvious correction — make the health check exercise the real request path, including dependencies — creates the opposite and worse failure. When the shared database blips, every instance fails its check simultaneously, and the fleet is evicted in one move. A partial degradation becomes a total outage, caused entirely by the health check (Correlated Failure: The Independence Assumption Is Usually False, Cascading Failure: When the Response to Failure Causes More Failure).
The resolution used by systems that have been burned by both: separate the signals. A liveness check answers "should this process be restarted?" and must not consult dependencies. A readiness check answers "should this instance receive traffic right now?" and may consider local capacity — queue depth, thread pool saturation — but still should not fail on a shared downstream, or you have rebuilt the fleet-wide eviction. Where dependency health genuinely must influence routing, it belongs behind a threshold that requires a *minority* of instances to be failing, which is the same self-preservation instinct described in Discovering Services: The Registry Is a Distributed System Too.
The rule for consumers of a failure detector
Whatever detector you have, the discipline on the consuming side is the part that prevents incidents.
Match the threshold to the cost. Cheap, reversible reactions may act on weak evidence. Expensive, irreversible ones must demand strong evidence and a delay.
Put a delay between suspicion and destructive action. Re-replication and rebalancing should never fire on first suspicion. A configurable "node has been gone for N minutes" gate before data movement is the single most effective safeguard against detector-driven cascades.
Never treat a mass failure signal at face value. If half the fleet goes silent at once, the detector or its network is the likely fault. Above a threshold, stop acting and raise an alarm instead — doubt the detector, not the fleet.
Fence rather than trust. An evicted node does not stop working merely because others believe it is gone. If it must not act, take its authority away at the resource: an epoch, a lease, a fencing token (Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely, Leases: Authority With an Expiry Date). Detection is a hint; enforcement is what makes it safe.
Key points
- A failure detector over an asynchronous network cannot be accurate; it can only be eventually complete.
- One timeout must serve consumers whose costs of being wrong differ by orders of magnitude, so no value is right.
- Phi accrual reports −log₁₀ of the probability that you are wrong, so each consumer picks its own threshold.
- Fitting the distribution to observed arrivals makes the detector automatically more tolerant on jittery links and during network degradation.
- Indirect probing distinguishes a broken path from a dead peer; refutation lets a wrongly-suspected node rescue itself in one round.
- Answering heartbeats is not the same as being able to serve, and a health check that tests shared dependencies evicts the whole fleet at once.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Each node sends periodic heartbeats, or is probed periodically by peers.
- • The receiver records arrival timestamps in a sliding window and fits a distribution to the inter-arrival times.
- • On demand it computes φ from the time since the last arrival and that distribution.
- • Consumers read φ and compare it to their own threshold, chosen from the cost of acting wrongly.
- • Before marking a peer suspect, ask k other members to probe it indirectly.
- • Mark suspect rather than dead, disseminate the suspicion, and allow the peer to refute with a higher incarnation.
- • After a suspicion timeout with no refutation, confirm the failure — and gate any destructive reaction behind an additional delay.
- • A long GC pause, a paused VM or a blocked event loop produces silence indistinguishable from a crash.
- • An asymmetric link makes two nodes disagree permanently about each other.
- • Network-wide slowness raises inter-arrival times everywhere and triggers mass suspicion.
- • A process answers heartbeats while unable to serve any real request.
- • A dependency-checking health probe fails on every instance at once.
- • Heartbeats are sent from a thread that is not affected by the saturation the check is supposed to detect.
- • False failover on a GC pause: a healthy leader is replaced during a four-second pause. The operator sees an election with no crash in any log, a latency spike at the moment of the pause, and in-flight work abandoned for no reason.
- • Asymmetric eviction: A evicts B while B continues serving clients that reach it directly, so writes land in both a member and a non-member. The operator sees data present on one node and missing from the others, with both nodes reporting healthy.
- • Detection slower than client patience: clients have already timed out and retried for thirty seconds before the cluster removes the node. The operator sees error rate lead the membership change, which makes the membership system look like a bystander when it is the bottleneck.
- • Fleet-wide eviction from a deep health check: a shared database blips, every instance reports unhealthy, and the load balancer removes all of them. The operator sees 100% failure caused by a dependency that was only partially degraded.
- • Zombie instance: the process answers every probe and fails every request, staying in rotation indefinitely. The operator sees one instance with a normal health status and a 100% error rate — a combination that no health-based system will resolve on its own.
- • Suspicion storm during network degradation: a fixed-timeout detector marks dozens of nodes suspect simultaneously, and the reactions — elections, rebalancing, re-replication — add load that deepens the degradation.
- • Detection itself needs no coordination; each node forms its own opinion from local observation, which is what makes it fast.
- • Indirect probing is a small, bounded amount of coordination — k messages — that buys a large reduction in false positives. An unusually good trade.
- • Turning a local suspicion into a cluster-wide decision requires either gossip (eventual, cheap, may disagree) or consensus (ordered, expensive) — see Cluster Membership: A Belief, Not a Fact.
- • Because detection is imperfect, any action taken on it must be independently safe. That safety comes from fencing at the resource, not from agreement about the detector’s output.
- • Under network degradation an adaptive detector widens its tolerance and produces fewer false positives, where a fixed detector produces a storm.
- • Under partition, both sides suspect the other and neither is wrong from where it stands.
- • A suspected-but-not-confirmed node continues to be routed to by consumers with high thresholds and avoided by those with low ones — a graceful, staged withdrawal rather than a cliff.
- • A confirmed-dead node that is actually alive keeps serving anyone who can reach it, until something fences it.
- • Detect: track φ per peer as a time series. Recurring peaks that resolve without failover are your GC-pause or jitter signature, and they are invisible if the detector exposes only a boolean.
- • Contain: raise thresholds and pause destructive automation during a suspected network event rather than letting reactions compound.
- • Recover: prefer refutation over eviction-and-rejoin — a node that can speak for itself should be allowed to.
- • Reconcile: after any false failover, check for writes accepted by the old leader after it lost authority (Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely, Split-Brain: Two Nodes, Both Certain They Are In Charge).
- • Verify: confirm the detector’s false-positive rate against reality by counting suspicions that resolved without any corresponding restart or crash.
- • φ (or time since last heartbeat) per peer, as a distribution rather than a threshold crossing. The peaks that never cross are your tuning evidence.
- • Suspicions raised versus failures confirmed. The gap is the false-positive rate and it should be a headline number.
- • Suspicions refuted — how often a node rescued itself. A high number means the thresholds are too tight, not that refutation is working well.
- • Heartbeat inter-arrival distribution per link, which shows immediately whether one timeout can serve links with different characteristics.
- • Detection latency: time from actual process death to confirmed failure, measured with deliberate kills rather than estimated.
- • Health-check pass rate correlated with request success rate per instance. Divergence between them is the zombie signature.
- • Heterogeneous networks — cross-zone, cross-region, mixed link quality — where one timeout cannot fit every path.
- • Clusters with expensive reactions to failure, where a false positive costs a failover or a data movement.
- • Environments with unpredictable pauses: large heaps, oversubscribed VMs, noisy neighbours.
- • Any system where several components consume the same liveness signal for decisions of very different cost.
- • Small, uniform clusters on a reliable LAN, where a fixed timeout is simpler and behaves identically.
- • When consumers ignore the grading and threshold everything at one value — you have paid for a graded detector and rebuilt a binary one.
- • When the fitted window is too short to have seen a real pause, so the distribution is over-confident and φ spikes on the first hiccup.
- • When it creates false confidence: φ = 12 is a statement about message arrival, not about whether the process can serve requests.
- • A fixed timeout with generous margins and a mandatory delay before destructive action. Simple, predictable, and adequate for many systems.
- • Leases, where liveness is a property the node must actively renew — silence removes authority automatically, with no eviction decision to make (Leases: Authority With an Expiry Date).
- • External observation by an orchestrator that knows the process state directly, replacing inference with a source of truth.
- • Client-observed health: let the actual request stream be the signal, ejecting endpoints that fail real requests. Closest to the truth, and it cannot see a node nobody is calling.
- • Deliberately not detecting: route to all replicas and let requests fail fast with hedging (Send a Second Request After p95 and Take Whichever Answers First), avoiding a failure decision entirely.
From alive-or-dead to a suspicion level
What people believe, and what is true
With the right timeout you can tell a crashed node from a slow one.
You cannot. It is the defining impossibility of asynchronous failure detection; a timeout only chooses how often you will be wrong in each direction.
Phi accrual detects failures more accurately.
It quantifies uncertainty rather than removing it. The gain is that each consumer can act at the confidence level its own cost curve justifies, and that the detector adapts to the link.
A node responding to health checks is healthy.
It proves a thread answered a probe. It says nothing about the request path, the thread pool, the disk, or the dependencies.
Health checks should verify dependencies so we catch real problems.
Then every instance fails the check simultaneously when a shared dependency blips, and the entire fleet is removed from service — converting partial degradation into total outage.
Once the cluster agrees a node is dead, it is safe to take over its work.
The node may be alive and still serving. Agreement about the detector’s output is not enforcement; only fencing at the resource prevents the old node from acting.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Heartbeats turn silence into a guess about liveness. A binary detector forces one timeout to serve every consumer; a graded one reports how suspicious a node is and lets each consumer decide.
Practical
Separate liveness from readiness, and keep shared dependencies out of both. Gate expensive reactions — election, re-replication, rebalancing — behind a delay after suspicion. Track suspicions raised against failures confirmed; that ratio is your false-positive rate and almost nobody measures it.
Advanced
φ = −log₁₀ P(a heartbeat arrives later than the current gap), fitted to observed inter-arrival times. A router acts at φ = 1 and is wrong a tenth of the time for the price of one retry; an election waits for φ = 12. The fit is what makes it adaptive: a jittery link earns a wider tolerance automatically, and a degraded network makes the whole detector more forgiving exactly when a fixed timeout would produce a false-positive storm.
Internals
Before acting on silence, ask someone else and give the accused a chance to answer. SWIM’s indirect probe — k peers probe B on A’s behalf — separates a broken A–B path from a dead B for a bounded constant cost, and refutation via a higher incarnation number lets a wrongly-suspected node cancel the suspicion in one round rather than being evicted, rejoining and triggering a rebalance. Both are cheap and both remove failure modes that no amount of timeout tuning can reach.
Apply it
- 🔧 Instrument one peer link with heartbeat inter-arrival times for a day and plot the distribution. Then check whether your configured timeout sits where you thought it did relative to the tail.
- 🔧 Count how many times last month a node was suspected and then recovered without any restart. That is your false-positive rate, and it is usually higher than anyone expects.
- ⚡ A leader is failed over during an eight-second GC pause and the old leader resumes and keeps writing. Trace every mechanism that should have prevented damage and identify which one was missing.
- ⚡ During a network degradation, forty nodes are marked down within a minute, triggering re-replication that saturates the network further. Name the two changes that break the loop.
- 💬 What timeout would you use to decide a node is down, and why is that question badly posed?
- 💬 Explain phi-accrual failure detection and what problem it solves that a fixed timeout does not.
- 💬 Why should a load balancer health check not verify the database connection?