The question this answers
How does the load balancer decide which instances are allowed to receive requests, and what should that check actually verify?
Six instances serve the API. One has a full disk, one has lost its database connection pool, and one is mid-deploy. Requests routed to any of them fail. Something has to notice, and it must notice faster than the users do.
Automatic removal of instances that cannot serve, and automatic return once they can — so a partial failure degrades capacity instead of producing a fraction of failed requests forever.
The loop is trivial; the predicate is the whole problem
A load balancer polls each registered target on an interval. Healthy responses within a timeout, repeated a configured number of times, mark the target in service; failures, repeated a configured number of times, take it out. That is the entire mechanism, and it is not where teams go wrong.
What goes wrong is the predicate. GET /health returning 200 OK from the web framework proves that a process is bound to a port and that its router works. It does not prove that the database connection pool is alive, that the disk has space, that a required configuration value was loaded, or that the background thread doing the actual work has not died silently forty minutes ago. A fleet of instances that are all "healthy" by that definition and all failing every real request is an entirely normal incident.
Notice the timing arithmetic hidden in the configuration. An interval of 30 seconds with an unhealthy threshold of 3 means a dead instance keeps receiving traffic for up to 90 seconds. Every request in that window fails. Reducing the interval reduces the exposure and increases check load on the instance and on any dependency the check touches — which is the trade that the next section is about.
Shallow proves nothing; deep is a shared fate
There is a spectrum, and both ends are bad. At the shallow end, the check confirms the process answers. Cheap, fast, no dependencies — and blind to every interesting failure. At the deep end, the check verifies the database, the cache, the message broker, the object store and two internal services. Honest, thorough, and catastrophic: when the cache has a two-minute blip, every instance fails its check simultaneously, the load balancer removes every target, and a degradation that the application could have absorbed becomes a hard outage with zero capacity.
This is the central asymmetry of health checking. A per-instance check should test *per-instance* health — things that can be true of one instance and false of another. A shared dependency being down is not an instance-level fact, and expressing it as one converts a partial failure into a total one. The blast radius of a deep check is the entire fleet, by construction.
The workable design has three layers. A shallow liveness signal that proves the process is not wedged. A readiness signal that checks only what *this* instance needs to serve — its own pool, its own disk, its own initialization state — and reports critical dependencies with a bounded timeout and a cached result rather than a live call on every probe. And separate dependency monitoring that pages a human, because "the cache is down" is an alert, not a routing decision.
| Check | Catches | Misses | Risk if it fails everywhere at once |
|---|---|---|---|
| TCP connect to the port | Process dead, port not bound | Everything else — a wedged process still accepts connections | Low. Rarely correlated across instances |
GET / from the framework | Router broken, process crashed | Pool exhausted, disk full, config missing, worker thread dead | Low, and that is the problem: it almost never fails |
| Per-instance state: pool open, disk space, init complete | The failures that genuinely differ per instance | A shared dependency being down | Low — these facts are naturally uncorrelated. This is the right default |
| Live call to the primary database on every probe | Database unreachable from this instance | Nothing much extra, at high cost | Total. One database blip removes every target simultaneously |
| Every downstream dependency, live | Any dependency problem, immediately | — | Total, and more likely. You have multiplied the failure probability of the fleet by the number of dependencies |
| Cached dependency status with a bounded timeout | Sustained dependency failure, without probe-time coupling | The first seconds of a failure | Bounded. A staleness window in exchange for not amplifying blips |
The correlated removal, drawn
The topology below is the shape of that outage. Three instances across two zones, all healthy, all serving. Each of their health endpoints performs a live query against the shared primary database. The database fails over — a routine, well-handled event lasting perhaps fifteen seconds. Every health check times out. Every target is removed. The load balancer now has no healthy targets and returns 503 to everyone, including for the many endpoints that never touch the database at all.
Fifteen seconds later the database is back, but the fleet is not: each instance needs several consecutive successful probes to return, and some load balancers reset connections or take targets through a slow-start ramp. A fifteen-second database event has produced a two-minute total outage, and the post-mortem will blame the database.
The mitigations are unglamorous and effective. Keep the shared dependency out of the routing predicate. Configure minimum healthy targets or fail-open behaviour so that a load balancer with zero healthy targets sends traffic anyway rather than guaranteeing failure. And design the application to degrade: an endpoint that does not need the database should keep answering while the database is away.
Key points
- The health check is the routing predicate: whatever it asserts becomes the definition of "able to serve".
- A framework-level
200 OKproves a port is bound and nothing else. It is the check that never fails during an incident. - A check that verifies every dependency converts any shared-dependency blip into a fleet-wide outage — its blast radius is 100% by design.
- Check per-instance facts, because only those are uncorrelated across instances. Monitor shared dependencies separately and page a human.
- Interval × unhealthy threshold is the length of time a dead instance keeps receiving traffic. Compute it deliberately.
- Configure what happens when zero targets are healthy; the default is frequently "fail everything".
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The load balancer polls each registered target on an interval, from within the network, with its own timeout.
- • Consecutive failures up to the unhealthy threshold move the target out of rotation; consecutive successes up to the healthy threshold move it back.
- • Removed targets stop receiving new requests but continue being probed, which is how automatic recovery happens.
- • Existing in-flight requests are drained rather than cut, subject to a deregistration delay you configure.
- • Some load balancers implement fail-open: when no target is healthy, traffic is sent to all of them on the theory that a broken guess beats a certain 503.
- • Own the predicate and review it like code. It defines availability more directly than almost anything else you write.
- • Keep a dedicated health path separate from application routes, exempt from authentication and from rate limiting, and cheap enough to call every few seconds forever.
- • Compute and document the detection window — interval × threshold — and make sure it is shorter than your patience for failed requests.
- • Decide the zero-healthy-targets behaviour explicitly rather than inheriting a default that guarantees a total outage.
- • Ensure health checks are excluded from access logs and business metrics, or your traffic graphs and p50 latency become fiction.
- • A check too shallow to fail: every instance reports healthy while every real request errors.
- • A check too deep: one dependency blip removes every target and produces a full outage from a partial one.
- • A check that is expensive: the probe itself consumes a connection from the pool it is testing, and under load the check causes the exhaustion it reports.
- • A slow check that times out under load precisely when the instance is most needed, removing capacity during a spike and worsening it.
- • A check that passes before initialization finishes, so traffic lands on a process with an empty cache and no open pool.
- • A check on the wrong port or path after a refactor: every target unhealthy at deploy, and a rollout that either stalls or takes the service down.
- • Probe volume is targets × load balancer nodes ÷ interval. At a few hundred targets with a short interval this is a meaningful, permanent request rate.
- • A deep check multiplies that rate against the dependency: 300 instances probing a database every 5 seconds is 60 extra queries per second forever.
- • Detection latency is independent of fleet size, so at large scale a fixed detection window means proportionally more failed requests before removal.
- • Very large target groups make health-check state propagation itself a scaling concern; removal is not instantaneous across all load balancer nodes.
- • The health endpoint is reachable from the load balancer and often from anywhere in the network. It must not disclose versions, dependency hostnames, configuration or stack traces.
- • Keep a detailed diagnostic endpoint separate from the routing predicate, and require authentication for it.
/healthzreturnsok;/debug/statusreturns detail and needs a credential. - • Health checks are usually exempt from authentication and rate limiting by necessity — which makes them an unauthenticated endpoint worth keeping trivially cheap.
- • A public health endpoint that reveals whether the database is reachable is a free reconnaissance signal during an attack (Public Exposure, Read With Context).
- • The probe traffic itself is negligible; the dependency load a deep check generates is not, and it runs at all times including at 3 a.m.
- • Failed checks cost capacity: instances removed from rotation are still billed while serving nothing.
- • Detection latency has a direct business cost — every second before removal is a second of requests routed into a broken instance.
- • Health-check logs at fleet scale become a real log-ingestion line item unless explicitly filtered out.
- • Healthy target count over time, against desired count. A slow bleed here precedes most capacity incidents.
- • Health-check failure reasons split by cause: timeout, connection refused, non-2xx. They mean different things and are usually collapsed into one chart.
- • Health-check latency, which rises before failures start and is the earliest available warning.
- • Flap rate per target — an instance oscillating in and out of rotation is worse for users than one cleanly removed.
- • The signal that lies: a green health dashboard, when the predicate is
return 200. It is measuring your web framework, not your service.
- • For a single instance behind DNS with no load balancer, no health check exists and none is needed — the correct answer at that scale is to restart on failure and monitor from outside.
- • Passive health checking: infer health from real request outcomes (consecutive 5xx, connection errors) instead of a synthetic probe. No extra traffic, no separate predicate to keep honest, but it needs real traffic to detect anything.
- • Outlier detection or circuit breaking at the client or service mesh, which ejects a misbehaving instance based on observed error rates rather than a self-report.
- • External synthetic monitoring from outside the network, which catches the whole-path failures an internal probe structurally cannot see — DNS, certificate expiry, the load balancer itself.
- • A deeper check detects more real failures and couples the fate of every instance to every dependency it names.
- • A shorter interval detects failures faster and multiplies probe load on the instance and its dependencies.
- • A generous unhealthy threshold avoids flapping on transient blips and leaves broken instances in rotation longer.
- • Fail-open behaviour prevents the zero-healthy-targets outage and sends traffic to instances that are genuinely broken.
What people believe, and what is true
A thorough health check is a better health check.
Thoroughness in the routing predicate is shared fate. Check what is true of this instance; monitor shared dependencies somewhere that pages a human instead of removing capacity.
All targets healthy means the service is fine.
It means the predicate is passing. If the predicate is return 200, it will keep passing through an outage.
Health checks are instant.
Interval × unhealthy threshold is your detection window, and the default configuration on most platforms leaves a dead instance in rotation for a minute or more.