Discovering Services: The Registry Is a Distributed System Too

Ask how out of date a discovery answer can be and most people name one number — the health-check interval. There are four, and they compose additively, not by taking the maximum.

Instance to registry. The instance heartbeats every H seconds and the registry evicts after missing K of them. A crashed instance stays listed for up to K·H seconds. With the common 30s/3 configuration that is 90 seconds before anything else even begins.

Inside the registry. The registry is replicated. If it is eventually consistent, one replica may not yet know about an eviction another has recorded. If it is consensus-backed, a follower serving a stale read can lag the leader. Either way there is a propagation term.

Registry to client. Clients either poll on an interval or hold a watch. A poll interval of 30s adds up to 30s. A watch is much better — push, not pull — but a watch that silently dies degrades to "never updates", which is the worst possible failure and is invisible without an explicit staleness metric.

Inside the client. The resolved set is cached, DNS answers are cached (often past their TTL, and JVM defaults historically cached forever), and connection pools hold open sockets to addresses resolved minutes ago. A pool that never re-resolves will keep talking to a decommissioned instance until its connections break.

Add a realistic set of numbers — 90s eviction, 5s replication, 30s poll, 60s connection reuse — and a terminated instance can receive traffic for roughly three minutes. That number is worth computing for your own stack, because everyone underestimates it and the estimate is usually the first layer alone.

This is the part Architecture’s pattern write-up does not need and you do. A registry is shared mutable state read by every process in your fleet, which makes it a distributed data store, which means it faces the same choice as any other (CAP: What the Theorem Actually Says).

A consistent registry — etcd, ZooKeeper, Consul with consistent reads — gives every client the same view and a totally ordered history of membership changes. During a partition, the minority side cannot serve reads that are guaranteed fresh and typically refuses. The consequence is severe and worth stating plainly: if discovery is unavailable, every service in the fleet loses the ability to find every other service, all at once. A registry outage becomes a total outage, even though every instance is healthy and every network path between them works.

An available registry — Eureka, a gossip-based membership, a DNS-based scheme — keeps answering during a partition with possibly-stale data. It never takes the fleet down, and in exchange it will sometimes hand out addresses of instances that are gone, and sometimes omit instances that are fine.

The Eureka design contains the insight worth stealing: self-preservation mode. If the registry suddenly stops hearing from a large fraction of instances, the far more likely explanation is that *the registry’s* network broke, not that half your fleet died simultaneously. So it stops evicting. Prefer stale entries — which cost a failed connection and a retry — over mass eviction, which costs an outage. The general rule: when a failure detector’s inputs fail en masse, doubt the detector, not the fleet.

The single most valuable design rule in this lesson: a client that cannot reach the registry should keep using the last set of addresses it had. Not fail. Not return an empty list. Keep the last known good configuration and carry on.

The reasoning is asymmetric in a way that makes the choice easy. If the registry is down and the instances are up, using stale data works perfectly. If the registry is down and some instances are also down, using stale data costs you failed connections to those instances — which your client already has to handle, because instances die between lookups anyway. There is no scenario in which failing closed is better, and one enormous scenario in which it is catastrophic.

The corollaries follow: never let the discovery client return an empty list from an error; write the last good set to local disk so a restart during a registry outage still works; and treat "resolved set is empty" and "could not resolve" as different conditions with different handling. Fleets have been taken down by a registry blip precisely because those two were conflated.

This is the same instinct as Graceful Degradation: Which Dependency Is Actually Critical: when a control-plane dependency fails, the data plane should keep running on its last instructions.

Because every layer above is lagging, the registry’s idea of "healthy" is always behind reality. So the client needs its own, closer-in health signal, and it already has one: the outcome of the requests it just sent.

That is what passive health checking, outlier detection and per-endpoint circuit breaking are for. The client observes that endpoint 3 has returned five consecutive connection errors, ejects it from its own local pool for thirty seconds, and re-admits it tentatively afterwards. This reacts in seconds where the registry reacts in minutes, and it requires no coordination with anything (the pattern itself is Architecture’s; the point here is *why* it is mandatory rather than optional).

The division of responsibility is clean once stated: the registry supplies the candidate set; the client decides which candidates are usable right now. Systems that rely on the registry alone spend their incident time waiting for eviction; systems that rely on client-side ejection alone lose track of newly added capacity.

One warning about the other direction. A *deep* health check — one that fails when a shared downstream dependency is unavailable — makes every instance report unhealthy simultaneously, and the registry dutifully evicts your entire fleet for a problem that made it only partially degraded. Health checks should report "can this instance serve" and not "is the whole system happy" (Correlated Failure: The Independence Assumption Is Usually False, Cascading Failure: When the Response to Failure Causes More Failure).

DNS is the oldest service-discovery system and it is still the most common, because it needs no client library. It is worth being precise about what it cannot do.

It has no per-instance health signal — a record is present or absent, with removal governed by TTL rather than by liveness. TTLs are advisory and widely ignored: resolvers round them up, stub resolvers cache independently, and some runtimes have historically cached forever regardless. It carries no weights, no metadata and no explicit failure signal, and negative caching means a record removed and re-added may be unreachable for longer than either change suggests. A short TTL mitigates some of this and multiplies query load.

None of this makes DNS the wrong choice — for stable, coarse-grained endpoints it is exactly right, and the operational simplicity is worth a great deal. It is the wrong choice for fast-changing sets of instances, which is why platforms that autoscale aggressively layer something else on top and use DNS only to find *that*.