NetworkingKUBERNETES-SPECIFICCLOUD-SPECIFIC
Service Discovery in Operation
Instances appear and disappear continuously, so callers cannot hold addresses. Discovery is a registry plus a health signal plus a propagation delay — and the delay is where the incidents are.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The production question
How does a caller find a healthy instance of a service whose instances are being created and destroyed all day?
The problem
Addresses are no longer stable. Every deploy, scale event, node failure and eviction changes the set of instances, and every caller needs a current answer without being redeployed.
What teams do first
Put the addresses in configuration. When they change, update the config and restart the callers — deployments are not that frequent.
How it breaks
Deployments are that frequent, and they are not the only source of change: autoscaling, evictions, node replacement and crashes all rewrite the set without anyone deploying anything (Reconciliation: The Loop Under Everything).
How it breaks in production
- Deployments are that frequent, and they are not the only source of change: autoscaling, evictions, node replacement and crashes all rewrite the set without anyone deploying anything (Reconciliation: The Loop Under Everything).
- A configuration change that requires restarting every caller turns one service's routine rollout into a fleet-wide event, which is the coupling discovery exists to remove.
- Static addresses have no health dimension. An address that is listed but not serving is worse than no address at all, because callers keep sending work to it.
- The set is never instantaneously correct anywhere. Whatever mechanism you use, there is a window in which a caller has an address that is already gone, and connections to it fail with a refusal or a reset (Draining: Stopping Without Dropping).
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Every discovery system is the same three parts: a registry (which instances exist), a health signal (which of them should receive traffic), and a resolution path (how a caller turns a name into an address). Systems differ in where each part lives, not in whether it exists.
- Registration is either self-registration — the instance announces itself and heartbeats — or third-party registration, where the platform registers on the instance's behalf because it already knows when it started and stopped.
- In Kubernetes the registry is the Endpoints or EndpointSlice objects for a service, populated by a controller from pods matching the selector, filtered by readiness. The resolution path is cluster DNS plus a node-local dataplane that rewrites the destination (Probes: Readiness, Liveness and Startup).
- On a cloud load balancer the registry is the target group, the health signal is the load balancer's own health check, and resolution is a single stable address — the caller never learns the instances at all.
- Client-side discovery gives the caller the whole list and lets it choose, which enables smarter balancing and means the caller now holds state that can go stale.
- Every layer of this is eventually consistent. A removed instance is removed from the registry, then from each dataplane, then from each client-side cache, each on its own schedule (Apply Is Not Running).
Registry, health, resolution
Every discovery system decomposes into these three parts. Naming them separately is what lets you locate a failure: a stale answer is a resolution problem, an instance that should not be listed is a health problem, and an instance that never appeared is a registration problem.
UserLLMAgentToolDataDecisionHumanGuardrail
Where the list lives, and how long a removal takes to matter
CLOUD-SPECIFICDeregistration delay, health-check thresholds and refresh intervals are per-provider and per-library settings with different defaults everywhere. The structure — registry, then dataplane, then client cache — is the same on all of them.
The operationally important column is the third one. Every mechanism removes an instance eventually; they differ in how many caches sit between the registry and the packet, and each of those caches is a window in which traffic still arrives at something that is gone.
| Mechanism | Where membership lives | How a removal propagates | What goes wrong |
|---|---|---|---|
| Cluster DNS + dataplane | EndpointSlices, filtered by readiness | Controller updates, then every node's dataplane, then any client DNS cache | Established connections are unaffected and keep using the old instance |
| Cloud load balancer | Target group, filtered by its own health check | Health check failures, then deregistration delay, then removal | The health check disagrees with the app's readiness definition |
| DNS records directly | Zone records | Only as fast as the TTL, and slower for resolvers that ignore it | Unusable for fast failover; caches outlive the change (DNS in Production) |
| Client-side list | In every caller process | A refresh interval per caller, plus whatever the client library caches | One caller with a broken refresh keeps a stale list indefinitely |
| Service mesh | Mesh control plane, pushed to sidecars | Control-plane push to each proxy | A control-plane outage freezes membership everywhere at once |
| Static configuration | A config file | A deploy of every caller | Nobody remembers every caller; one is always missed |
The failures are in the seams
Nearly every discovery incident is a timing or an ownership problem rather than a broken registry. These are the ones worth recognising on sight.
Discovery failures and what they actually are
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Rolling update in progress | Connection refused or reset, briefly, on every deploy | The process stopped before the registry removed it | Deregister first, then wait, then stop (Draining: Stopping Without Dropping) |
| Caller resolved once at startup | One caller keeps failing after everything else recovers | A cached address in a long-lived client or connection pool | Bound connection lifetime and re-resolve; check the client library's caching (DNS in Production) |
| Readiness always returns success | Traffic sent to instances that cannot serve | The health signal does not reflect readiness to serve | Make readiness mean something specific to this path (Probes: Readiness, Liveness and Startup) |
| Readiness checks a shared dependency | Every instance leaves the registry at once | A shared failure is evaluated identically by every replica | Consider degraded readiness rather than total removal |
| Connection reuse | Traffic heavily skewed towards a few instances | Discovery resolved once per connection, and connections are long-lived | Balance per request at L7, or bound connection lifetime (Operating a Load Balancer) |
| Selector does not match | Healthy instances, empty membership, nothing served | Labels and selector diverged — a manifest error, not a runtime one | Compare the selector to the actual labels (Reading a Broken Workload) |
| Discovery control plane down | New instances never receive traffic; existing ones keep working | Membership is frozen at its last known state | Treat the discovery layer as a tier-one dependency with its own alerting (An Alert Should Demand Action) |
How to do it properly
Most important first.
- Make readiness the single source of truth for membership, and make sure it means what you think it means — a pod that is ready but cannot serve is worse than one that is honestly unready (Probes: Readiness, Liveness and Startup).
- Assume staleness on the caller side: retry idempotent requests on connection failures, with a bound and with jitter, because a stale address is a normal event rather than an exception (Retries in the backend view).
- Remove an instance from the registry before stopping it, and give the propagation time to happen. That ordering is the whole of graceful removal (Draining: Stopping Without Dropping).
- Know where the caller's address cache is. A long-lived connection pool, a client library that resolves once at startup, or a runtime that caches DNS answers indefinitely will all outlive the registry's view (DNS in Production).
- Prefer one discovery mechanism per environment. Two overlapping ones — cluster DNS and a mesh, or a mesh and a client-side list — means two registries with different propagation delays and a failure mode nobody owns.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Blast radius if this is wrongEveryone
One testEveryone
What contains it
Discovery is a shared dependency of every caller, so a registry or resolution failure affects all traffic between services at once, not one workload. What contains it in practice is caching: callers holding a recent answer keep working through a short control-plane outage, which is the same caching that causes staleness the rest of the time.
What can go wrong
Failure modes, including of the mitigation
- Connections refused during every rollout because the address was removed after the process stopped rather than before.
- A caller that resolved a name once at startup and has been holding the answer ever since, sending traffic to instances that were replaced days ago.
- Readiness that always passes, so the registry lists instances that cannot serve and the health signal is decorative.
- A registry outage that becomes a total outage because callers cannot resolve anything — the discovery layer is a shared dependency with the widest possible blast radius.
- Long-lived connections that pin a caller to one instance forever, so a "balanced" service is not balanced at all and a drained instance keeps serving (Operating a Load Balancer).
- Two mechanisms disagreeing, so traffic follows one path in some cases and another elsewhere, with no single place to look.
Misreads this invites
- "Discovery means callers always have the current list." It means they converge on it. There is always a window, and designing for that window is the operational skill.
- "A DNS name means round-robin balancing." It means resolution. With connection reuse, one resolution can pin a caller to one instance for its entire lifetime (Operating a Load Balancer).
- "Readiness only matters during deploys." Readiness is the health input to the registry at all times, which is why a readiness probe that checks a shared dependency can empty a service's membership entirely.
- "Service discovery is a Kubernetes feature." It is a requirement of any dynamic fleet. Kubernetes provides one implementation; a target group and a health check provide another.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
How you know it worked
- The registry's membership for a service matches the set of ready instances — checked directly, not inferred from the deployment status.
- Connection errors at rollout time are flat rather than spiky. A spike per deploy is the propagation window, and it is measurable.
- A request made through the service address reaches an instance you expect, from inside and outside the network path.
- Traffic distribution across instances is roughly what you intended; a heavily skewed distribution usually means connection reuse rather than a discovery fault.
How you get back
- Discovery configuration is a deployable like any other: restoring the previous service or selector definition reverts membership, and the registry converges without restarting the instances themselves.
- What does not roll back is the caller-side cache. Callers holding a stale list keep holding it until their own TTL or connection lifetime expires, no matter what the registry says.
- A selector change that empties a service's membership takes effect immediately and completely, which makes it one of the fastest ways to cause a total outage and one of the fastest to reverse.
What to automate, and what stays human
- Automate registration and deregistration entirely. Any process that requires a human to add or remove an instance from a load balancer will be wrong within a day (Toil).
- Automate the ordering: deregistration before shutdown, as part of the platform rather than as something each service implements (Service Templates).
- Keep the health definition owned by the service team. What "ready" means is a product decision about which requests can be served, not a platform default (Probes: Readiness, Liveness and Startup).
What this costs
- Server-side discovery through a load balancer is simple for callers and adds a hop and a shared component whose failure is everyone's failure.
- Client-side discovery removes the hop and gives better balancing, and moves state and staleness into every caller — including callers written in languages where nobody has checked the caching behaviour.
- A service mesh gives uniform discovery, retries and observability at the cost of a sidecar per pod, another control plane to operate, and a new class of failures between the application and the network (Scoring Operational Complexity in the cloud view).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICEndpointSlices populated from readiness, cluster DNS and a node-local dataplane are Kubernetes. A VM fleet uses a load balancer target group with the load balancer's own health check, so instances are never addressed directly and the propagation delay lives in the load balancer rather than in a DNS cache. A PaaS hides the registry entirely and gives you one hostname.
- CLOUD-SPECIFICRegistration timing, health-check semantics and deregistration delay differ by provider and by load balancer type, and those differences decide how long a removed instance keeps receiving traffic. The parameter name is not portable even where the concept is.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
Domains that do not exist yet
- — Distributed Systems — membership, failure detection and why a registry can never be simultaneously correct everywhere.