Services: A Stable Address Over Moving Pods
Pod IPs change every time a pod is replaced. A Service is a name and address that keeps meaning "the currently ready pods for this workload", updated continuously as that set changes.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How does a caller reach a workload whose instances are replaced on every deploy, restart and node failure?
Every pod gets an IP, and every pod loses it when it is replaced. Callers need something that does not change, that routes only to instances currently able to serve, and that keeps working through a rollout that replaces the entire fleet.
Have callers resolve pod IPs and connect to them directly — from an environment variable, a config file, or a service registry the application writes to itself.
A pod replaced during a rollout takes its IP with it. Any cached address becomes a connection to nothing, which surfaces as errors that clear on retry and are therefore easy to dismiss.
- A pod replaced during a rollout takes its IP with it. Any cached address becomes a connection to nothing, which surfaces as errors that clear on retry and are therefore easy to dismiss.
- A registry the application maintains has its own staleness window, and that window is widest during exactly the events that cause churn: deploys, node failures, scale-downs.
- Health becomes the caller's problem. Without a shared notion of readiness, every caller has to decide independently which instances are worth talking to (Probes: Readiness, Liveness and Startup).
- The set of addresses is different in every environment, so the config that names them is the least-tested input you have (Artifact Plus Configuration).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A Service has a selector over pod labels. A controller keeps the matching, ready pods in an endpoint list; the list is the live membership, and readiness is what gates entry into it.
- The Service gets a stable cluster-internal IP and a DNS name. Cluster DNS resolves the name to that IP for the lifetime of the Service, regardless of what happens to the pods (DNS in Production).
- Routing happens on the client's node, not at a central proxy: a per-node component programs the node's packet-handling rules so that connections to the Service IP are rewritten to one of the current endpoints. There is no hop through a load balancer, and there is no single point of failure to scale.
- Balancing is per-connection and effectively random. It is not aware of load, latency or request cost, which matters for long-lived connections: an HTTP/2 or gRPC client opens one connection and pins to one pod until something closes it (Keep-Alive and Connection Reuse).
- A headless Service —
clusterIP: None— skips the virtual IP entirely and returns the pod addresses from DNS. That is what clients needing per-pod addressing use, and it is the foundation of StatefulSet identity (StatefulSets: Identity, Storage and Order). - Service types stack:
ClusterIPis internal only,NodePortalso exposes a port on every node,LoadBalanceradditionally asks the cloud provider for an external load balancer (Load Balancers as Infrastructure is the primitive it provisions).
Selector in, endpoints out
The whole object is a query over labels plus a port mapping. Everything that goes wrong with Services goes wrong in one of those two places.
1apiVersion: v12kind: Service3metadata:4 name: checkout5 namespace: shop6spec:7 type: ClusterIP8 selector:9 app: checkout # must match the pod template labels, not the Deployment name10 ports:11 - name: http12 port: 80 # the port callers use: checkout.shop.svc.cluster.local:8013 targetPort: 8080 # the port the container actually listens on14 protocol: TCPCallers reach checkout.shop.svc.cluster.local. A short name resolves within the caller's own namespace, which is why cross-namespace calls need the longer form. If targetPort does not match the container's listening port, every request is refused by a completely healthy pod.
How a request actually gets to a pod
Worth tracing once, because the surprising part is that there is no proxy in the middle. The routing decision happens on the node the caller is running on, using rules a per-node component keeps in sync with the endpoint list.
That design is why Service routing does not become a bottleneck and why it cannot do anything request-aware: by the time the packet leaves the calling node, the destination pod has already been chosen for the whole connection.
Choosing how a workload is reachable
The types are not a progression from worse to better; they answer different questions about who needs to reach the workload and from where.
Who needs to connect to this workload, and from where?
when Only other workloads in the cluster call it — the default and the right answer for most internal services.
cost Not reachable from outside; debugging from a laptop needs a port-forward or a jump pod.
when The client needs individual pod addresses — client-side balancing, or a stateful system whose members address each other (StatefulSets: Identity, Storage and Order).
cost No stable virtual IP; the client owns balancing and must handle membership changes itself.
when Something outside the cluster must reach it and you have no ingress or cloud load balancer — mostly local clusters and bare metal.
cost A port on every node, an awkward port range, and no TLS termination or routing.
when The workload genuinely needs its own external address — a non-HTTP protocol, or a dedicated ingress point.
cost One cloud load balancer per Service: its own bill, its own certificate, its own health check configuration (Certificates as an Operational Object).
when HTTP or gRPC traffic from outside, alongside other services sharing one entry point.
cost An ingress controller to operate, and routing rules that are now a shared object (Getting Traffic Into the Cluster).
How to do it properly
Most important first.
- Address workloads by Service DNS name everywhere, and never let a pod IP into configuration.
- Make readiness mean "can serve requests right now", because it is the membership predicate for this list. A probe that is too permissive routes traffic to pods that cannot serve; one that is too strict removes healthy capacity (Probes: Readiness, Liveness and Startup).
- Check
targetPortagainst the port the container actually listens on. A Service whose ports do not line up produces connection refused with a completely healthy pod behind it. - For gRPC and other long-lived connections, do not rely on connection-level balancing. Use client-side balancing over a headless Service, or a proxy that balances per request.
- Expect a short window after a pod is removed where in-flight connections still exist. Handle the termination signal, stop accepting new work, and finish what you have (Draining: Stopping Without Dropping).
- Use one
LoadBalancerService per externally exposed thing only if you want one cloud load balancer per thing. Usually you want one ingress in front of many Services (Getting Traffic Into the Cluster).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A Service change applies to all traffic for that workload instantly, with no rollout to stop; containment is review before apply and an endpoint-count alert after it.
What can go wrong
- Selector does not match the pod labels — most often after a rename — so the endpoint list is empty and every call fails immediately. The pods are perfectly healthy.
- Every pod fails readiness at once, usually because readiness depends on a shared dependency. The endpoint list empties and the Service becomes a black hole even though nothing crashed (Probes: Readiness, Liveness and Startup).
- Traffic imbalance under long-lived connections: a few pods carry most of the load and new pods added by scaling receive almost nothing, because existing clients never reconnect (Hot Keys: When Aggregate Metrics Hide a Saturated Node is the analogous shape in caching).
targetPortandcontainerPortdisagree, giving connection refused that looks like an application crash.- Cross-namespace calls using the short name, which resolves within the caller's namespace and silently reaches the wrong workload or nothing at all.
- A
LoadBalancerService per service, producing a per-workload cloud bill and a per-workload certificate to renew (Renewal: Automating the Thing That Expires).
- "A Service is a load balancer." It is a stable address plus a membership list, with connection-level distribution. It does not do health-aware balancing, retries, or anything request-level.
- "The Service proxies my traffic." Routing rules are programmed on the calling node; there is no central process in the path to become a bottleneck.
- "Round-robin means even load." Distribution is per connection. With persistent connections, even distribution of connections can be very uneven distribution of work.
- "An empty endpoint list means the Service is broken." It usually means no pod is passing readiness — the Service is faithfully reporting that there is nothing safe to route to.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The Service's endpoint list contains exactly the pods you expect, and its size changes as pods become ready and are removed.
- During a rollout the endpoint set turns over completely with no change in client-visible error rate.
- Request distribution across pods is roughly even — and where it is not, you can explain why in terms of connection lifetime rather than being surprised by it.
- DNS resolution for the Service name works from a pod in the calling namespace, using the fully qualified name.
- A Service is small, declarative and independently revertible: reapply the previous definition and the endpoint list recomputes immediately.
- The subtle part is that a Service change is instant and cluster-wide, with no gradual rollout. A wrong selector takes effect everywhere at once, so review it like a config change rather than like a code change (A Config Change Is a Production Change).
- Automate the Service alongside its Deployment from one template, so labels and selectors cannot drift apart (Service Templates).
- Automate an alert on a Service with zero ready endpoints, which is one of the highest-signal, lowest-noise alerts a cluster can produce (Alert on Symptoms, Not on Causes).
- Keep readiness semantics human. What "ready" means for a given workload is a design decision, not a template default.
- Node-local routing removes a central proxy and its failure mode, and gives up per-request balancing, retries and observability at that layer. Getting those back means a proxy or a mesh, which is another system to operate.
- Readiness as the membership predicate is elegant and couples routing to probe correctness — a bad probe is now a traffic problem, not just a status problem.
- A stable virtual IP hides pod churn from callers and hides which pod served a request, which is why per-pod labels on metrics matter more than people expect (Label Sets That Survive a Year).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICLabel selectors, readiness-gated endpoints and node-local routing are Kubernetes' mechanism. Outside it the same job is done by a cloud load balancer with a target group whose membership is managed by an autoscaling group, or by a service registry such as Consul that instances register with — both of which put a real proxy in the data path where Kubernetes does not.
- SIMPLIFIEDOmits topology-aware routing, session affinity,
ExternalNameservices and service meshes. A mesh in particular changes several statements here by putting a per-pod proxy in the path and adding request-level balancing and retries.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.