Kubernetes Networking, Just Enough

Kubernetes fixes the addressing model and leaves the implementation open. Every pod gets its own IP from the cluster’s pod range, and any pod can reach any other pod by that IP directly, across nodes, without NAT — the source address a pod sees is the real pod that sent the packet. Containers within a pod share one network namespace (Network Namespaces): they talk over localhost and must not bind the same port. The node itself can reach every pod, and every pod can reach the node.

This is the opposite of Docker’s per-host private bridge with masquerading (Container Networking). The cost is that pod IPs must be routable between nodes; how that is done is the job of the CNI plugin. Flannel builds a VXLAN overlay; Calico advertises pod prefixes with BGP so the underlying network routes them natively, or uses IP-in-IP; the AWS VPC CNI hands pods real VPC addresses from the node’s elastic interfaces; Cilium does routing and policy in eBPF. Each node typically owns a /24 of pod addresses (its podCIDR), so the routing question reduces to "which node owns this /24" — one route per node.

Pods are ephemeral; a Service is the stable name in front of them. A ClusterIP service gets a virtual IP from the service range (say 10.96.0.12) that is not assigned to any interface anywhere. It works because every node’s kernel rewrites it: kube-proxy watches the API for Services and their EndpointSlices — the current list of ready pod IPs — and programs the node so that a packet to 10.96.0.12:80 is DNAT-ed to one of 10.244.1.7:8080, 10.244.2.9:8080, 10.244.3.4:8080. This is the L4 balancer pattern of Load Balancers: L4 vs L7, built from the NAT: Many Private Hosts Behind One Public Address machinery, running on every node at once.

In iptables mode kube-proxy emits a KUBE-SERVICES chain with one rule per service jumping to a KUBE-SVC-… chain, which picks an endpoint with the statistic module (random, probability 1/n, then 1/(n−1), …) and DNATs; conntrack keeps the choice for the rest of the connection. Thousands of services mean tens of thousands of rules evaluated linearly, and the full table is rewritten on every endpoint change — the known scaling limit. IPVS mode uses the kernel’s IPVS hash tables instead: O(1) lookup, real scheduling algorithms (rr, least-connection, source-hash). eBPF implementations such as Cilium bypass both and do the service translation in a socket-level hook, so a pod’s connect() to a ClusterIP is rewritten before the packet exists.

NodePort reserves a port from 30000–32767 on every node and DNATs it to the service, so any node’s IP reaches it — with a source NAT on the way that hides the client unless externalTrafficPolicy: Local restricts it to nodes that host a pod. LoadBalancer asks the cloud for an external L4 balancer whose targets are the NodePorts (or, with modern CNIs, the pods directly). A headless service (clusterIP: None) has no VIP: DNS returns the pod IPs themselves, for clients such as database drivers that want to choose.

Every pod’s /etc/resolv.conf points at the cluster DNS service (CoreDNS behind its own ClusterIP, typically 10.96.0.10). A service api in namespace shop is api.shop.svc.cluster.local → its ClusterIP; a headless service returns pod addresses; individual pods have records too. The kubelet also writes a search list — shop.svc.cluster.local svc.cluster.local cluster.local — and ndots:5, so that api alone resolves within the namespace and api.shop across namespaces.

The ndots:5 rule has a famous cost: any name with fewer than five dots is tried with each search suffix first. curl https://api.stripe.com from a pod generates api.stripe.com.shop.svc.cluster.local, api.stripe.com.svc.cluster.local, api.stripe.com.cluster.local — three NXDOMAIN round trips to CoreDNS, each often as A and AAAA in parallel — before the real query. A trailing dot (api.stripe.com.) or a lower ndots in the pod spec avoids it; see Following One Lookup Through Every Cache for why the resolver behaves this way.

Services are L4. An Ingress (or its successor, the Gateway API) is a set of L7 routing rules — host and path → service — that an ingress controller turns into configuration for a reverse proxy (Forward and Reverse Proxies) it runs: the nginx ingress controller runs nginx, Contour and Istio run Envoy, Traefik runs itself. The proxy pods are exposed with a LoadBalancer service; they terminate TLS with certificates stored as Secrets and forward to pod IPs, watching endpoints to stay current. Everything from the Load Balancers: L4 vs L7 lesson about health checks, draining and timeout alignment applies unchanged, with the added twist that the ingress proxy’s timeouts, the cloud balancer’s timeouts and the pod’s all have to agree.

By default any pod can reach any pod. A NetworkPolicy is a namespaced allow-list: it selects pods by label and states which ingress sources and egress destinations they may talk to, by pod selector, namespace selector or CIDR, plus ports. Once a policy selects a pod, everything not allowed is denied — the same default-deny-once-you-start principle as Firewalls. Policies are enforced by the CNI in the node kernel (Calico and Cilium do; a bare Flannel does not — the objects are accepted and silently ignored), which is why "we added a policy and nothing changed" is the first thing to check.

The CNI plugin is therefore the pluggable implementation of everything above the pod: it creates the pod’s namespace and veth, assigns the IP, makes pod prefixes routable between nodes (overlay or native), optionally replaces kube-proxy, and enforces NetworkPolicy. Kubernetes specifies the contract — pod IPs, no NAT, Services, DNS, policies — and the CNI decides the packets.