Load Balancing

A single application server saturates at some request rate — say 1,200 req/s at 100% CPU with p99 climbing past 800 ms — and when it reboots, the service is gone. Adding a second server does nothing until something decides which server each request goes to. That something is the load balancer: it owns the public address, keeps a list of backends, and forwards each request to one of them. Clients only ever see one endpoint.

The balancer sits on every request, so its two jobs are to spread work so no backend is hotter than the others, and to stop sending work to backends that are dead or dying. Everything else — TLS termination, routing by path, rate limiting — is optional and pushes it toward being an API Gateway.

Round robin hands requests out in turn and is right when requests cost about the same. Weighted round robin exists because fleets are rarely uniform — a 16-core box next to 8-core boxes should get twice the share. Least connections exists because requests are not uniform either: one slow report request pins a server, and round robin keeps feeding it while it is busy; least connections routes to whichever backend currently has the fewest in-flight requests. Power of two choices picks two backends at random and takes the less loaded — nearly as good as global least-connections with none of the shared state, which is why fleets of balancers use it.

Consistent hashing answers a different question: not "who is least busy" but "who already has this key". Hash the user id or cache key onto a ring and send all requests for that key to the same backend, so its in-process cache stays warm. When a backend leaves, only its keys move — about K/N of them, not all of them as with hash mod N. The mechanism is its own lesson: Consistent Hashing; the data structure underneath is a Hash Table with a sorted ring on top.

Active checks probe each backend on a timer (GET /healthz every 5 s, unhealthy after 3 failures, healthy again after 2 successes). Passive checks watch real traffic: N consecutive 5xx or connection resets eject the backend for a cool-down. Use both — active checks catch a dead process fast, passive checks catch a backend that answers /healthz fine while every real request times out on a locked database.

The trap: a health check that depends on a shared dependency. If /healthz pings the database and the database stalls for 10 s, every backend fails its check simultaneously, the balancer ejects all of them, and a database blip becomes a total outage — with the balancer returning 503 to everyone even after the database recovers, until checks pass again. Two defences: keep the liveness check shallow (is the process up and able to serve), and use a panic threshold — when more than, say, 50% of backends are unhealthy, ignore health and route to all of them, because a degraded backend beats none.

Sticky sessions pin a client to one backend (by cookie or source IP) so a server that keeps session state in memory keeps seeing the same user. It works until it does not: a scale-in event or a deploy kills the pinned server and the user is logged out; a NAT gateway puts 5,000 corporate users behind one IP and they all land on one backend; autoscaling adds servers that get no traffic because every existing client is pinned elsewhere. Stickiness is a workaround for stateful servers; the real fix is Stateless vs Stateful Services — move the state out and let any backend serve any request.

A layer-4 balancer forwards TCP/UDP by IP and port: it never looks inside the connection, so it is fast, protocol-agnostic and can pass TLS straight through. A layer-7 balancer terminates HTTP: it can route /api/orders to the order pool, read cookies for stickiness, retry an idempotent GET on a different backend, and observe status codes for passive health. L7 costs CPU per request and must hold certificates; most systems run L4 in front for raw throughput and L7 behind it for routing decisions.

You removed the single server and added a single balancer. It is made redundant the same way everything is: more than one, and a way to fail between them. DNS can return several balancer IPs, but resolvers cache records, so a dead IP keeps receiving traffic for the TTL — DNS is a coarse tool. An active–passive pair shares a virtual IP that moves to the standby on failure (VRRP/keepalived), which is seconds of failover with no client change. Anycast advertises the same IP from many locations and lets BGP route each client to the nearest live one — how large CDNs and cloud balancers stay up. Managed cloud balancers hide all of this behind one hostname and are the right default unless you run your own network.

For scaling, this is rung one of the ladder in Scale This System: the balancer buys horizontal capacity for the stateless tier and hands you the next problem — every backend now opens its own connections to a database that has not grown at all.