Load Balancers: L4 vs L7
A load balancer either forwards TCP connections it never reads (L4: cheap, protocol-blind) or terminates HTTP and routes requests it understands (L7: smarter, costlier) — and in both cases its health checks, draining and idle timeouts decide what 502 and 504 mean.
The problem
L4: forwarding connections without reading them
A layer-4 balancer works on the transport header: it sees source and destination IP:port and the TCP flags, and nothing above that. When a SYN arrives for the virtual IP (VIP) it chooses a backend, records the 5-tuple → backend mapping in a connection table, and forwards every later packet of that connection to the same backend by lookup. It never terminates TCP: the backend completes the handshake with the client, and the balancer is a stateful packet rewriter.
Two forwarding styles exist. In NAT mode the balancer rewrites the destination address to the backend’s on the way in and the source address back to the VIP on the way out, so replies must return through it (compare NAT: Many Private Hosts Behind One Public Address). In DSR (direct server return) it rewrites only the destination MAC and leaves the IP header alone; the backend, which has the VIP configured on a loopback alias, replies directly to the client. Request traffic goes through the balancer, the far larger response traffic does not — which is why DSR is the choice for video and object serving. Linux IPVS implements both; Google’s Maglev and cloud NLBs are L4 balancers built the same way.
What L4 buys: it is protocol-agnostic (works for a database port, gRPC, a game server), it does no parsing so it forwards at line rate with tiny per-packet cost, and it preserves end-to-end TLS. What it cannot do: route by URL, retry a failed request, or see that a backend is returning 500s, because it never sees a request. And the backend sees the client’s real source IP only in DSR or with the PROXY protocol header (PROXY TCP4 203.0.113.9 10.0.0.5 51234 443\r\n) prepended to the stream in NAT mode.
L7: terminating and routing requests
A layer-7 balancer is a reverse proxy (Forward and Reverse Proxies) with a routing table: it terminates TCP and usually TLS, parses each HTTP request, chooses a backend per request (not per connection), and forwards it over its own pooled connection. Because it sees requests it can route by Host, path, header or cookie, can retry an idempotent request on another backend when one fails, can return its own error page, and can multiplex thousands of client HTTP/2 streams onto a small pool of HTTP/1.1 upstream connections (HTTP/2: Streams on One Connection).
The cost is that it is a full HTTP implementation on the data path: every byte is parsed, TLS is decrypted and re-encrypted (or sent plain inside the trusted network), and the balancer’s own memory, descriptors and CPU become the ceiling. A cloud ALB, nginx, HAProxy in mode http and Envoy are L7; HAProxy in mode tcp and a cloud NLB are L4. Many deployments stack them: L4 in front to spread connections across a fleet of L7 proxies.
System Design decides whether you need an L7 tier and where; this lesson is about what each tier can physically see and therefore do.
| L4 balancer | L7 balancer | |
|---|---|---|
| Reads | IP + TCP/UDP headers | full HTTP request (after TLS) |
| Decision unit | one connection (5-tuple) | one request |
| Terminates TCP/TLS? | no | yes |
| Route by path / host / header | no | yes |
| Retry a failed request | no (cannot see it) | yes, if idempotent |
| Client IP at backend | preserved (DSR) or via PROXY protocol | X-Forwarded-For |
| Cost per byte | near line rate | parse + copy + (re)encrypt |
| Examples | IPVS, Maglev, NLB, HAProxy tcp mode | nginx, Envoy, ALB, HAProxy http mode |
Picking a backend: hashing and connection tables
Round robin rotates through the backend list; weighted round robin skews it; least connections picks the backend with the fewest open connections (the balancer already has the table, so it is free); random-two-choices picks two at random and takes the less loaded, which avoids the herd behaviour of everyone choosing the same "least loaded" server. All of these are stateless with respect to the client: a second connection from the same client may land anywhere, and any state the backend kept about the first is lost.
Hashing fixes that. Hash the source IP, or a cookie, or a request field, modulo the number of backends, and the same client lands on the same backend — until the backend count changes and hash mod n reassigns almost every client. Consistent hashing (a ring, or Maglev’s permutation table) moves only ~1/n of the keys when one of n backends is added or removed. This is the Hash Table you know from DSA: the connection table is a hash map keyed on the 5-tuple, and consistent hashing is what you reach for when the number of buckets changes under live keys.
The connection table is finite state. Each entry costs memory and lives until the connection closes or an idle timer expires; a SYN flood or a long-lived-connection workload can exhaust it, at which point new connections are dropped even though every backend is idle.
1on packet p:2 key = (p.src_ip, p.src_port, p.dst_ip, p.dst_port, p.proto)3 if key in conn_table: # existing connection: same backend, always4 backend = conn_table[key]5 elif p.tcp.SYN: # new connection: choose once6 backend = choose(healthy_backends, policy) # rr | least_conn | hash(key)7 conn_table[key] = backend8 start_idle_timer(key)9 else:10 drop(p) # mid-stream packet for an unknown connection11 forward(p, backend) # NAT: rewrite dst ip; DSR: rewrite dst MAC onlyHealth checks and draining
A balancer only helps if it stops sending traffic to a dead backend. Active checks probe each backend on a schedule — a TCP connect for L4, GET /healthz expecting 200 for L7 — and mark it unhealthy after N consecutive failures and healthy again after M successes. Passive checks (Envoy calls it outlier detection) watch real traffic: a backend that returned five 5xx in a row or timed out is ejected for a cooling period. Active checks catch a backend that is down; passive checks catch one that answers /healthz fine while every real request fails.
The flapping problem: a backend that is overloaded times out the health check, is removed, recovers because it has no traffic, passes the check, is re-added, receives a burst, and times out again. Hysteresis (require 3 failures to eject, 5 successes to readmit), slow-start weighting on readmission, and a panic threshold (Envoy: if fewer than 50 % of backends are healthy, ignore health and send to all — an overloaded fleet is better than an empty one) are the standard defences.
Connection draining (deregistration delay) is how a backend leaves gracefully: the balancer stops sending it new connections or requests, waits until in-flight ones finish or a deadline passes (AWS defaults to 300 s), and only then drops it. Deploys that skip this — or a backend that exits before the balancer’s next health check notices — produce a burst of connection-refused errors that the L7 tier surfaces as 502.
Timeouts: where 502 and 504 come from
An L7 balancer holds two connections per request and has a timer on each. The idle timeout on the upstream side decides how long a pooled connection may sit unused before the balancer closes it. If the backend’s own keep-alive timeout is shorter than the balancer’s, the backend closes the idle connection first; the balancer, unaware, sends the next request into a socket that is being closed, receives a RST or an empty reply, and returns 502 Bad Gateway for a request the backend never saw. Node’s default server.keepAliveTimeout is 5 s; an AWS ALB’s idle timeout is 60 s; the fix is to make the backend’s timeout longer than the balancer’s, always.
The response timeout decides how long the balancer waits for the backend to answer. If the backend is still working when it expires, the balancer closes the upstream connection and returns 504 Gateway Timeout — while the backend keeps computing a result nobody will read, and a client that retries adds a second copy of the same work. A slow database query behind a 30 s request timeout and a 60 s balancer timeout does not time out at the balancer; behind a 60 s request timeout and a 30 s balancer timeout it does, and the 504 is blamed on "the network".
503 Service Unavailable is the balancer saying it has no healthy backend to try. The trio is a standard interview question because each number points at a different failing component: 502 the backend closed or answered garbage, 503 the pool is empty, 504 the backend is alive but too slow for the balancer’s patience.
client ──── LB (idle 60 s, response 60 s) ──── app (keep-alive 65 s, request timeout 30 s) idle: app keep-alive > LB idle → LB always closes first; no reuse of a dead socket response: app request < LB response → app fails first with its own error; LB never has to 504 draining: LB deregistration delay ≥ longest in-flight request → deploys drop no requests
Key points
- L4 forwards connections by 5-tuple lookup and never reads a request; L7 terminates TCP/TLS, parses HTTP and routes per request.
- NAT mode rewrites addresses both ways; DSR rewrites only the MAC inbound and lets backends reply directly — responses bypass the balancer.
- Backend choice: round robin, least connections, random-two-choices, or hashing; consistent hashing keeps ~1/n of keys moving when the pool changes.
- Active health checks find dead backends; passive checks find backends that pass
/healthzbut fail real traffic; hysteresis prevents flapping. - Draining stops new work to a leaving backend and waits for in-flight work; skipping it turns every deploy into a burst of 502s.
- Backend keep-alive timeout must exceed the balancer’s idle timeout, and backend request timeout must be shorter than the balancer’s response timeout.
- 502: backend closed or answered badly. 503: no healthy backend. 504: backend too slow for the balancer.
Why does this exist?
Mechanisms are answers to constraints. Open each question before reading the answer.
▸Why does an L4 balancer need a connection table at all?
Because TCP is a stream: every packet of one connection must reach the same backend, and the balancer has no other way to know which backend that was. The table is the memory of the first decision.
▸Why can an L7 balancer retry and an L4 one cannot?
Retrying needs a request to resend. L4 sees packets of a stream that the client and backend are conducting; L7 holds the parsed request and can send it to another backend before the client sees an error.
▸Why do idle timeouts cause 502s?
A pooled keep-alive connection can be closed by either side. If the backend closes first, the balancer’s next write goes into a closed socket, and the only honest answer to the client is 502.
▸Why consistent hashing?
With hash mod n, changing n reassigns almost every key; every cache or session on every backend becomes cold at once. Consistent hashing bounds the damage to the keys that belonged to the backend that changed.
Load balancer: L4 vs L7
| Press Run to send 12 requests. |
How it fails
What the failure looks like from inside real software.
- Deploy causes a spike of 502s for ~10 s: pods exit before the balancer’s health check interval notices, or without draining, and in-flight connections are reset.
- Intermittent 502 at low traffic, none at high traffic: backend keep-alive timeout is shorter than the balancer’s idle timeout, so only connections that sat idle are affected.
- 504 on a reporting endpoint while the app log shows the query completing 40 s later: balancer response timeout is shorter than the app’s request timeout; retries triple the database load.
- All backends healthy, clients time out: the L4 connection table or the balancer’s own ephemeral ports to the upstream are exhausted;
ss -son the balancer shows tens of thousands ofTIME_WAIT. - Backends flap in and out every minute: health check timeout is shorter than the backend’s p99 under load; the check itself is what an overloaded backend fails first.
- Application rate limiting and audit logs show every request from the balancer’s IP: L4 NAT mode without PROXY protocol, or L7 without reading
X-Forwarded-For.