Keep-Alive and Connection Reuse
Reusing a connection removes a handshake from every request — and introduces a timeout you must coordinate with every hop.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What does reusing a connection actually save, and what new failure does it create?
The service handles a high request rate from a small number of callers, and we want the per-request cost to be the request rather than the connection.
Open a connection, send the request, read the response, close. Simple, obviously correct, and one request never interferes with another.
Every request pays a TCP handshake and a TLS handshake before a single byte of application data moves. At a high request rate, most of your CPU is cryptography for connections that live for one request.
- Every request pays a TCP handshake and a TLS handshake before a single byte of application data moves. At a high request rate, most of your CPU is cryptography for connections that live for one request.
- Connections in
TIME_WAITaccumulate on whichever side closes; at high churn a client can run out of ephemeral ports and start failing to connect at all. - Outbound calls from your service to a dependency show latency that scales with call count rather than with work, because each call reconnects (Calling Something You Do Not Control).
- Once you do enable keep-alive, a new intermittent failure appears: sporadic
ECONNRESETon the first byte of a request, on perhaps one call in a few thousand, with no pattern in the payload.
What is actually happening
- HTTP/1.1 connections are persistent by default. After a response completes, the connection stays open and the next request is written to the same socket — one at a time, in order (Parsing HTTP).
- What is saved is per-connection setup: the TCP handshake, the TLS handshake and its asymmetric cryptography, and TCP's slow start, which means a fresh connection is also slower for large responses until its congestion window grows.
- Both ends have an idle timeout: how long a quiet connection is kept before being closed. Every hop has one — client library, load balancer, your server, the dependency — and they are usually different numbers.
- The race is structural. A connection is closed by one side while the other is writing a request onto it, because "idle" was decided independently at each end. The writer sees a reset for a request the reader never saw.
- A server can also close after a maximum number of requests or a maximum lifetime, which is how a fleet behind a load balancer gets rebalanced at all — permanent connections mean permanently uneven distribution (Load Balancing, From the Backend's Side).
- Outbound, connection reuse is a pool: your HTTP client keeps N idle sockets per destination host. Concurrency above N queues, which is a limit most people discover during an incident (Unbounded Concurrency).
- HTTP/2 changes the shape: many concurrent streams multiplex over one connection, so reuse is no longer serial and one connection can carry your entire request rate — which also concentrates all of it behind one TCP connection's head-of-line blocking.
What a handshake costs, per request
The argument for reuse is not subtle: without it, every request pays for a TCP handshake, then a TLS handshake with asymmetric cryptography, before your server has seen a single byte of the request. With it, those costs are amortised across every request on the connection.
The relative shape is what transfers: connection setup is a fixed cost paid per *connection*, and application work is paid per *request*. Reuse changes which of those two numbers your traffic multiplies.
The idle-timeout ordering rule
keepAliveTimeout, headersTimeout); Gunicorn calls it keepalive, Nginx keepalive_timeout, and Go exposes it via Server.IdleTimeout. The ordering rule is the same everywhere because it comes from the protocol, not the server.This is the practical takeaway of the lesson, and it is one line: the hop nearer the client should give up on an idle connection first. If your origin closes idle connections after 5 seconds and the load balancer in front of it holds them for 60, the balancer will keep dispatching requests onto sockets your server has already closed.
The symptom is distinctive once you know it: a small, steady rate of connection resets or 502s, worse at low traffic (because connections go idle more often), unrelated to payload, and unchanged by any application deploy.
# load balancer idle_timeout = 60s # your server keepAliveTimeout = 5s # closes idle sockets after 5s # result: for 55 seconds the balancer believes it has a # usable connection that the origin has already closed. # Every request dispatched in that window may reset.
# load balancer idle_timeout = 60s # your server keepAliveTimeout = 75s # comfortably longer than 60 headersTimeout = 80s # must exceed keepAliveTimeout # result: the balancer always decides when a connection # ends. The origin never closes one out from under it.
The race window is the difference between the two timeouts, and it belongs to whichever side does not know the connection is gone. Making the downstream timeout strictly longer moves the decision to the side that is dispatching, which is the only side that can avoid writing into a closed socket.
Outbound reuse is a pool, and pools have a bottom
The inbound side is mostly configuration. The outbound side is where connection reuse becomes an application concurrency limit, usually without anyone deciding it: your HTTP client keeps a fixed number of sockets per destination host, and request N+1 waits.
This is worth making explicit, because it is a bulkhead with a default value. Set it deliberately and it protects a dependency and bounds your own concurrency; leave it at a library default and it will be discovered during an incident as unexplained latency (Bulkheads).
1import { Agent, request } from 'undici'2 3// Every one of these is a production decision, not a tuning knob.4const agent = new Agent({5 connections: 32, // max sockets per origin -> your concurrency6 // ceiling to this dependency7 keepAliveTimeout: 10_000, // how long an idle socket is kept8 keepAliveMaxTimeout: 60_000,9 connect: { timeout: 2_000 }, // TCP+TLS setup budget10})11 12async function chargeCard(body: unknown, correlationId: string) {13 const res = await request('https://payments.example.com/charges', {14 method: 'POST',15 dispatcher: agent,16 headersTimeout: 3_000, // waiting for the response to START17 bodyTimeout: 10_000, // waiting for the response to FINISH18 headers: {19 'content-type': 'application/json',20 'idempotency-key': correlationId, // makes ONE retry safe21 },22 body: JSON.stringify(body),23 })24 return res25}The two timeouts are different questions — "did it start answering?" and "did it finish?" — and a single timeout value cannot express both. The idempotency key is what turns the retry story from "hope" into a rule (Idempotency Keys).
How to build it
Most important first.
- Order the idle timeouts so that the side closer to the client closes first: your server's keep-alive timeout should be longer than the load balancer's idle timeout in front of it, so the balancer retires the connection rather than discovering yours already gone.
- Configure the outbound HTTP client explicitly: keep-alive on, a maximum number of sockets per host, and a maximum idle time. Defaults differ wildly between libraries and are rarely right for a service.
- Make outbound retries aware of this failure: a connection-level reset on an idle-reused connection before any bytes were processed is safe to retry once. A reset mid-response is not, unless the operation is idempotent (Retries, Idempotency in Backends).
- Cap connections per destination, and treat that cap as your concurrency limit to that dependency — it is a bulkhead whether or not you called it one (Bulkheads).
- Set a maximum requests-per-connection or a maximum connection age on the server when you are behind a load balancer, so new instances receive traffic (Rolling Deployments).
- During shutdown, stop advertising keep-alive and close idle connections before the process exits, so in-flight requests finish and no new ones arrive on a dying connection (Graceful Shutdown).
What can go wrong
- Timeout inversion: the upstream idle timeout is longer than the downstream one, so the upstream keeps sending requests onto connections the downstream has already closed. Symptom: a low, steady rate of resets that no code change explains.
- Pool exhaustion on outbound calls: requests queue waiting for a free socket, and the added latency looks like the dependency got slower (Connection Pool Exhaustion).
- Uneven load after a deploy: long-lived connections pin traffic to old instances, so new ones sit idle while old ones are hot.
- Half-open connections after a network partition: one side believes the connection is alive and writes into a void until a TCP-level timeout, which is far longer than any application timeout you set (Timeouts).
- The mitigation failing: a retry-on-reset rule applied to non-idempotent requests, converting a rare reset into a rare duplicate charge.
- The idle-close race: the server closes an idle connection at the same moment the client writes a request onto it. Unavoidable in principle — every reuse scheme has a window — which is why the ordering rule and a single safe retry exist.
- Two requests dispatched onto the same HTTP/1.1 connection concurrently by buggy client code interleave on the wire and corrupt both.
- Shutdown racing reuse: a process closing connections while a load balancer is still dispatching onto them produces exactly the reset pattern above, concentrated at deploy time (Graceful Shutdown).
- A connection carries no identity. Authentication is per request, and a persistent connection does not mean the same authenticated user — a connection through a proxy can carry requests from many callers (Authentication in a Backend).
- Never cache per-connection state that encodes authorization. Connection reuse plus connection-scoped identity is how one user's data ends up in another user's response.
- Persistent connections are the substrate for request smuggling: the attack works precisely because the next request on the connection may belong to someone else (Parsing HTTP).
- Idle connection limits are an availability control. Without a cap on connections per client, an attacker can hold thousands of idle connections for the cost of almost no bandwidth (Accepting Connections).
- "Keep-alive means the connection stays open forever." It stays open until either side's idle timeout, maximum age or request count says otherwise — four independent reasons to close.
- "Connection reuse makes requests concurrent." On HTTP/1.1 a connection carries one request at a time. Concurrency comes from having several connections, or from HTTP/2 multiplexing.
- "The reset is a network problem." A steady low rate of resets on reused connections is almost always an idle-timeout ordering bug between two hops you own.
- "A reset is always safe to retry." Only if nothing was processed, or the operation is idempotent. Those are different claims and only one of them is about the connection.
Operating it
- Graph requests per connection. Near 1 means reuse is not happening; the cause is a client, a proxy or a
Connection: closesomewhere in the chain. - Graph new connections per second next to requests per second. Connection rate that tracks request rate is money spent on handshakes.
- Count connection-reset errors on outbound calls separately from timeouts and 5xx. Resets clustered at low traffic are the idle-timeout race; timeouts are a different problem entirely.
- Watch outbound pool metrics — sockets in use, sockets idle, requests queued for a socket. Queue depth here is invisible in request logs and shows up only as latency.
- On Linux, a large count of sockets in
TIME_WAITon the client side is direct evidence of connection churn.
- At 10x, reuse stops being an optimisation: TLS handshakes at high connection rates are a meaningful share of CPU, and connection churn starts consuming ephemeral ports.
- At 100x, the per-connection memory across a fleet becomes a capacity number, and connection distribution across instances becomes a load-balancing problem in its own right.
- HTTP/2 collapses many connections into one, which reduces handshakes and concentrates risk: one connection's congestion or head-of-line blocking now affects every stream on it (The Request Lifecycle).
- Persistent connections cost memory and file descriptors while idle. A service with many low-rate clients pays to keep connections that are almost never used.
- Long idle timeouts maximise reuse and maximise the window for the close race. Short ones reduce the race and reduce the benefit.
- Pinning traffic to existing connections is efficient and defeats load balancing during scale-out. A maximum connection age trades a little handshake cost for even distribution.
- HTTP/2 reduces connection count and makes a single connection a shared fate for every request on it.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- PROTOCOL-SPECIFICHTTP/1.1: persistent by default, one request at a time per connection, so concurrency equals connection count. HTTP/1.0 required
Connection: keep-aliveto opt in. HTTP/2 multiplexes streams over one connection, so reuse is automatic and the limiting number becomes max concurrent streams; HTTP/3 over QUIC removes TCP head-of-line blocking between streams but keeps the single-connection concentration. - CLOUD-SPECIFICManaged load balancers impose their own idle timeout, and it is usually the one that matters — your application keep-alive setting only decides who closes first. The default value and whether it is configurable differ by provider and product, so the rule "origin timeout longer than balancer timeout" has to be checked against the actual number, not assumed.
- FRAMEWORK-SPECIFICOutbound defaults vary enormously: Node's
undici-basedfetchpools connections by default while the olderhttp.requestneeded an explicit keep-alive agent; Python'srequestsreuses only within aSession, so code that callsrequests.get()directly opens a fresh connection every time and pays a handshake per call.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.