Where the Time Goes: The Request Timeline

A request is a chain of segments, each waiting on something different. Draw it as a waterfall and label every span; the ones that surprise you are the ones the application never sees. The timeline below is illustrative; the interactive lets you change RTT, loss and connection state and watch which bars grow.

Read it as a budget. The application’s 6 ms is under 1 % of the total; two handshakes are 22 %; one retransmission timeout alone is 28 %; slow-start round trips for the body are a third. Every one of those bars shrinks or vanishes with a warm connection, a shorter path or no loss — and none of them is visible in the server’s own request timer, which starts when the handler runs and stops when it returns.

Connection setup. DNS (Following One Lookup Through Every Cache), one RTT for TCP (The Three-Way Handshake), one for TLS 1.3 — two for TLS 1.2 (The TLS Handshake). All of it happens before the server process is involved, and all of it disappears with a warm pooled connection (Keep-Alive and Connection Reuse, Connection Pooling). A service whose p50 is fine and whose p99 is +3 RTTs is usually a service whose connection pool is too small or whose idle timeout closes connections between requests.

Propagation and serialization. The bytes take ½ RTT each way (Latency: Same Machine to Cross-Continent), plus size / bandwidth to put them on the wire (Bandwidth vs Latency). Packet loss turns one lost segment into either a fast retransmit (~1 RTT, if enough later segments arrive to trigger duplicate ACKs) or a retransmission timeout — the minimum RTO on Linux is 200 ms, and a loss of the last segment of a small response, with nothing behind it to trigger fast retransmit, waits for the full RTO. That is a 200 ms hole in the timeline for one dropped packet (Packet Loss: Duplicate ACKs, Fast Retransmit and the RTO).

Retries at any layer multiply the rest. A client that retries after a 1 s timeout on a request that the server completed at 1.1 s has doubled the server’s work and will now receive two responses (or a duplicate side effect). Tail latency is the sum of the slowest segment on each hop; a fan-out to ten backends waits for the slowest of ten, so a p99 on each becomes roughly the p90 of the whole — retries with jitter, hedged requests and deadlines propagated through the chain are the tools, and all of them cost extra work.

Between the client and the handler are at least five queues, and time spent in any of them is invisible to the handler’s own timer. The client’s send buffer and the network’s router buffers: bufferbloat — oversized buffers on a saturated link hold hundreds of milliseconds of other people’s packets, so your small request sits behind a bulk upload; latency climbs while nothing is lost and no CPU is busy (The Buffer Chain). The load balancer’s queue and its pool of upstream connections: a request waits for a free upstream connection when the pool is smaller than the concurrency (Load Balancers: L4 vs L7).

The server’s accept queue: completed handshakes wait here until the application calls accept(). If the process is busy — blocked on a database call in a single-threaded design, or with every worker occupied — the connection sits fully established, the client thinks it is connected, and nothing reads its request. When the queue overflows (Linux somaxconn / the listen() backlog), new SYNs are dropped or ignored and the client retransmits them at 1 s, 3 s, 7 s — the "connects take exactly 1 or 3 seconds" signature. Then the socket receive buffer, where the request waits for read(), and the application’s own work queue or thread pool (The Thread Pool Server, C10K: Ten Thousand Connections, Then a Million). Finally, the send buffer on the way out: a slow client fills it and write() blocks or the event loop stalls (What Happens When the Receiver Is Slow).

The rule for idle-CPU slowness: find the queue. ss -ltn shows the accept queue depth (Recv-Q on a listening socket) against its limit; ss -tn shows per-connection Recv-Q/Send-Q; nstat -a | grep -i listen counts overflows; balancer metrics show upstream queue time; ping under load shows bufferbloat as RTT that climbs with traffic.

Two TCP optimisations, each sensible alone, deadlock for tens of milliseconds together. Nagle’s algorithm (on by default) holds back a small segment while a previous small segment is still unacknowledged, to coalesce dribbles of tiny writes into fewer packets. Delayed ACK (also default) has the receiver wait — up to 40 ms on Linux, 200 ms on some other stacks — before acknowledging a lone segment, hoping to piggyback the ACK on data of its own.

Now a client does two write()s for one request — a header, then a body — and waits for the reply. The first small write goes out. The second is held by Nagle until the first is ACKed. The server has received a partial request, has nothing to send, and delays its ACK. Both sides wait; the delayed-ACK timer fires after 40 ms; the ACK arrives; Nagle releases the second write; the request completes. Every request pays a fixed ~40 ms for nothing, and the server’s handler time is unchanged. The signature is a latency histogram with a spike at exactly RTT + 40 ms, on requests whose bodies are sent in more than one write.

Fixes: set TCP_NODELAY on the socket (disables Nagle; every serious HTTP client, database driver and RPC library does so, but hand-written clients forget), or write the request in one buffer, or TCP_CORK it and uncork once. Go, Node.js and most modern runtimes set TCP_NODELAY by default on outbound connections; check before assuming.

The procedure, in the order that finds the answer fastest. Is the connection warm or cold? (Client-side timing — curl -w gives time_namelookup, time_connect, time_appconnect, time_starttransfer — separates handshakes from everything else.) Is the RTT what you expect? (ping; a climbing RTT under load is bufferbloat.) Was there loss? (ss -ti retrans counters; a 200 ms step is an RTO.) Where is the queue? (accept queue, socket buffers, pool wait time, balancer queue.) Is there a fixed 40 ms? (Nagle.) Only then: is the handler slow?

The slow-request-idle-server question is an interview staple because it separates people who think in application time from people who think in wall-clock time. The application is one bar on the waterfall. Everything else is the network, the kernel and the queues between them, and none of it shows up in CPU.