Flow Control: The Receive Window

Every TCP socket has a receive buffer in the kernel. Segments that arrive in order are appended to it; the application’s read() drains it. If the application reads slower than data arrives, the buffer fills. The receiver reports the free space — the receive window, rwnd — in the 16-bit window field of every segment it sends, and the sender obeys one rule: bytes sent but not yet acknowledged ≤ rwnd. This is flow control: the receiver’s capacity, advertised continuously, bounding the sender.

The window slides. As the receiver ACKs bytes, the left edge moves right, and as the application reads, the right edge moves right; the sender may transmit anything between the last ACK and the last ACK plus the window. When the application stops reading, the right edge stops and the sender runs into it. Nothing is dropped, nothing is retransmitted; the sender simply waits, and the data that is waiting sits in *its* send buffer, which then fills too, so that its write() eventually blocks. That chain — receiver’s application → receiver’s buffer → window → sender’s buffer → sender’s application — is backpressure, and TCP provides it for free.

The window field is 16 bits, so the largest window TCP could originally advertise was 65,535 bytes. A sender may not have more than that in flight, and in flight means "sent, ACK not yet back" — a full round trip. So throughput ≤ window / RTT: 64 kB per 100 ms is 5.2 Mbit/s, no matter how fast the link. To fill a 1 Gbit/s path with a 100 ms RTT you need 1 Gbit/s × 0.1 s = 12.5 MB in flight — the bandwidth-delay product, the amount of data the path itself is holding when it is full.

The window scale option (RFC 7323) multiplies the window field by 2^n, n up to 14, allowing windows up to 1 GiB. It is negotiated only in the SYN and SYN-ACK (The Three-Way Handshake) — a middlebox that strips it, or a stack with it disabled, caps every connection at 64 kB forever, which on a long path is a few Mbit/s with no error and no obvious cause. Modern kernels autotune buffer sizes: Linux grows the receive buffer as the connection demonstrates it needs more, between the bounds in net.ipv4.tcp_rmem (default max 6 MB, often raised on servers that talk across continents).

When the buffer is completely full the receiver advertises window 0. The sender must stop, and now there is a deadlock risk: the receiver will announce a reopened window in its next ACK — but ACKs are sent in response to data, and no data is arriving. If that window-update segment is lost, both sides wait forever. So the sender runs a persist timer and sends periodic zero-window probes (one byte beyond the window, at increasing intervals) purely to elicit an ACK that carries the current window. The probe does not stop after backoff; Linux keeps probing, so a zero-window connection to a stuck reader stays open indefinitely.

A related pathology is silly window syndrome: the receiver frees a few bytes, advertises a tiny window, the sender sends a tiny segment, and the connection limps along at one small packet per round trip. Both ends guard against it — the receiver does not advertise a window smaller than an MSS or half its buffer until it has that much free; the sender (Nagle) does not send tiny segments while others are unacknowledged. The combination of Nagle with delayed ACKs is the source of the classic 40 ms stalls in request/response protocols that write headers and body separately.

The advertised window is, at bottom, a statement about the receiving *process*: how far behind it is. A server that reads a request, then does 500 ms of work before reading the next byte, has a full receive buffer and a zero window for those 500 ms as far as a streaming client is concerned. A consumer that calls read() only from a single-threaded event loop that is currently busy advertises the same. On the other side of the same coin, a *sending* process whose peer is slow ends up blocked in write() — and if that process is a server writing to a slow client, one slow client can pin one worker thread or, in an event loop that writes synchronously, stall everyone (What Happens When the Receiver Is Slow, slow-client-blocks-server).

This is the point where the OS and the network meet. Recv-Q in ss -tan is bytes in the receive buffer the application has not read; a persistently non-zero Recv-Q on a server is a server that is not keeping up. Send-Q is bytes in the send buffer not yet acknowledged; a large Send-Q toward one peer is a slow or windowed peer. The Buffer Chain follows the bytes through both queues, and the socket-buffer-full-backpressure challenge diagnoses a stall from these two numbers alone.

Two windows, two purposes, one sender. The receive window is set by the receiver and protects the receiver’s buffer. The congestion window (Congestion Control: Protecting the Network) is computed by the sender and protects the network’s queues. The sender obeys both: in flight ≤ min(rwnd, cwnd). A connection can be receiver-limited, congestion-limited, or — the good case — application-limited, sending everything it has as soon as it has it. ss -ti reports which (rwnd_limited, sndbuf_limited, and cwnd against the bandwidth-delay product).

They fail differently too. Receiver-limited throughput is flat and steady, capped at rwnd/RTT, with no loss; congestion-limited throughput has the sawtooth of growth and loss. Knowing which one you are looking at is the first step of every "the transfer is slow" investigation.