The Buffer Chain

On the sender: the application often has its own buffer (a BufferedWriter, a framework’s response buffer, Node’s writable stream) that batches small writes into fewer syscalls. write() copies into the socket send buffer, bounded by SO_SNDBUF; TCP keeps bytes there until acknowledged. Segments the kernel decides to transmit go to the interface’s device queue (qdisc), then to the NIC’s transmit ring, then to the wire. Each of these is a queue with a limit.

On the receiver, in reverse: the NIC DMA-writes frames into the receive ring (a fixed number of descriptors, typically 256–4096, set by ethtool -g); the kernel drains it into sk_buffs and, after TCP processing, appends payload to the socket receive buffer, bounded by SO_RCVBUF; the application’s read() copies out of it into its own buffer. The receive buffer’s free space, minus some accounting, is the window the receiver advertises back to the sender.

The whole chain is what Follow send() Through the OS to recv() walked through; this lesson looks only at the queues and their coupling. One rule governs all of them: a full queue stops the producer above it. The producer’s stop is itself a full queue for the layer above that, and so on up to the application, which experiences it as a blocked write(), an EAGAIN, or a false return from a stream. That is backpressure, and it is the same mechanism whether the queue is a NIC ring or a socket buffer.

TCP can have at most one send buffer’s worth of unacknowledged data in flight, and it takes one RTT for an acknowledgment to return. So a connection’s throughput is bounded by buffer / RTT. To fill a 1 Gbit/s link with 100 ms RTT you need 1 Gbit/s × 0.1 s = 12.5 MB in flight; a 64 kB buffer on that path caps you at 64 kB / 0.1 s = 5 Mbit/s no matter how fast the link is. That product — bandwidth × delay — is the minimum buffer for full throughput, and it grows with distance: the same buffer that saturates a data centre link at 0.5 ms RTT is 200× too small across an ocean. See Bandwidth vs Latency and Throughput: Requests, Packets and Bytes per Second.

The receive side must match, because the receiver’s buffer bounds the window it advertises: a sender with a 12 MB send buffer talking to a receiver with a 64 kB receive buffer is limited by the receiver. The historical 64 kB TCP window limit is why the window-scale option exists; without it no TCP connection could exceed 64 kB per RTT.

Large buffers are not free. Each byte in a send buffer is memory held until ACKed; on a 100,000-connection server 12 MB each is impossible, which is why the kernel sizes buffers per connection dynamically rather than at the maximum.

Linux does not allocate SO_SNDBUF and SO_RCVBUF at their maximum. net.ipv4.tcp_wmem and net.ipv4.tcp_rmem are triples — min default max, e.g. 4096 16384 4194304 and 4096 131072 6291456 — and each connection starts at default and grows toward `max` as the connection demonstrates it needs the space: the receive buffer grows when the application drains it and the window would otherwise limit the sender; the send buffer grows with the congestion window. Idle or slow connections stay small. This is what makes 100,000 connections and a 12 MB long-haul transfer coexist on one box.

Setting SO_RCVBUF or SO_SNDBUF explicitly on a socket disables autotuning for that socket and fixes the size (the kernel doubles the value you pass, for bookkeeping overhead, and caps it at net.core.rmem_max/wmem_max). This is the single most common self-inflicted throughput bug: an application sets a 64 kB buffer "to be safe" and caps itself at a few Mbit/s on any long path. Leave the socket options alone unless you have measured; tune the sysctl maxima if long paths need more than the defaults allow.

On the device side, Byte Queue Limits (BQL) bound how many bytes sit in the NIC ring so that the qdisc, not the hardware, holds the queue and can apply a policy; TCP Small Queues (TSQ) bound how many bytes one socket may have in the qdisc, so a bulk flow cannot monopolise the device queue. Both exist to keep the queue where a scheduler can see it. Other operating systems have equivalents with different names and defaults; do not carry the sysctl names across.

Buffers absorb bursts. A buffer that is *larger than the burst it needs to absorb* does something worse than waste memory: it adds latency without adding throughput. A 1 Gbit/s link with 100 MB of queue in front of it delivers 1 Gbit/s and 800 ms of queueing delay; every packet waits behind the queue before it is sent. Loss-based congestion control (see Congestion Control: Protecting the Network) makes it worse, because the sender keeps increasing cwnd until a packet is dropped, and a huge buffer drops nothing until it is full — so the sender fills it, and everyone sharing the link waits.

This is bufferbloat, and it lives in every queue on the chain: over-deep qdiscs and NIC rings (why BQL exists), home routers and cable modems with megabytes of FIFO, and oversized socket buffers that let one application pump a whole RTT’s worth of data ahead of a keystroke. The fix is not smaller buffers everywhere but smarter ones: fq_codel and cake measure queueing delay and drop or mark early; BBR estimates the bottleneck bandwidth and RTT directly instead of filling buffers to find loss; ECN lets routers signal congestion without dropping.

The practical rule: at the socket, let autotuning size the buffer; at the device, keep the default fq_codel; on the path, measure latency under load, not just idle ping. A link that shows 5 ms idle and 300 ms with one download running has bufferbloat somewhere, and the throughput graph will never show it.