Congestion Control: Protecting the Network

The receiver can say "I have 1 MB of room" (Flow Control: The Receive Window); the path cannot. Its capacity depends on the slowest link and on how many other flows are sharing it right now, and both change by the second. Routers keep queues to absorb bursts; when a queue is full they drop. That drop is the only message the network sends, and it sends it late — after the queue is already full and every packet in it is already delayed.

Congestion collapse is what happens without control: senders see loss, retransmit at the same rate, the queues stay full, more is dropped, and the network carries mostly retransmissions of retransmissions. Van Jacobson’s 1988 fix, still the skeleton of everything since, was to make every sender maintain a second window — the congestion window, cwnd — estimated from the network’s behaviour, and to send no more than min(cwnd, rwnd). Congestion control is therefore distributed, cooperative and inferred: every endpoint runs the same algorithm on the same weak signal, and the result approximates a fair share.

A new connection knows nothing about the path, so it starts small and grows fast. Slow start (badly named — it is the fast phase) begins with an initial window of 10 segments (~14 kB; RFC 6928, Linux default) and adds one segment to cwnd for every ACK received, which doubles cwnd every round trip: 10, 20, 40, 80… until either loss occurs or cwnd reaches the slow-start threshold, ssthresh. Above ssthresh, congestion avoidance adds roughly one segment per round trip — additive increase — probing gently for more.

On a loss event the sender infers the bottleneck queue overflowed, sets ssthresh to a fraction of the current cwnd, and cuts cwnd — to half in classic Reno, to 70% in CUBIC — then resumes additive increase. Additive increase, multiplicative decrease (AIMD) produces the famous sawtooth of cwnd over time, and it has a proof behind it: multiple AIMD flows sharing a bottleneck converge toward equal shares, because a flow with a bigger window loses proportionally more on each cut. An RTO, as opposed to a fast-retransmit loss, is treated as evidence of severe congestion and resets cwnd to one segment with slow start from scratch (Packet Loss: Duplicate ACKs, Fast Retransmit and the RTO).

Loss is a bad signal in two directions. It is late: a drop happens only after the bottleneck queue is full, and a full queue means every packet through it is delayed by the queue’s length — a loss-based sender therefore *operates at maximum latency by design*, filling whatever buffer the router has before it learns to slow down. It is also ambiguous: a corrupted Wi-Fi frame is not congestion, but a loss-based sender halves its window anyway, which is why TCP over lossy wireless was so poor for so long.

The late signal is amplified by bufferbloat: router and modem buffers sized in seconds rather than milliseconds, into which a loss-based flow expands until the queue is full, at which point every other packet on the link — a DNS query, a game packet, an SSH keystroke — waits behind a second of bulk data. The remedies are on both sides. In the network, active queue management (CoDel, fq_codel, PIE) drops or marks packets when queueing *delay* grows, before the buffer is full. In the packet, Explicit Congestion Notification (ECN): a router with AQM sets a mark in the IP header instead of dropping, the receiver echoes it in the TCP header (ECE), and the sender reduces its window as if for loss but without losing anything. ECN needs both ends and the path to cooperate; it is widely negotiated but unevenly honoured, and its low-latency successor (L4S) is being deployed.

CUBIC (Linux default since 2.6.19, also Windows and macOS) is loss-based AIMD refined for large windows: after a loss it grows cwnd along a cubic curve — fast at first, flattening as it approaches the window size where the last loss happened, then accelerating past it — so that a high-bandwidth, high-RTT path recovers in seconds rather than the minutes Reno’s one-segment-per-RTT would take. It still fills the bottleneck buffer and still reads all loss as congestion.

BBR (Google, 2016; Linux since 4.9) does not use loss as its primary signal. It continuously estimates two quantities — the bottleneck bandwidth (the maximum delivery rate observed) and the minimum round-trip time (the RTT with empty queues) — and paces sends at roughly their product, the bandwidth-delay product, deliberately keeping the queue near empty. Random loss barely affects it; bufferbloat is avoided rather than caused. The cost is fairness: early BBR could starve CUBIC flows sharing a link, and BBRv2/v3 add loss and ECN as secondary signals to coexist. QUIC stacks ship their own controllers (usually CUBIC or BBR variants) in user space, which is why they can change per application.

Do not treat either as "the" TCP. Which controller a host runs (sysctl net.ipv4.tcp_congestion_control) is a per-host choice, changes the shape of throughput under loss and delay, and is one of the few knobs worth knowing about when a service talks across long, lossy paths.

Slow start is why a new connection cannot use the bandwidth you paid for. From an initial window of 10 segments (~14 kB), a response of 100 kB needs about four round trips just to deliver the data: 14 kB, then 28, then 56, then the rest — on a 100 ms path, 400 ms of transfer for a file that the link could move in a millisecond. Small responses fit in the initial window and cost one RTT; that boundary is why the first ~14 kB of a web page are special and why TLS certificates and critical CSS are kept under it.

It is also why connections are reused (Keep-Alive and Connection Reuse, Connection Pooling): an established connection has already found its window. With one caveat — after an idle period longer than one RTO, Linux resets cwnd to the initial window (net.ipv4.tcp_slow_start_after_idle = 1, the default), so a pooled connection that has been idle for a second starts over. Servers that stream large responses on long-lived connections often turn this off. The same slow-start cost, repeated per request across an ocean, is the "the server is idle and the request still takes 800 ms" of Where the Time Goes: The Request Timeline.