Sequence Numbers, ACKs and Reassembly

TCP numbers bytes, not segments. The connection’s ISN (The Three-Way Handshake) is byte zero’s number minus one; every byte written afterwards gets the next number. A segment’s sequence number is the number of its first byte; its length tells the receiver which range it covers. In the relative numbering that Wireshark shows and this lesson uses, the first segment carries bytes 0–999, the second 1000–1999, the third 2000–2999.

An acknowledgment number names the *next byte the receiver expects* — one past the highest byte received in order. After segment one, the receiver ACKs 1000. That single number says "I have everything below 1000, contiguous, no gaps". ACKs are cumulative: an ACK of 3000 acknowledges everything below it, including earlier segments whose own ACKs were lost, so lost ACKs cost nothing as long as a later one gets through.

Now lose segment two. Segment three arrives; the receiver stores bytes 2000–2999 but cannot deliver them, because 1000–1999 is missing and the stream must be in order. What does it ACK? The only number a cumulative ACK can carry is 1000 — the next byte it needs — the same number it already sent. The sender sees a duplicate ACK. Segment four arrives (3000–3999): another ACK 1000. The sender now knows *something* after byte 999 is missing and that *some* later data arrived, but not which; it cannot tell "lost segment two only" from "lost two, three and four".

The classic sender behaviour (Reno) retransmits from the ACK number and then waits to see what the next ACK says: if it jumps to 5000, everything was there; if it moves to 2000, another hole. Each hole costs a round trip to discover. On a path with 150 ms RTT and a burst of loss, that is a very slow way to recover, and it is the whole reason the next mechanism exists.

Selective acknowledgment (RFC 2018) adds an option to the ACK listing up to three or four ranges the receiver holds beyond the cumulative point: ACK 1000, SACK 2000–3000, 3000–4000 (the ranges merge into 2000–4000 as they become contiguous). The sender now knows precisely that bytes 1000–1999 are the only hole and retransmits exactly those, once. With several holes it retransmits each of them in one round trip instead of one per round trip. Both sides must have agreed SACK in the handshake; every mainstream stack has it on by default, and a SYN with the option stripped is a measurable performance loss.

The sender keeps a scoreboard: for every byte in flight, has it been SACKed, is it presumed lost, has it been retransmitted. Modern loss detection (RACK on Linux) uses the scoreboard plus timing — a segment is presumed lost if a segment sent *later* has been SACKed and enough time has passed — which catches losses that never generate three duplicate ACKs, such as the last segments of a transfer. Packet Loss: Duplicate ACKs, Fast Retransmit and the RTO covers the timers.

Out-of-order segments are not discarded; they are held in an out-of-order queue attached to the socket until the hole before them fills. The moment the retransmission of 1000–1999 arrives, the receiver splices it in, finds 1000–4999 contiguous, and moves all of it to the in-order receive buffer in one step; a single read() can then return 4000 bytes. The application sees a stall of one round trip (or an RTO) followed by a burst — never an error, never a gap. That stall is Head-of-Line Blocking in its purest form.

Held segments occupy receive-buffer memory and count against the window advertised to the sender (Flow Control: The Receive Window); a window full of out-of-order data with one missing segment is a stalled connection. Linux exposes the queue in ss -ti as ofo counters and in nstat as TcpExtTCPOFOQueue; a high rate on a link that should not lose packets often means reordering — ECMP hashing gone wrong, link aggregation, or a bonding driver — which TCP treats as loss unless the reordering detection (tcp_reordering, RACK) absorbs it.

If TCP numbered segments, a retransmission would have to be the same segment — same boundaries, same size. Numbering bytes frees the sender to re-segment: retransmit 1000–1999 as two 500-byte pieces if the path MTU shrank, coalesce 1000–2999 into one segment if the MSS allows, or let the NIC do segmentation offload (TSO/GSO) on a 64 kB chunk and split it any way it likes. The receiver does not care how the bytes were packaged; it only cares which byte numbers arrived. Any two segments with overlapping ranges are simply merged.

Byte numbering also makes the ACK a byte count, which is exactly what the sender’s buffer management needs: "ACK 5000" means bytes below 5000 can be freed from the send buffer, whatever segments they travelled in. The only cost is the 32-bit space: at 10 Gbit/s the numbers wrap every ~3.4 seconds (4 GiB), and an old segment from before the wrap could be mistaken for a new one — so the timestamp option (PAWS, RFC 7323) tags every segment with a monotonically increasing clock and the receiver discards anything from the past.