Follow send() Through the OS to recv()

send() on a stream socket promises one thing: the bytes it reports as sent have been copied into the kernel’s send buffer for that socket. It does not promise they were transmitted, acknowledged, or read. That single fact explains most of the surprises in this lesson: the return value measures a copy into kernel memory, not delivery.

From there the bytes cross a fixed sequence of layers — transport, IP, the device queue, the NIC, the wire — and on the far side the same layers in reverse, ending in a copy out of the receiving kernel into the receiver’s user buffer. Each layer has its own state (sequence numbers, windows, routing cache, ring-buffer slots), its own queue, and its own way of saying "not now".

The picture below is the whole path at a glance. The rest of the lesson expands each side. The names in the ladders (sk_buff, qdisc, NAPI, softirq) are Linux; the structure — buffer, gate, queue, DMA, interrupt, wake-up — is what every mainstream kernel does, with different names.

The application calls send() — via write(), a language runtime, or a library; it all ends at the same system call. The CPU switches to kernel mode (see System Calls), the kernel looks up the descriptor in the process’s table (see File Descriptors) and finds a socket. If the socket’s send buffer has room, the kernel copies the user bytes into kernel-owned sk_buff structures and returns. If it has no room and the socket is blocking, the calling thread goes to sleep here; if non-blocking, the call returns EAGAIN.

TCP then decides how much of the buffered stream it may transmit right now. Two windows gate it: the receiver’s advertised window rwnd (see Flow Control: The Receive Window) and the sender’s congestion window cwnd (see Congestion Control: Protecting the Network). Whatever fits is cut into segments no larger than the MSS, each stamped with a sequence number (see Sequence Numbers, ACKs and Reassembly), and handed to IP. With TSO/GSO the kernel hands one large segment down and lets the NIC (or the driver, late) do the cutting, saving per-packet CPU.

IP consults the routing table (see The Routing Table and Longest-Prefix Match), chooses the outgoing interface and next hop, resolves the next hop’s link address (see ARP and Neighbor Discovery: From an IP to a Local MAC), and prepends the IP header. The packet is enqueued on the interface’s qdisc — the queueing discipline, fq_codel by default on most modern distributions — which decides ordering and can drop under load. The driver then places a descriptor pointing at the packet in the NIC’s transmit ring; the NIC reads the bytes by DMA, computes the checksum if offload is enabled, and puts the frame on the wire.

The receiving NIC matches the frame’s destination MAC (see MAC Addresses: Identity for One Hop), writes the bytes by DMA into a pre-allocated buffer that the driver posted on the receive ring, and raises an interrupt. Linux uses NAPI: the first interrupt disables further interrupts for that queue and schedules a poll; the kernel then drains the ring in a batch (netdev_budget, 300 packets by default) before re-enabling interrupts. Under a high packet rate this turns thousands of interrupts per second into a polling loop, which is why a busy server shows CPU in softirq rather than in irq.

Each packet becomes an sk_buff and climbs the stack in software-interrupt context: link layer strips the Ethernet header, IP validates the checksum and decides "for me" or "forward", TCP matches the 4-tuple to a socket (see Ports: Addressing a Process, Not a Machine), checks the sequence number against what it expects, reassembles in-order data, and queues an ACK. In-order bytes are appended to the socket’s receive buffer; out-of-order segments wait in a separate queue until the gap is filled (see Head-of-Line Blocking). The receive window the kernel advertises back shrinks by exactly what it just queued.

Now the kernel looks for someone waiting. If a thread is blocked in recv() on that socket, it is marked runnable and the scheduler will run it within a time slice or on the next idle core (see Context Switching). If the socket is registered with epoll, the socket is added to the ready list and the thread blocked in epoll_wait() wakes instead (see I/O Multiplexing: select, poll, epoll, kqueue, IOCP). Either way the wake-up is a scheduler event, not a network one — this is where OS latency enters the path. When the thread runs, recv() copies the bytes from the receive buffer into the user buffer, frees the kernel memory, and the window can open again.

On a conventional path there are exactly two CPU copies of the payload — user → kernel on send, kernel → user on receive — plus two DMA transfers that the CPU does not perform. Everything else is header manipulation on a pointer. The copies matter at high throughput (a 10 Gbit/s stream is ~1.25 GB/s of memcpy per direction) and are what Zero-Copy: Serving a File Without Touching It removes for file-to-socket cases; for ordinary request/response traffic they are not the bottleneck.

Latency accumulates at the queues, not the copies. On the sender: waiting for send-buffer space, waiting for cwnd/rwnd to open, waiting in the qdisc behind other flows. In the network: serialization, propagation (~5 µs per km in fibre), router queueing. On the receiver: the interrupt-to-poll delay, the softirq run, and above all the scheduler: a woken thread runs only when a core is free. Under load the gap between "bytes in the receive buffer" and "application reads them" is frequently larger than the network RTT, and it shows up in the network’s tools as a shrinking receive window.

The rule for reading these layers in an incident: ss -ti on both ends tells you which queue holds the bytes. Send-Q non-zero on the sender and Recv-Q zero on the receiver means the network or the windows; Recv-Q growing on the receiver means the application is not reading — an OS problem wearing a networking symptom. See TCP Debugging: Reading the Handshake on the Wire and Combined Failure Simulator: Break a Layer, Watch It Propagate.

With TCP, everything from segmentation to ACKs to retransmission happens in the kernel; the application sees a byte stream. With QUIC (the transport under HTTP/3 and QUIC) the kernel only sees UDP datagrams. Loss detection, congestion control, flow control per stream, encryption and reassembly run in the application’s library — Chrome’s, Cloudflare’s, quiche, msquic. The ladder above still exists, but the "TCP" rung becomes user-space code, and the kernel rungs shrink to sendmsg()/recvmsg() on a UDP socket.

That has two consequences worth remembering. Deployment is faster — a congestion-control change ships with the app instead of a kernel upgrade — and the per-packet cost is higher, because each datagram crosses the syscall boundary individually unless the stack uses sendmmsg/recvmmsg batching or UDP GSO. Server-side QUIC stacks spend real engineering effort recovering CPU efficiency that TCP gets from TSO, GRO and kernel-resident state.

Windows and the BSDs implement the same structure with different names (Winsock, AFD, kqueue instead of epoll, different offload knobs). Do not assume SO_SNDBUF semantics, autotuning behaviour or ring-buffer sizes carry across; the shape of the path does.