Zero-Copy: Serving a File Without Touching It

read(file_fd, buf, n): the kernel finds the file’s pages in the page cache (loading them from disk by DMA if absent — see Follow a File Read), then copies them into buf in user space (copy 1: CPU). write(sock_fd, buf, n): the kernel copies buf into the socket send buffer (copy 2: CPU); later the NIC copies from the socket buffer by DMA (copy 3: DMA). Counting the disk-to-page-cache DMA, that is four copies of every byte, two of them by the CPU, and each syscall is two mode switches — four crossings per chunk. For a 1 GB file in 64 kB chunks: 32,768 syscalls and 2 GB of memcpy the CPU performs for no reason except that the data visited a buffer it did not need.

At low throughput this is invisible. At 10 Gbit/s, 1.25 GB/s of memcpy per direction is a substantial fraction of a core, cache pollution for everything else on it, and memory bandwidth you are paying for twice. File servers, CDN edges, video origins, log shippers and message brokers hit this wall first, which is why the mechanisms below come from exactly those systems.

sendfile(out_fd, in_fd, offset, count) asks the kernel to move bytes from one descriptor to another without visiting user space. On Linux, with a NIC that supports scatter-gather DMA and checksum offload, the kernel does not even copy from the page cache to the socket buffer: it appends *descriptors* — page address plus length — to the socket’s send queue, and the NIC gathers the payload directly from the page-cache pages. That leaves two DMA transfers and zero CPU copies, with one syscall for the whole file: one mode-switch pair instead of thousands. Without scatter-gather the kernel makes one CPU copy (page cache → socket buffer) — still half the naive cost. This is what nginx’s sendfile on does and what Kafka uses (via Java’s FileChannel.transferTo) to ship log segments to consumers at near line rate.

splice(fd_in, …, fd_out, …) generalises the idea with a pipe as the intermediary: data moves between two descriptors by passing references to the pages through the pipe buffer, not by copying. It handles socket-to-socket (a proxy forwarding a stream), socket-to-file (recording an upload) and file-to-socket alike; sendfile on modern Linux is implemented on top of it. HAProxy uses splice for TCP forwarding for precisely the proxy case.

The APIs are not portable. Linux sendfile takes (out, in, offset*, count) and originally required a socket as the output; FreeBSD and macOS have a sendfile with a different signature and extra header/trailer arguments; Windows has TransmitFile on a socket handle; Java hides the differences behind transferTo, with fallbacks when the platform cannot do it. The concept — "let the kernel move bytes between descriptors" — is universal; the calls, limits and edge cases are not.

mmap the file and write() the mapping to the socket, and one copy disappears: the page cache *is* the user-visible buffer (see Memory Mapping), so write() copies page cache → socket buffer directly, three copies total. It costs page-table setup and faults, it is unsafe if the file is truncated while mapped (SIGBUS), and for one-shot sends sendfile beats it; it wins when the process also needs to read the data (a database serving from a mapped file, a search engine scanning a mapped index).

When the data must be assembled in user space — an HTTP header followed by a body, a protocol frame with a computed prefix — writev/sendmsg take an array of buffers and copy them into the socket in one syscall, avoiding either a concatenation copy or one syscall per piece. Linux additionally offers MSG_ZEROCOPY on send(), which pins the user pages and lets the NIC DMA from them directly, with a completion notification when the pages may be reused; it pays off only above roughly 10 kB per call, because the pinning costs more than a small copy.

The extreme is kernel bypass: DPDK, netmap, AF_XDP, and user-space TCP stacks map the NIC’s rings directly into a process and poll them, so packets never touch the kernel at all. A single core can then handle tens of millions of packets per second. The price is total: no kernel TCP, no epoll, no firewall, no sharing the NIC with anything else, and a process that busy-polls one core forever. It is the right design for a software router, a load balancer at the edge, or an exchange’s market-data gateway, and wrong for everything else.

Zero-copy assumes the bytes on the wire are the bytes in the page cache. TLS breaks that: every byte must be encrypted, and the encryption has to happen somewhere with the key. If TLS runs in the application (OpenSSL in nginx, the JVM’s TLS in Kafka), the data must come up to user space to be encrypted and go back down as ciphertext — the naive path plus AES. Kafka’s documentation says it directly: enabling TLS disables the zero-copy path. For most services TLS overhead is dominated by the handshake, not the record encryption; for a file origin at 40 Gbit/s it is the whole story.

Kernel TLS (kTLS, Linux since 4.13 for transmit, 4.17 for receive) moves the record encryption into the socket: the application does the handshake in user space, hands the negotiated keys to the kernel with setsockopt(SOL_TLS), and thereafter sendfile on that socket encrypts on the way out — with hardware offload on NICs that support it, zero CPU copies again. nginx supports it with OpenSSL 3; FreeBSD has an equivalent used by Netflix’s origins, which is the canonical case study for this whole lesson. Windows has no equivalent exposed the same way; again, the idea is portable and the API is not.

Two more limits. Zero-copy removes CPU copies, not disk reads: a file not in the page cache still costs a disk DMA and a wait, and sendfile on a blocking socket blocks for the disk read as well. And it removes nothing from the network: a 100 ms RTT path is still bounded by bandwidth × delay (The Buffer Chain) whatever the server’s CPU is doing. Zero-copy is a CPU and memory-bandwidth optimisation for high-throughput file-to-socket paths, and only that.