Everything Is I/O

The descriptor table (File Descriptors) does not care what its slots point at. Behind slot 3 might be a regular file on ext4; behind 4, a TCP socket; behind 5 and 6, the two ends of a pipe; behind 7, a terminal or /dev/null; behind 8, an eventfd (an 8-byte counter you can signal from another thread); behind 9, a timerfd (becomes readable when the timer fires); behind 10, an inotify instance; behind 11, an epoll instance watching 4 through 10. Each is a kernel object with its own implementation of the file operations table: read, write, poll, close, sometimes mmap, ioctl, lseek.

That uniformity is the design, not an accident. Because every source of events can be a descriptor, a single epoll_wait (I/O Multiplexing: select, poll, epoll, kqueue, IOCP) can block on "a client sent bytes, or the 30-second timer expired, or another thread signalled the eventfd, or a config file changed, or a child exited (pidfd)". A program that needed a separate waiting mechanism for each of those would need a thread for each. Linux has spent twenty years turning things that were not descriptors (signals → signalfd, timers → timerfd, processes → pidfd, memory → memfd) into descriptors precisely so they can join that one wait.

The same verbs also mean the same tools. strace shows a socket read and a file read as the same read(fd, …) line; lsof lists sockets, pipes and files in one table; /proc/<pid>/fd links to all of them; cat < /dev/tcp/host/80 in bash opens a socket because the shell only needs open-shaped semantics. The Socket: A Descriptor With Two Kernel Buffers Behind It and Pipes: A Kernel Buffer Between Two Processes describe two of the objects; this lesson is about what they share and where they part.

A regular file has a length and random access: lseek moves the offset, pread/pwrite read and write at an explicit position, and a read at the end returns 0. Its size is known, its content is stable between reads unless someone writes, and the page cache (Follow a File Read) makes re-reading free. A stream — socket, pipe, terminal, /dev/random — has none of that. There is no offset to seek to (lseek fails with ESPIPE), no size, and bytes read are gone: the kernel hands them over once. Data arrives in whatever chunks the other end and the network produced, with no boundaries preserved.

The consequence for code is that stream reads must be treated as "some bytes, one to n": read(fd, buf, 4096) on a socket returns whatever is in the receive buffer right now, which may be 1 byte, or 1460 (one TCP segment), or 4096, and a message the peer sent as one write may arrive across three reads or merged with the next message. Framing — length prefixes, delimiters, HTTP’s Content-Length and chunked encoding — is the application’s job. Regular-file reads also can be short, but only at end-of-file or after a signal, which is why file code that ignores the count usually works and socket code that does the same never does.

Datagram sockets (UDP, SOCK_DGRAM Unix sockets) are the third shape: each recv returns exactly one message, truncated if your buffer is too small. Pipes sit between: they are streams, but writes of at most PIPE_BUF bytes (4096 on Linux) are atomic — never interleaved with another writer’s — which is what makes many processes appending lines to one log pipe safe.

write may write fewer bytes than asked. On a blocking regular file this happens only on disk-full or a signal; on a blocking socket or pipe it happens when a signal interrupts a partially completed transfer; on a non-blocking socket it happens routinely — the send buffer had room for 65,536 of your 1,048,576 bytes and the call returns 65536. Correct code loops, advancing a pointer, until everything is written or EAGAIN says to wait for writability. Libraries wrap this (write_all, sendall, Node’s stream backpressure); raw syscall code that ignores the return value corrupts protocols silently.

End-of-file is read returning 0, and it means something different per object. Regular file: the offset reached the size. Pipe: every write end has been closed — one forgotten write end, inherited by a child or left in the parent, and the reader never sees EOF (the classic hang in a shell pipeline or a subprocess call). TCP socket: the peer sent FIN — it will send no more, though you may still be able to write to it (half-close; shutdown(SHUT_WR)). A tty: the user pressed Ctrl-D.

Writing to a closed peer is the mirror image. A write to a pipe with no readers, or to a socket whose peer has reset, delivers SIGPIPE, which kills the process by default — a server dies because a client disconnected mid-response. Every network daemon either ignores SIGPIPE or uses MSG_NOSIGNAL/SO_NOSIGPIPE, and then handles EPIPE as the error it should have been.

Because the accept loop, the upstream connections, the pipes to helpers and the epoll instance all draw from the same per-process pool, a server that handles 2,000 concurrent clients with a 50-connection database pool, a 100-connection HTTP client to another service, three log files and an epoll instance holds about 2,154 descriptors at steady state — twice the default 1024 soft limit. It does not fail on open. It fails on accept, which returns EMFILE; the connection sits in the listen backlog until it times out, the client sees a stall or a reset, and the server’s own logs say nothing unless the accept error is logged. Under a burst, nginx logs "socket() failed (24: Too many open files)"; Node throws EMFILE from net or from fs, whichever asked first.

A subtle version is the accept loop spin: accept fails with EMFILE, the listening socket is still readable (the connection is still queued), the event loop wakes immediately, accept fails again — a core at 100% and no progress. The known mitigation is to keep a spare descriptor open at startup (/dev/null), and on EMFILE close it, accept the connection, immediately close that, and reopen the spare, so the queued connection is drained and the client gets a clean close instead of a timeout.

The fix is a budget, not a bigger number: raise RLIMIT_NOFILE to cover the designed concurrency with headroom, bound every pool, close on every path, and alert on the descriptor count of the process. fd-leak-under-load in the challenges is the investigation. C10K: Ten Thousand Connections, Then a Million is the same arithmetic applied to ten thousand connections.

Windows also unifies many objects under HANDLEs — files, pipes, events, mutexes, processes, threads, timers — and ReadFile/WriteFile work on files and pipes alike. But sockets are SOCKETs from Winsock, and although a SOCKET is a HANDLE underneath, the practical rules differ: select accepts sockets only, WaitForMultipleObjects waits on synchronisation handles (with a limit of 64) and not on file readiness, and the scalable path for both files and sockets is completion-based I/O through an I/O completion port (I/O Multiplexing: select, poll, epoll, kqueue, IOCP). There is no PIPE_BUF atomicity guarantee on anonymous pipes in the same form, no SIGPIPE, and no /proc/<pid>/fd — handle.exe or the Sysinternals tools take its place.

The mental model — "one namespace of kernel objects, one way to wait" — survives; the specific verbs, limits and failure codes do not. Cross-platform runtimes (libuv, .NET, Java NIO) exist to paper over exactly this gap, and their edge cases are usually where the two models disagree.