Build a Tiny Server: V0 to V5
Six versions of the same server, each one born from the specific failure of the previous one: a single request, a blocking loop, a thread per client, a pool, non-blocking sockets, and finally an event loop.
The problem
V0 → V1: one request, then a loop
V0 is the smallest program that speaks TCP: socket(), bind(), listen(), accept() once, recv(), send(), exit. It works, and it teaches the four calls every later version keeps. Its failure is trivial: it serves one client and dies. See The Socket: A Descriptor With Two Kernel Buffers Behind It for what each call creates in the kernel.
V1 wraps V0 in while True. Now the server lives forever and serves clients one after another. Its failure is the one that motivates everything else: `recv()` blocks. While the server waits for client A to send its request, client B’s connection sits in the accept backlog, and clients C through Z queue behind it. A single client that connects and never sends stalls the whole server. The The Blocking Server lesson dissects this.
1import socket2srv = socket.socket(); srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)3srv.bind(("0.0.0.0", 8080)); srv.listen(128)4while True:5 conn, addr = srv.accept() # blocks until a client arrives6 data = conn.recv(4096) # blocks until THIS client sends7 conn.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok")8 conn.close() # next client only after thisV2: a thread per connection
The fix for "one slow client blocks everyone" is to let each client block its own thread. accept() hands the new socket to a fresh thread; the main thread returns to accept() immediately. The kernel scheduler now does the interleaving: when thread A blocks in recv(), thread B runs. This is why the design is so natural — the code stays sequential and the OS supplies concurrency. See Threads: Several Instruction Streams in One Process and Process versus Thread.
Its failure is arithmetic. Each thread costs a kernel structure, a stack (8 MB *reserved* by default on Linux, of which only touched pages are real memory — see Thread per Connection for the honest accounting), and a scheduler entry. Ten thousand idle connections is ten thousand threads the scheduler must consider; a burst of new connections means ten thousand clone() calls. And an attacker or a buggy client can open connections until the server cannot create threads at all.
1import socket, threading2def handle(conn):3 data = conn.recv(4096) # blocks only this thread4 conn.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok")5 conn.close()6srv = socket.socket(); srv.bind(("0.0.0.0", 8080)); srv.listen(128)7while True:8 conn, _ = srv.accept()9 threading.Thread(target=handle, args=(conn,), daemon=True).start()V3: a fixed pool and a queue
If unbounded threads are the problem, bound them. V3 creates N worker threads once and puts accepted connections on a queue; workers pull the next connection when they finish the previous one. Thread creation cost disappears, memory is capped, and the queue length becomes a visible measure of overload. This is the shape of Apache’s worker MPM, Java servlet containers, Python’s ThreadPoolExecutor servers, and most database servers’ connection handling. See The Thread Pool Server.
Its failure is the return of V1’s problem at a smaller scale: a slow client occupies a worker for its whole lifetime. With 32 workers, 32 slow clients — deliberately or by accident — stall the server exactly as one client stalled V1. The queue grows without bound unless you cap it, and if you cap it you must decide what to do with the overflow: reject, or make the client wait in the accept backlog. See Semaphores and Condition Variables for the counting primitive behind a bounded pool.
1import socket, threading, queue2work: "queue.Queue[socket.socket]" = queue.Queue(maxsize=1024)3def worker():4 while True:5 conn = work.get() # blocks until a connection is queued6 conn.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok" if conn.recv(4096) else b"")7 conn.close()8for _ in range(32): threading.Thread(target=worker, daemon=True).start()9srv = socket.socket(); srv.bind(("0.0.0.0", 8080)); srv.listen(1024)10while True:11 work.put(srv.accept()[0]) # blocks when the queue is full: backpressureV4: stop blocking at all
The deepest cause of every failure so far is that recv() blocks a thread, and threads are the expensive unit. V4 marks every socket non-blocking: recv() returns immediately with data or with EAGAIN. One thread can now hold every connection in a list and try each one in turn. No thread is ever parked on a slow client. See Blocking, Non-blocking, Multiplexed, Asynchronous.
Its failure is that "try each one in turn" is a busy loop. With 10,000 mostly idle sockets the thread performs 10,000 system calls per pass and burns a core to discover that nothing happened. The server is correct and unusable. What it needs is a way to ask the kernel "which of these have something to do?" — and that question is exactly what I/O Multiplexing: select, poll, epoll, kqueue, IOCP answers.
1import socket, errno2srv = socket.socket(); srv.bind(("0.0.0.0", 8080)); srv.listen(1024); srv.setblocking(False)3conns: list[socket.socket] = []4while True:5 try: c, _ = srv.accept(); c.setblocking(False); conns.append(c)6 except BlockingIOError: pass # EAGAIN: nobody is connecting right now7 for c in list(conns):8 try: data = c.recv(4096) # EAGAIN if nothing has arrived9 except BlockingIOError: continue10 c.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok"); c.close(); conns.remove(c)V5: the event loop
V5 registers every socket with the kernel’s readiness mechanism — epoll on Linux, kqueue on BSD/macOS, IOCP (a completion model, not readiness) on Windows — and blocks in one call, epoll_wait(), that returns only the sockets with work. The thread sleeps while nothing happens and handles exactly the ready sockets when something does. Ten thousand idle connections cost one sleeping thread and a few kilobytes of kernel state each. This is nginx, Node.js, Redis, HAProxy, Envoy, and the core of every asyncio/Tokio/Netty runtime. See The Event-Driven Server and The Event Loop.
Its failure is the one the other models never had: since one thread serves everyone, any blocking or CPU-heavy work in a handler stalls every connection at once. The discipline "never block the loop" is the price of this design, and the standard remedy — hand CPU work to a thread pool and return to the loop — brings V3 back as a component rather than as the architecture. Production servers are almost always this hybrid.
The matrix compares the six versions. The right column is the point: none of these is "correct"; each is the answer to a specific constraint, and green threads (How C++, JavaScript and Python Map onto the OS such as Go’s goroutines or Java’s virtual threads) let a runtime give you V2’s code with V5’s cost, which is why the choice is a runtime question as much as an OS one.
1import socket, selectors2sel = selectors.DefaultSelector()3srv = socket.socket(); srv.bind(("0.0.0.0", 8080)); srv.listen(1024); srv.setblocking(False)4sel.register(srv, selectors.EVENT_READ)5while True:6 for key, _ in sel.select(): # sleeps until at least one socket is ready7 if key.fileobj is srv:8 c, _ = srv.accept(); c.setblocking(False); sel.register(c, selectors.EVENT_READ)9 else:10 c = key.fileobj; data = c.recv(4096)11 c.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok"); sel.unregister(c); c.close()| Version | Connections handled | CPU while waiting | Memory per connection | Complexity | What breaks |
|---|---|---|---|---|---|
| V0 one request | 1, ever | idle (blocked) | — | trivial | exits after one client |
| V1 blocking loop | 1 at a time | idle (blocked) | one socket | trivial | one slow client stalls all; backlog fills |
| V2 thread/connection | thousands | idle (each blocked) | thread stack (touched pages) + kernel task | low | thread count; creation storms; scheduler load |
| V3 thread pool | N active, rest queued | idle | bounded: N stacks + queue entries | moderate | N slow clients stall the pool; queue growth |
| V4 non-blocking poll | thousands | 100% (busy loop) | socket + app state | moderate | burns a core doing nothing |
| V5 event loop | tens of thousands+ | idle (epoll_wait) | socket + app state | high (inversion of control) | any blocking handler stalls everyone |
Key points
- V1 fails because
recv()blocks the only thread; V2 fixes it with threads and fails on thread cost; V3 bounds threads and fails on slow clients occupying workers; V4 removes blocking and fails on busy-polling; V5 asks the kernel what is ready. - The expensive unit is a blocked thread, not a connection. Every architecture is a strategy for not parking a thread on a socket.
- Thread pools turn overload into a visible queue; event loops turn it into latency. Both need a limit somewhere.
- The event loop’s discipline — never block the loop — is the cost of its efficiency; production servers pair it with a worker pool for CPU work.
- Green threads (Go, Java virtual threads, Erlang) give V2’s programming model at close to V5’s cost, which is why "threads vs events" is a runtime question.
- No version is universally right: a batch tool is best as V1, an interactive high-fan-out proxy as V5, most business services as V3 or a runtime that hides the choice.
Why does this exist?
Mechanisms are answers to constraints. Open each question before reading the answer.
▸Why not start at the event loop, since it handles the most connections?
Because its programming model — callbacks or coroutines, and the rule that nothing may block — is harder to write and harder to debug, and for a service with 200 connections and CPU-bound handlers a thread pool is simpler and just as fast. The event loop is the answer to a specific constraint: many mostly-idle connections.
▸Why is a thread pool so common if it has the slow-client problem?
Because the problem is bounded and visible (queue length, active workers), the code is sequential, every language has one, and the fix — timeouts plus a front proxy that absorbs slow clients — is well understood. Most production Java, Python and Ruby services are V3 behind an nginx that is V5.
▸Why does non-blocking I/O alone (V4) not solve anything?
Non-blocking changes what a call does when it cannot proceed — return instead of sleep — but not how you learn when it can proceed. Without a readiness or completion mechanism the only way to find out is to ask repeatedly, which is the busy loop. Readiness APIs are the actual invention.
Server evolution V0–V5
listen(fd)
for (;;) {
c = accept(fd) // blocks
req = read(c) // blocks on a slow client
write(c, handle(req))
close(c)
}How it fails
What the failure looks like from inside real software.
- V1 in production behind a load balancer: one client that opens a connection and sends nothing makes the health check time out; the balancer marks the instance dead.
- V2 with unbounded threads: a connection flood leads to
pthread_create: Resource temporarily unavailable(EAGAIN) onceulimit -uorthreads-maxis hit, and the accept loop dies with it. - V3 with an unbounded queue: clients get accepted and then wait minutes in the queue; they time out and retry, which adds to the queue. Latency grows until restart.
- V5 with one synchronous file read or a JSON.parse of a 50 MB body in a handler: every connection’s latency spikes to the duration of that call. See Timers fire late, health checks fail, one core at 100% in the challenges.
- Any version with
listen()backlog left at a tiny value: under a connection burst the kernel drops SYNs or refuses, and clients see connect timeouts while the server appears idle.