Capstone: A Server With 50,000 Concurrent Connections
Explain, layer by layer, how a Node.js, Python or C++ server holds 50,000 open connections — process, threads or event loop, sockets, descriptors, per-connection memory, the scheduler, system calls, the kernel network stack, buffers, the I/O model and CPU — then diagnose the five faults the simulator injects.
The problem
The layer walk
Walk the stack top to bottom and, at each layer, answer three questions: *what exists per connection*, *what it costs*, *what breaks here*. The ladder is the answer key; the interactive lets you expand each rung. Numbers are Linux and order-of-magnitude — a well-configured server holds 50,000 idle connections in a few hundred MB of kernel memory plus whatever the runtime allocates per connection, and the interesting question is always which layer’s cost is dominant for *your* server.
Notice which rungs scale with the connection count and which scale with the *request rate*. Descriptors, socket objects and per-connection user-space state scale with connections and are paid while the connection is idle; syscalls, scheduler activity, CPU and NIC interrupts scale with traffic and are nearly free for an idle connection. A server that is “fine at 50,000 connections” may be fine because they are idle; the same 50,000 connections each sending one request per second is 50,000 recv/send pairs per second and a very different machine.
- Processone address space, one descriptor table (needs `ulimit -n` ≥ ~60,000), one cgroup↓
- Threads / event loopNode: 1 loop thread + 4 libuv pool threads · CPython: asyncio on 1 thread, or threads under the GIL · C++: N reactor threads, one per core↓
- I/O model`epoll` (edge- or level-triggered) waits on all 50,000 descriptors in one call; `io_uring` on newer kernels↓
- Sockets & descriptorsone descriptor + one socket object per connection; ~2–4 kB of kernel structs each while idle↓
- System calls`accept4`, `epoll_ctl`, `epoll_wait`, `recv`, `send`, `close`; at 50k connections the syscall rate, not the connection count, drives CPU↓
- Per-connection memorykernel buffers grow on demand (up to 6 MB rcv / 4 MB snd, autotuned); user-space state: Node ~5–10 kB of JS objects, CPython ~5 kB of transport/protocol objects, C++ whatever you allocate↓
- Scheduleran event-loop server has one runnable thread per core at most; a thread-per-connection server has 50,000 and switches constantly↓
- Kernel network stackfour-tuple hash lookup per segment, TCP state machine, conntrack if NAT/firewall (`nf_conntrack_max` default 65,536 — right at the edge)↓
- NICinterrupts coalesced, multiple queues (RSS) spread segments across cores; ~50k idle connections generate almost no traffic, so this layer is idle until load
Per-runtime notes
Each runtime maps the same kernel primitives differently, and the differences decide which fault hits first.
The shared truth under all three is that the kernel offers exactly one scalable primitive — readiness notification via epoll (or kqueue, or IOCP on Windows, I/O Multiplexing: select, poll, epoll, kqueue, IOCP) — and the runtime’s job is to keep the runnable thread count near the core count while never blocking the thread that owns the epoll set. Everything in the per-runtime notes is a consequence: Node’s thread pool exists because file I/O cannot be made non-blocking on Linux without io_uring; CPython’s multi-process model exists because of the GIL; C++’s reactor-per-core exists because there is nobody to make that decision for you (How C++, JavaScript and Python Map onto the OS).
- Node.js (
scope: nodejs): one JavaScript thread runs the event loop over libuv’sepoll; every connection is an object plus aBufferor two, so 50,000 idle connections cost ~300–500 MB of heap and the default--max-old-space-size(~2–4 GB depending on version and RAM) is the ceiling to watch. Blocking file I/O and DNS go to a 4-thread pool (UV_THREADPOOL_SIZE), so a slow disk stalls file reads but not the sockets — until the pool is exhausted. One synchronousJSON.parseof a 50 MB body blocks all 50,000 connections. CPU beyond one core needsclusteror worker processes withSO_REUSEPORT-style sharing (Node’s cluster round-robins from the primary). - CPython (
scope: cpython):asynciouses theselectorsmodule overepoll; each connection is a transport + protocol object, ~5 kB. The GIL means one thread executes bytecode at a time, so the event loop is effectively single-core for Python work, and any CPU-heavy handler stalls every connection;uvloopreplaces the loop with libuv for 2–4× throughput. Threads (ThreadPoolExecutor) help only for blocking I/O and C extensions that release the GIL. Multi-core means multiple *processes* (Gunicorn workers,multiprocessing) each with their own loop, sharing the listening socket. Free-threaded builds (3.13+) change the GIL story but not the per-connection accounting. - C++ (
scope: cpp): nothing is decided for you. A reactor per core, each with its ownepollinstance and a share of the accepted sockets, keeps the runnable thread count at the core count; edge-triggeredepollwith non-blocking sockets andEPOLLEXCLUSIVEorSO_REUSEPORTavoids the thundering herd onaccept. Per-connection memory is whatever the connection struct holds — 200 bytes is achievable, and the kernel’s socket buffers dominate. The failure modes are yours too: a blockingread()on a disk file inside the reactor stalls that core’s connections, a missingEPOLLRDHUPleaksCLOSE_WAITsockets, and one lock shared by the reactors turns 32 cores into 4.
The five injected faults
The simulator injects one fault at a time into the layer walk. For each, name the layer, read the symptom, run the diagnostic and choose the fix. The matrix is the answer key; the discipline from Break the OS: Predict, Break, Diagnose applies — predict before you look.
The faults are ordered from the outermost layer inward, and their fixes are at different altitudes: a descriptor limit is a one-line configuration change, high CPU is a profiling exercise, memory pressure is an accounting exercise (bytes per connection × connections), slow clients are a design change (backpressure), and blocked disk is an architecture change (separate the disk-bound path). Recognising which altitude a fault lives at is what keeps a 3 a.m. fix from being a ulimit bump on a design problem.
| Injected fault | Layer | Symptom | Diagnostic | Fix |
|---|---|---|---|---|
| Too many open files | Process: descriptor table | accept4: EMFILE in logs; new connections refused; existing ones fine; connection count stuck at 1,024 or 65,536 | cat /proc/<pid>/limits, ls /proc/<pid>/fd | wc -l, ss -s; is it CLOSE_WAIT (leak) or legitimate load past the limit? | Leak: close on every error path, set EPOLLRDHUP/on('end') handlers. Limit: LimitNOFILE=200000, fs.nr_open, and stop accepting at a soft ceiling so the process degrades instead of failing |
| High CPU | Threads / event loop / syscalls | one thread (Node/CPython) at 100% and latency for all connections; or all reactor threads (C++) at 100% with flat throughput | top -H for which thread; %us vs %sy; perf top / py-spy top / --cpu-prof; strace -c for syscall storms (a level-triggered epoll spinning on a readable socket nobody reads is the classic) | CPU work off the loop (worker threads/processes); fix the busy-poll (edge-triggered + drain, or stop registering interest until you can read); add cores only when the profile is legitimate work |
| Memory pressure | Per-connection memory + kernel buffers | RSS climbing with connection count; page faults; then OOM kill (exit 137) or RangeError: heap out of memory (Node) | RssAnon trend vs connection count → per-connection cost; ss -tm for socket buffer memory; /proc/net/sockstat TCP: mem; heap snapshot diff | Shrink per-connection state (drop per-connection closures/buffers, pool them); cap socket buffers with SO_RCVBUF if autotuning over-allocates; raise the heap ceiling only after the per-connection cost is known; a hard connection limit |
| Slow clients | Socket send buffers + I/O model | send() returns EAGAIN; Send-Q grows on many sockets; responses buffered in user space grow RSS; with a blocking write, the thread stalls | ss -tan Send-Q and rwnd (ss -ti shows the peer’s window at 0 — a zero-window client); user-space write-queue length per connection | Backpressure: stop reading from the upstream/producer when the write queue is full (writable return value in Node, drain events; transport.pause_reading() in asyncio; EPOLLOUT gating in C++), per-connection write caps, idle/slow timeouts (What Happens When the Receiver Is Slow) |
| Blocked disk | Threads + I/O model (the one blocking call) | Node: file-serving requests hang while sockets stay responsive until the 4-thread pool is full; CPython/C++ with a sync read() in the loop: *every* connection stalls; thread in D state | ps -o stat,wchan shows D / io_schedule; iostat -x await; strace -p stuck in read(7, …) on a regular file; in Node, UV_THREADPOOL_SIZE saturation | Never block the loop on disk: thread pool or io_uring for file I/O, sendfile for static files (Zero-Copy: Serving a File Without Touching It), bigger UV_THREADPOOL_SIZE; separate the disk-bound path from the connection-bound path |
Putting the explanation together
A complete answer to “how does this server hold 50,000 connections?” runs top to bottom and names a number at every rung. *One process with a descriptor limit raised to 200,000. One event loop (or N reactors) using epoll to wait on every socket in a single syscall, so the runnable thread count is at most the core count and the scheduler is idle. One socket per connection, each a descriptor plus ~3 kB of kernel state plus buffers that grow only when data flows. Per-connection user-space state of a few kB (Node, CPython) or a few hundred bytes (C++), so 50,000 connections is hundreds of MB, not tens of GB. Syscalls only when a socket is ready — at 50,000 mostly idle connections, epoll_wait returns a handful at a time. The kernel network stack demultiplexes each segment by four-tuple hash to its socket; conntrack, if present, needs its table raised past 65,536. The NIC spreads interrupts across cores.* Then the failure story: each of the five faults lives at one rung, and the diagnostic for each is the observation at that rung.
The historical version of this question is C10K: Ten Thousand Connections, Then a Million — ten thousand connections was once hard because the only model was a thread or a process per connection, and Thread per Connection at 10,000 threads is exactly the switching regime from the The OS Simulator: Cores, Processes, RAM, I/O and Locks’s first experiment. The Event-Driven Server is the answer that made 50,000 routine; Combined Failure Simulator: Break a Layer, Watch It Propagate injects the same faults from the networking side; Capstone: Three Seconds from Warsaw adds a latency budget to the same server.
1server.on('connection', (sock) => {2 upstream.on('data', (chunk) => {3 const ok = sock.write(chunk) // false: the socket's send buffer and user queue are full4 if (!ok) upstream.pause() // backpressure: stop producing…5 })6 sock.on('drain', () => upstream.resume()) // …until the kernel has drained the send buffer7 sock.setTimeout(30_000, () => sock.destroy()) // a client that never reads should not hold state forever8})Key points
- Walk the layers: process → threads/event loop → I/O model → sockets/descriptors → syscalls → per-connection memory → scheduler → kernel stack → NIC — and name what each holds per connection.
- 50,000 connections is a descriptor-limit, buffer-memory and syscall-rate problem, not a thread problem, once the I/O model is
epoll. - Node: one loop thread, a 4-thread pool for files/DNS, a heap ceiling; CPython: asyncio under the GIL, processes for cores; C++: reactors per core, everything explicit.
- Too many open files → descriptor table; high CPU → loop/reactor threads and syscall storms; memory → per-connection state and socket buffers; slow clients → send buffers and backpressure; blocked disk → the one blocking call.
- Every fault has one rung and one observation; the fix is at that rung, not at the symptom.
Why does this exist?
Mechanisms are answers to constraints. Open each question before reading the answer.
▸Why can one thread hold 50,000 connections?
Because idle connections cost memory, not CPU. epoll lets one thread ask the kernel “which of these 50,000 have something for me?” in one call and touch only those, so the thread count is decoupled from the connection count.
▸Why does the runtime matter if the kernel primitives are the same?
The runtime decides how many threads exist, how much memory each connection’s state takes, whether CPU work can run in parallel, and which calls silently block the loop. The kernel offers the same epoll to all three; the failure that arrives first is the runtime’s choice.
▸Why is backpressure the fix for slow clients and not a bigger buffer?
A bigger buffer is a bigger bill for the same slow client: memory per connection times 50,000. Backpressure bounds the per-connection cost by making the producer wait, which is what the pipe and TCP already do one layer down.
Capstone: 50,000 connections
How it fails
What the failure looks like from inside real software.
- The service is deployed with the distribution’s 1,024-descriptor default and falls over at 1,000 connections, long before any interesting limit.
- A
JSON.parseof a large body — or any synchronous call — on the loop thread stalls 50,000 connections for 300 ms; p99 latency is unexplained until--cpu-profshows it. - Per-connection closures and buffers cost 40 kB each; 50,000 connections is 2 GB and the heap ceiling kills the process on a traffic spike.
- No write backpressure: a few thousand slow mobile clients make the server buffer responses in user space until it is OOM-killed — a memory incident caused by the network.
- Static files served with a blocking read inside the reactor; a degraded disk stalls every connection on that core and the incident is filed as “network”.
- conntrack table at its 65,536 default on the load balancer in front: connections beyond that are dropped and the server never sees them.