How does one server handle 10,000 (or 100,000) concurrent connections?
“Explain what limits a server’s concurrent connections and how a modern server reaches 100K. Be specific about what each connection costs in the kernel and in the process.”
What this tests
- Per-connection resource accounting: descriptors, buffers, threads, memory
- Why thread-per-connection fails and event-driven I/O succeeds
- Knowledge of epoll/kqueue/IOCP and readiness vs completion
- Kernel limits and where they are configured
Answers by level
Read the beginner answer first and notice what is missing.
Each connection is a socket: a file descriptor, kernel send and receive buffers (from a few kB up to MBs, auto-tuned), and TCP state. 100K connections need the descriptor limit raised (ulimit -n, fs.file-max) and a few GB for buffers — that part is fine.
The problem is what the process does per connection. A thread per connection means 100K threads: each with a stack (8 MB virtual, tens of kB resident), a scheduler entry, and context-switch cost, and the scheduler spends its time switching rather than working. That is the C10K problem.
The solution is I/O multiplexing: one thread asks the kernel "which of these 100K sockets is readable?" with epoll_wait() (Linux), kqueue (BSD/macOS) or IOCP (Windows), and handles only the ready ones. Cost is proportional to active connections, not total. Nginx, Node, Go’s netpoller and Java NIO all work this way, with a small pool of threads for CPU work.
Green flags · Red flags
- Costs out a connection in descriptors, buffers and memory
- Explains why threads do not scale to 100K
- Names epoll/kqueue/IOCP and readiness
- Mentions accept queue and port limits
- "Node is single-threaded so it can’t"
- Thinks a thread pool of 100 solves it without multiplexing
- Has never heard of file descriptor limits
- Confuses concurrent connections with requests per second
Follow-up questions
Scenario
ps shows 8,000 threads. What is happening and what would you change?