Break the OS: Predict, Break, Diagnose

Each button injects one fault into the running model from The OS Simulator: Cores, Processes, RAM, I/O and Locks. The discipline is the same every time. Predict: before pressing, write down which panel will move first and in which direction. Break: press it and watch. Diagnose: using only what the panels show — not the label on the button — answer the diagnostic question for that fault. Then read the “real symptom” column and find the same shape in the tool output you would see on a Linux host.

The point of the diagnostic question is that real failures do not come with a button label. “High CPU” is four different faults (The OS Debugging Playbook: Four Symptoms, Fourteen Causes); a hung process is three. The simulator lets you learn the *distinguishing* observation for each while you know the answer, so that in production you can run the distinction the other way round.

Each row is one button. The simulator column is what the panels do; the real column is what you would see on a Linux host; the question is what you must be able to answer from evidence alone.

Read the rows in pairs. Exhaust memory and Spawn 10,000 threads both raise memory (frames vs stacks) but differ in CPU; Create deadlock and Block disk I/O both stop a process cold but differ in the state letter; Exhaust file descriptors and Trigger stack overflow both fail a single operation but one leaves the process alive and one kills it. The diagnostic question in the last column is always the observation that separates the row from its neighbour.

Exhaust memory is Memory Pressure, Swap and the OOM Killer end to end: reclaim first (page cache dropped, then anonymous pages swapped), the thrashing knee, then the OOM killer choosing by oom_score — largest RSS wins, adjusted by oom_score_adj. In a container it is the cgroup OOM killer (Containers Are Processes With the Kernel’s View Narrowed) and the host is fine. The log line names the victim, its total-vm and anon-rss; a victim whose anon-rss had been growing linearly for days is a leak, one that plateaued is undersizing.

Create deadlock is Deadlocks: four conditions, and the observation that identifies it is a cycle in the wait-for graph (Cycle Detection). The simulator draws the graph; in production you build it from a thread dump by hand. Spawn 10,000 threads is Context Switching plus per-thread memory — each thread reserves a stack (8 MB virtual by default with glibc, committed on touch), so 10,000 threads is 80 GB of virtual address space and the scheduler’s run queue is now a large data structure. The C10K: Ten Thousand Connections, Then a Million lesson is the cure.

Block disk I/O is the D state from Process States and Follow a File Read: a thread inside the kernel waiting for a device cannot be interrupted, even by SIGKILL, until the I/O completes — which is why the load average rises while CPU sits idle (Linux counts D-state tasks in load) and why a dying NFS mount can make a machine look hung. Exhaust file descriptors is File Descriptors at its limit: ulimit -n (often a 1024 soft default), EMFILE for the per-process limit, ENFILE for the system-wide one, and the near-universal cause — sockets the peer closed that the application never did (too-many-open-files). Trigger stack overflow is Stack Overflow: the guard page below the stack mapping turns runaway recursion into a SIGSEGV, and language runtimes convert it into a catchable error only because they check depth before the guard page is reached.

The reason to drill the diagnostic questions is that each fault has a twin with a different fix. Exhausted memory and a large legitimate cache both fill RAM; only one is a bug. A deadlock and a thread blocked on a dead database both look like a hung process at 0% CPU; D vs S and the thread dump separate them. 10,000 threads and an infinite loop both show 100% CPU; %us vs %sy and vmstat’s cs separate them. EMFILE from a leak and EMFILE from 2,000 legitimate connections against a 1024 limit have opposite fixes. A stack overflow and a null dereference are both SIGSEGV; the fault address and the backtrace separate them.

Run each button, then its twin from the playground’s controls (add a large cache instead of a leak; add I/O instead of a lock; add legitimate connections instead of leaking them), and check that you can tell them apart from the panels alone. That skill is the whole of The OS Debugging Playbook: Four Symptoms, Fourteen Causes.