Combined Failure Simulator: Break a Layer, Watch It Propagate

The simulator models a client application, its kernel (descriptor table, socket buffers, TCP state), the network between (loss, delay), and a server with the same three layers in reverse. Every control injects one fault into one layer; the display shows the fault propagating as state changes — queue depths, window sizes, thread states, error codes — until it reaches the user as a symptom. The numbers are simulated: the timings and sizes are chosen to make the propagation visible, not measured from a real system. The propagation *paths* are real, and they are what to learn.

The lesson is in the direction of travel. Faults in the application propagate down and across: a blocked thread stops draining a buffer, the buffer’s fullness becomes a window, the window crosses the network as an ACK field, and it stops a thread on the other machine. Faults in the network propagate up: a lost packet becomes a retransmit timer, then a stalled send buffer, then a blocked write(), then a request timeout in user code. Learning to run these chains in both directions is the skill the Capstone: Three Seconds from Warsaw tests.

Each row of the matrix is one control: where the fault lands, how it propagates, what the user experiences, and the command that confirms it on a Linux host. The confirming tool is chosen to answer one question — "is the fault in this layer?" — because that is how a real diagnosis proceeds (see The OS Debugging Playbook: Four Symptoms, Fourteen Causes and The Tools, and Which Layer Each One Answers). Read the rows as chains, not as facts.

Two of the controls deserve a closer look because their propagation crosses domains twice. Block thread on the server: the handler thread stops calling read(); the server kernel’s receive buffer for that connection fills; the window advertised to the client shrinks to zero; the client’s kernel stops sending and starts zero-window probes; the client’s send buffer fills; the client’s write() blocks (or returns EAGAIN, or its stream returns false); the client’s request times out at the application layer. Six state changes, three layers, two machines — from one thread doing CPU work. Drop packets: the client’s segment is lost; the server never ACKs; the client’s retransmission timer (RTO, minimum 200 ms on Linux) fires and resends with cwnd collapsed; on repeated loss the RTO doubles each time; the send buffer fills behind the unacknowledged bytes; write() blocks; the request slows from milliseconds to seconds with no error until a timeout — the classic "the network is fine, ping works" incident (see Packet Loss: Duplicate ACKs, Fast Retransmit and the RTO and the challenge A 1 Gbit/s link, and the backup crawls at 9 Mbit/s).

The ladder below is the "Block thread" chain end to end, with the state that changes at each rung. Notice that every rung is observable with a different tool on a different machine, and that an engineer who only has one of those tools will name the wrong layer: the client team sees a network stall; the network team sees a zero window from the server and a healthy path; the server team sees a busy thread and a healthy socket. All three are right about their layer and wrong about the cause unless they read the chain.

The general procedure: find the rung where the state is at its limit — a buffer at its size, a window at zero, a descriptor count at ulimit, a queue at its backlog — then ask what drains that state and why it stopped. The drain of a receive buffer is a thread; the drain of a send buffer is the network and the far window; the drain of an accept queue is accept(); the drain of a retransmit queue is an ACK. Whatever should be draining it and is not is the next rung up.

The transcript below is the same chain seen from the two hosts with the tools from the matrix. The habit to build is to run both sides before naming a cause: a stalled connection has a sender and a receiver, and the kernel on each records which queue is full. When the queues are empty on both sides and the request is still slow, the fault is above the kernel (the application is slow to respond, not slow to read) or in a layer the kernel cannot see (DNS, TLS, a proxy).

The Break the OS: Predict, Break, Diagnose and Network Failure Lab: Predict, Inject, Observe, Diagnose simulators inject faults in one domain each; this one exists because production incidents do not respect the boundary. The capstones for each domain — Capstone: A Server With 50,000 Concurrent Connections and Capstone: What Happens When You Visit https://example.com — and the Capstone: Three Seconds from Warsaw ask you to do this without the simulator’s labels.