Network Failure Lab: Predict, Inject, Observe, Diagnose

The lab is a simulated client → DNS → router → server path with a request in flight; every number it shows is simulated, chosen to be plausible, not measured. The value is not in watching it break but in committing to a prediction first. For each button, write down three things before pressing it: the symptom at the application (stall, slowness, an error — which error?, and what the client’s retry logic will do), the mechanism layer by layer (which packet is affected, what the transport does about it, what the application sees), and the confirming tool — the one command whose output would prove this cause and rule out the others.

Then inject, observe, and diagnose as if you did not know which button was pressed. The point is calibration: after a few rounds, "the request took 4 seconds and then succeeded" and "the request failed instantly with a certificate error" and "the request hung for 130 seconds" are three different diagnoses you can make from the symptom alone. Why Can’t I Connect? is the procedure; this lab is the practice.

The matrix is the lab’s answer key. Cover the right-hand columns and fill them from the left before you read them.

Two of the buttons produce no error at all, only time, and they are the hardest to diagnose in production because every tool reports "success". Latency taxes each round trip equally. A fresh HTTPS request needs a DNS lookup, a TCP handshake, a TLS handshake and the request itself — three to four RTTs before the first byte — so adding 200 ms adds ~0.8 s to time-to-first-byte, and a page of 40 assets on a cold HTTP/1.1 connection pool becomes unusable. The server’s CPU is idle throughout; this is the "slow request, idle server" interview question and the subject of Bandwidth vs Latency. Transfers suffer twice: cwnd grows once per RTT, so slow start takes four times longer to fill the pipe, and the bandwidth-delay product grows so a fixed receive window caps throughput at window / RTT.

Loss is worse than its percentage suggests. TCP interprets every loss as congestion and halves its sending rate, so throughput scales with 1/√p: 5% loss on a 50 ms path bounds a connection to roughly 1.3 Mbit/s no matter how fast the link is. Small requests that fit in a few segments often succeed after one retransmission; large downloads crawl and time out. Packet Loss: Duplicate ACKs, Fast Retransmit and the RTO and Congestion Control: Protecting the Network show the wire-level dance; the packet-loss challenge is the incident version. The confirming evidence is loss that persists to the destination in mtr, not loss at one intermediate hop (traceroute: Discovering the Path Hop by Hop explains why the distinction matters).

The hard failures are distinguishable by when and how they fail. A DNS failure fails before any connection — the error text says "resolve" or "not found", and curl -v never prints "Trying <IP>". A blocked port and a dead router both fail by silence: the SYN goes out and nothing returns, so the client waits its full connect timeout. They are separated by traceroute (the router failure stops the trace one hop short; the firewall lets the trace reach the host but the port probe hangs) and by whether ping still works (a DROP rule on one port usually leaves ICMP alone). A server reset fails instantly with a RST: ECONNREFUSED at connect time means nothing is listening, ECONNRESET mid-request means the process died or a middlebox cut the flow.

Retry behaviour is the other tell. Retries against DNS failure fail identically until a cache expires or the resolver recovers, and different clients fail at different times as their cached answers age out. Retries against a blocked port cost the full timeout each time, which is how one blocked backend consumes a thread pool. Retries against a refused connection are cheap and immediate, which is how a crash-looping backend gets hammered with thousands of connections per second by a gateway that never backs off. The TCP Debugging: Reading the Handshake on the Wire and DNS Debugging: Who Answered, and With What? procedures name the check for each.

The expired certificate is the cleanest failure in the lab: DNS, TCP and the first half of TLS succeed, and then the client rejects the certificate with a deterministic error that no retry will fix. It is also the most preventable — certificates expire at a known second — and the one most often missed by monitoring that checks HTTP without TLS. The TLS Debugging: Why the Certificate Is "Invalid" lesson has the full set of reasons a certificate is "invalid"; the expired-certificate challenge is the incident version.

One real failure deserves a mention even though it is not a button: the MTU black hole. A link on the path has an MTU below 1500 (a VPN tunnel, PPPoE, an overlay network); large packets with the DF bit set are dropped there and the router sends ICMP Fragmentation Needed — which a firewall filters. Small packets (the handshake, a small request) pass, so the connection opens and then hangs the moment a full-size segment is sent: a login page loads, the dashboard behind it never does. tcpdump shows the same 1448-byte segment retransmitted with no ACK progress; ping -M do -s 1472 (Linux) fails where -s 1400 succeeds. The fix is MSS clamping on the tunnel or allowing ICMP type 3 code 4 through. The MTU black-hole challenge attached to this lesson is this incident.