Failure & Recovery in Production
9 lessons. Every one names the guarantee it claims, what a node can know, and how it fails.
The five-step spine of every distributed incident. Most teams execute three of them, declare victory when the error rate returns to baseline, and leave derived state permanently wrong — because nothing in the dashboard was ever measuring it.
Q · The errors have stopped and the graphs are green. Is the system actually correct again?
The recommendation service is down and checkout still works. That outcome is not a virtue of the code — it is the result of somebody having decided, in advance and in writing, which dependencies are on the critical path for which feature. Nobody makes that decision well during an incident.
Q · When a dependency fails, which parts of my product should keep working — and did anyone decide that before today?
An experiment on a live system has six parts: a hypothesis, a measured steady state, an injected failure, an observation, an abort condition and a conclusion. The abort condition is not optional paperwork — it is the entire difference between an experiment and an outage you caused on purpose.
Q · How do I test a failure assumption on a real system without the test becoming the incident?
B slows down. A’s threads sit waiting on B. A saturates and starts timing out. A’s clients retry. Load on A rises. Services that depend on A begin to fail. The defining property is the positive feedback: every step of the failure response increases the load that caused the failure.
Q · Why did one slow dependency take down five services that were all healthy a minute ago?
Draw the dependency graph and, for each node, ask what stops working if it disappears. The answer is almost always worse than the team expects — because the graph everyone reasons from shows logical calls, and the failures propagate through shared infrastructure that appears on no diagram.
Q · If this component disappears right now, what exactly stops working — and what have I forgotten?
The discipline is deliberately introducing controlled failures to test assumptions you already hold about how the system behaves. Every word carries weight: deliberate, controlled, to test an assumption. Remove any one of them and what remains is an outage with a fashionable name.
Q · What is chaos engineering actually for, and what separates it from causing an outage on purpose?
Latency, packet loss, node crash, dependency error, disk full, network partition, clock skew. The first four are easy to inject and mostly confirm what you expect; the last three are hard to inject and are where the assumptions actually break. Difficulty and value point the same way, which is why most programmes only ever test the easy half.
Q · Which faults can I actually inject, and which of them will tell me something I do not already know?
Which request? Which service? Which region? Which version? Which dependency? Which state transition? Six questions, in order. If your system cannot answer them for a single failing request, every incident is solved by guessing, and the guesses are shaped by whoever spoke first.
Q · One request in ten thousand fails. How do I find out why, instead of guessing?
The obvious way to reconstruct what happened is to merge the logs and sort by time. It does not work: clocks on different machines disagree by more than the intervals you are trying to order, so the merged view can show an effect before its cause. Identifiers that carry causality are the answer, and this is where the time module pays off.
Q · I have logs from three services. How do I reconstruct what actually happened, in order?