From Symptom to Root Cause

A symptom is what someone experiences: checkout feels slow, the export never finishes, the mobile app times out. A signal is the telemetry that corresponds to it. A measurement is a specific reading of that signal — a number, a percentile, a window. Only then does a hypothesis become possible, and a hypothesis is a claim about *mechanism*: "requests are queueing for database connections because the pool is smaller than the concurrency".

Evidence is the measurement that discriminates between hypotheses. This is the step most investigations skip: they look for data consistent with their theory rather than data that would rule it out. If the pool-saturation hypothesis is right, wait-time-for-connection should be high while database CPU stays low. If database CPU is also pegged, the hypothesis is wrong and you have learned something in one reading.

Root cause is the mechanism you can explain end to end without hand-waving. Change follows. Validation is the same measurement from step three, taken again. Regression check is the guard that stops the same failure returning silently in three months. An investigation that stops at "we restarted it and it went away" has completed none of the last three steps.

The single highest-leverage habit in incident debugging costs thirty seconds: before opening the next dashboard, write down what you expect to see and what it would mean. Not because writing is magic, but because it forces the hypothesis to be falsifiable and it makes you notice when you are pattern-matching instead of reasoning.

Without it, confirmation bias runs unopposed. Every dashboard has some line that is moving; an investigator with an unstated theory will find one that fits. With the expectation written first, "database CPU is at 30%" is a *refutation* of the database theory rather than an ambiguous data point that gets quietly reinterpreted.

This also fixes the handoff problem. An incident that changes hands at shift boundary is a disaster when the state is "we looked at some things". It is manageable when the state is a numbered list of hypotheses with each one marked confirmed, refuted, or untested, and the next measurement named.

Two failure modes bracket this process. Stopping too early gives you a mitigation mistaken for a root cause: the restart worked, the symptom is gone, and the mechanism is still there waiting. Stopping too late means chasing an interesting mechanism past the point of usefulness while the incident is still burning.

The practical split is mitigate first, diagnose fully second — but only if the mitigation is recorded as a mitigation. A restart that clears a memory leak buys time; it is not the fix, and the incident should not be closed on it (see Memory Leaks: Growth That Does Not Come Back). Writing "MITIGATED, root cause unknown" is a genuinely different statement from "RESOLVED", and the distinction is what stops the same page firing next week.

The stopping rule for the diagnosis itself: you can explain the causal chain from the change to the symptom, every link is supported by a measurement, and you can predict what *else* that mechanism would cause — and then confirm it. That last part is the strongest available check. If the theory is "the retry loop tripled outbound load", the theory also predicts elevated provider error rates and a specific pattern in retry-attempt counters. Checking a prediction the theory did not need to make is how you catch a plausible-but-wrong root cause.