Using Observability, Not Building It
The operator's path from a page to a hypothesis: alert, dashboard, trace, logs — and what each hop is actually for.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The instrumentation is already there. What does an operator do with it in the first ten minutes of an incident?
Teams buy or build an observability stack and then discover that having the data and being able to answer a question with it under time pressure are different capabilities, learned separately.
Instrument everything, ship it to the platform, and operators will be able to debug production because the data is there.
The data being present says nothing about it being reachable in the ninety seconds before someone starts guessing. An operator who has to remember a query language mid-incident will instead restart something.
- The data being present says nothing about it being reachable in the ninety seconds before someone starts guessing. An operator who has to remember a query language mid-incident will instead restart something.
- Without a path — a default first screen, a default second question — every incident starts from a blank search box, and two operators investigating the same page take two unrelated routes.
- Instrumentation coverage is measured at build time; the thing that matters is whether the specific question this incident raises can be asked at all, which is only discovered during the incident.
- The signal that would have answered it is frequently there and unfindable: a metric with the wrong labels, logs at the wrong level, traces sampled away exactly on the erroring requests.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The three signal families answer different shapes of question, and the operational skill is knowing which shape you currently have. Metrics answer "how much, how often, how bad, over time" — cheap, aggregated, no per-request detail. Traces answer "where in the request did the time or the failure go". Logs answer "what exactly happened to this one thing".
- That gives a natural narrowing order. Metrics establish scope and shape: how many users, since when, which endpoints, is it rising. Traces establish location: which hop in the call path changed. Logs establish detail: the actual error, the actual identifier, the actual input.
- Running that order backwards is the most common operational mistake. Starting in logs during a broad outage means reading thousands of lines to learn something one metric graph would have shown in a second — and it burns the minutes when user impact is still growing.
- Underneath all three is one operational question the signals only serve: what changed? Signals describe the current state; incidents are caused by a transition. The signal work exists to locate the transition (Change Correlation).
- The Observability & Performance domain owns how these signals are produced, stored, sampled and costed. This lesson owns only the order in which an operator consumes them.
Page to hypothesis, in the order that costs least
This is the operational path, not the observability pipeline. Each hop narrows the search space, and each one is cheap only because the previous one already reduced it.
The failure mode of every hop is the same: skipping it. An operator who jumps to logs is searching an unbounded space; an operator who jumps to a fix is guessing.
- 1Alert
Tells you that something a user cares about is wrong, and links to where to look.
fails by Firing on a cause rather than an effect, so nobody can tell whether it matters.
evidence The alert text states the user-visible symptom and carries a runbook link.
- 2Scope
Establishes how many, since when, and which surface — from metrics.
fails by Missing labels, so "all users" and "one tenant" look identical.
evidence You can state impact in a sentence without a caveat.
- 3Change
Lists what was deployed, flagged or configured in the relevant window.
fails by Changes not recorded on the same timeline as the symptom.
evidence A single view shows deploys against error rate (Deploys on the Same Timeline as the Symptom).
- 4Locate
Finds which hop in the request path degraded — from traces.
fails by The failing requests were sampled out, or the path is not instrumented across a boundary.
evidence A trace of a failing request, not a healthy one.
- 5Detail
Reads the actual error and the actual input — from logs.
fails by Unstructured lines that cannot be filtered to the affected requests.
evidence You can filter to one request id end to end.
- 6Hypothesis
A statement specific enough to imply a mitigation.
fails by Stopping at a correlation and acting on it.
evidence The mitigation you chose follows from the statement rather than from habit.
Mitigation does not wait for the end of this path. If scope shows growing user impact and change shows a recent deploy, you roll back at step three and finish the investigation afterwards (Stop the Harm Before You Understand It).
Which signal answers which question
Operators lose time by asking a signal a question it structurally cannot answer. A metric cannot tell you why one customer's request failed; a log line cannot tell you whether the rate is rising.
The depth of each of these lives in Observability & Performance. What matters operationally is the second and third columns.
| Signal | Answers | Cannot answer | Operational use |
|---|---|---|---|
| Metrics | How much, how often, how bad, trending which way | Anything about one specific request | Scope and severity; the first sixty seconds |
| Traces | Where in the call path the latency or error appeared | How many users are affected | Locating the degraded hop or dependency |
| Logs | What exactly happened to this one request or job | Whether it is widespread | Naming the error once you know where to look |
| Events / change feed | What was deployed, flagged, scaled or configured, and when | Whether it caused anything | The highest-prior-probability suspect list |
| Synthetic probes | Whether the user path works from outside your network | Which internal component broke | Confirming impact and confirming recovery |
| Profiles | Where CPU or allocation is actually being spent | When it changed, usually | Rare in incidents, decisive in slow-burn regressions |
What to open first, and why it is a decision
The right first screen depends on what the alert already told you. An alert that fires on a user-visible symptom has done the scope work; one that fires on a saturation threshold has not.
None of these options is wrong in general. Each is wrong in some situations, and knowing which situation you are in is the skill.
Which first move buys the most information for the next sixty seconds?
when The alert is symptom-based and you do not yet know scope or shape.
cost Requires that the dashboard exists and is maintained; a stale one misleads faster than no dashboard.
when The symptom started sharply rather than drifting — a step change almost always follows a change.
cost Anchors you on the most recent change, which is sometimes innocent and always suspicious-looking.
when You already know the scope and the symptom is latency or a dependency error.
cost Costs minutes if error-biased sampling is not configured and you have to find a failing example.
when Several unrelated services degraded at the same moment.
cost Provider status pages lag real incidents; absence of a posted incident is not evidence of health.
when This alert has fired before and someone wrote down what it meant.
cost Only as good as the last time it was corrected (Runbook Anti-Patterns).
How to do it properly
Most important first.
- Establish scope before cause. Answer "how many users, since when, getting what" before opening a single log line — that is also the answer the incident channel needs first (Telling People What Is Happening).
- Make the first screen deterministic. Every alert should link to the dashboard an operator should open, and every service should have a runbook entry saying what to look at (Runbooks).
- Ask "what changed" in parallel with reading signals, not after. Deploys, config changes, flag flips and infrastructure events are the highest-prior-probability suspects (Deploys on the Same Timeline as the Symptom).
- Follow one failing request end to end at least once per incident. Aggregates hide the failure mode; a single trace usually names it.
- Treat an unanswerable question as an action item. "We could not tell whether the errors were one tenant or all tenants" is a missing label, and it is a defect in the same sense as a bug (Action Items That Change the System).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing contains it directly — poor operational use of signals does not cause the outage, it lengthens every outage you have.
What can go wrong
- The observability platform is degraded by the same event you are trying to investigate — a shared dependency, a network partition, or simply the log volume the incident itself produced.
- Sampling drops the interesting traces. Error-biased sampling exists precisely for this and is often not configured.
- Dashboards showing infrastructure health look green throughout, because the failure is in a dependency or in correctness rather than in the machine.
- The operator finds a correlation, believes it, and mitigates the wrong thing. Coincident graphs are not causation (Reconstructing What Actually Happened).
- A high-cardinality query executed during an incident makes the monitoring system itself slow for everyone else responding.
- "We have observability, so we are ready for incidents." Readiness is the ability to answer a question fast, which is a rehearsed skill, not an installed product.
- "More dashboards means more visibility." Past a point it means longer to find the right one (Dashboards an Operator Can Act On).
- "The logs will tell us." They will, eventually, and eventually is the wrong budget while users are failing.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- An operator who has never seen this service before can go from page to scope statement using only the links in the alert.
- During the last incident, the question "how many users are affected" was answered from a dashboard rather than estimated.
- Traces exist for the failing path, and at least one was actually read before the mitigation was chosen.
- Observability changes are ordinary deploys and roll back the same way — but note that reverting an instrumentation change removes evidence you may still need, so prefer disabling an expensive query or reducing sampling over removing the signal.
- If the monitoring stack itself is the thing failing, the fallback is the platform's own status: load balancer metrics, cloud provider health, and direct probes from outside your infrastructure.
- Automate the mapping from alert to context: the alert payload should carry the dashboard link, the runbook link, the owning team and the recent deploys.
- Automate the mechanical first queries — recent deploys, error rate by endpoint, dependency health — so the operator reads results rather than composes queries.
- Do not automate the hypothesis. Correlations are cheap to compute and easy to believe; deciding which one to act on is judgement under uncertainty and belongs to a human (The Automation Trap).
- A prescribed investigation path speeds up the common case and can anchor an operator on the wrong track in the uncommon one. It is still worth it, because the common case is common.
- Everything that makes signals more useful in an incident — more labels, more traces, more log detail — costs money continuously and is spent mostly on days when nothing happens.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe narrowing order — scope, then location, then detail — holds for any stack, because it follows from what each signal family can and cannot express, not from any vendor's product.
- TOOL-SPECIFICWhich of these is cheap depends entirely on the platform: some make high-cardinality metric queries routine and traces expensive, others the reverse. That changes which hop you lean on, not the order.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — rehearsing the investigation path before an incident needs it, so the first time an operator uses these tools is not at 3am.