Alerting & Operational Signals

Using observability rather than building it: alerts that demand action, symptom-based paging, dashboards an operator can act on, and the cost of noise.

Using Observability, Not Building It

The operator's path from a page to a hypothesis: alert, dashboard, trace, logs — and what each hop is actually for.

Q · The instrumentation is already there. What does an operator do with it in the first ten minutes of an incident?
An Alert Should Demand Action

The rule that decides what is allowed to page a human: if there is no action a person would take right now, it is not an alert.

Q · What earns the right to wake someone up, and what does not?
Alert on Symptoms, Not on Causes

"Users cannot check out" beats "CPU is 81%" — with the honest exception of infrastructure conditions that have a specific, immediate response.

Q · Should the pager fire on what users experience, or on the machine conditions that might produce it?
Alert Fatigue

Noise trains responders to ignore the pager, and the cost is paid on the one night the alert was real.

Q · What does a noisy alert set actually cost, given that each individual false page is only a minor annoyance?
Dashboards an Operator Can Act On

Built around the questions an incident asks, in the order it asks them — not around everything the system can emit.

Q · What should be on the screen an operator opens at 3am, and what should not?
Deploys on the Same Timeline as the Symptom

The highest-signal overlay there is: a vertical line at each release, drawn across the error rate graph.

Q · Why is "what changed?" so hard to answer during an incident, when every change was recorded somewhere?