Alerts Worth Waking Someone For

The distinction that fixes most alerting is between symptom-based and cause-based conditions. A symptom alert says "users are experiencing failure": error rate is up, latency is past the threshold, the queue's oldest message is older than the SLA for that work. A cause alert says "a thing that sometimes precedes failure is happening": CPU is high, a pod restarted, disk is 85% full.

Symptom alerts have two properties cause alerts lack. They catch failures you did not anticipate — you cannot enumerate every cause of user-visible breakage, but you can measure breakage directly. And they do not fire when the anticipated cause occurs without the consequence, which is most of the time. High CPU on a service that is comfortably serving its traffic is not a problem; it is a well-utilised machine. Paging for it trains people to dismiss pages.

Cause signals are not worthless — they are what you look at *after* a symptom alert fires, to find out which layer is responsible. That is a dashboard job, not a paging job (Dashboards Built Around Questions). The narrow exception is the predictable, slow-moving, irreversible cause: a disk that will be full in four hours, a certificate expiring on Thursday, a quota that will be hit at the current growth rate. These earn a ticket or a low-urgency notification because the lead time is the entire value — but they still should not wake anyone at 03:00.

Before a rule is allowed to wake someone, it should have an answer to four questions, and the absence of any one of them is grounds for demoting it to a ticket or deleting it. Is a user being harmed? If not, it is not a page. Is it urgent? If it can wait until working hours without getting materially worse, it is a ticket. Is it actionable? If the runbook says "wait and see" or the only fix is a change that needs a code review, waking someone buys nothing. Is it novel? If this alert fires alongside three others for the same root cause, it should be suppressed or grouped, or the responder spends their first ten minutes triaging notifications rather than the incident.

The runbook requirement deserves emphasis because it is the one most often waived. Every paging alert needs a link to a document that says what this alert means, what to check first, what the common causes are, and what actions are safe. Without it, alert quality depends on which person is on call, and the team's worst night is whenever its most experienced engineer is on holiday. Writing the runbook also functions as a design review of the alert: if you cannot write down what to do about it, that is the alert telling you it should not page.

Severity should map to response, not to how bad the condition feels. Two tiers handle almost everything: page (wake a human now) and ticket (someone looks during working hours). Adding a third and fourth tier mostly produces arguments about classification, and in practice everything below the top tier gets treated identically anyway.

Working through the The Four Golden Signals gives a compact and nearly complete paging set for a request-driven service. Errors and latency become budget-burn alerts against the SLO, which is the bulk of the value (Burn-Rate Alerts: How Fast Is the Budget Going?). Traffic is almost never a paging condition by itself — high traffic is success, low traffic is only alarming if it means requests are not arriving, which is worth one carefully tuned rule for total loss of traffic. Saturation is where judgement is needed: it is a cause, so it usually belongs on a dashboard, except where saturation is irreversible or has a long lead time.

Two additions complete the set. A freshness or age alert for anything asynchronous — the oldest unprocessed message, replication lag, the age of the newest row in a pipeline — because a queue can be quietly failing while every request-level metric stays perfect (Depth Is Not an Emergency; Age Is). And a watchdog: an alert that fires continuously and is routed to a check confirming it is still firing. Without one, a monitoring outage looks exactly like perfect health, which is the failure mode nobody discovers until afterwards.

A useful sanity check on any alert set is to ask what would have fired during each of the last five incidents, and what fired when nothing was wrong. A set that would have caught all five and stayed quiet otherwise is working. A set that caught two and fired sixty times is training the team to ignore it, which is worse than having no alerts at all — because a silent system prompts people to check, while a noisy one persuades them not to.