8 lessons

SLOs & Alerting

Turning user experience into a measurable objective: SLIs, SLOs and SLAs kept distinct, error budgets as a decision tool, alerts worth waking up for, burn rates, and dashboards built around questions.

Symptom→Signal→Measurement→Hypothesis→Evidence→Root Cause→Change→Validation→Regression Check

Every lesson below starts from an observable symptom and ends with the measurement that proves the fix worked. Numbers carry a label saying whether they were measured, estimated, simulated or invented to show a shape.

SLIs: Measuring What the User Actually Feels

An SLI is a ratio: good events over valid events. The hard parts are not the arithmetic — they are deciding what counts as good, what counts as valid, and where in the request path you measure, because each choice moves the number by more than most outages do.

Symptom · Every dashboard is green, CPU is comfortable, error rate reads 0.02% — and support is fielding complaints that the app "does not work". Nobody can say who is right, because nothing on the wall measures the user's experience.
SLOs: A Target, a Window, and a Reason
▶ lab

An SLO is an SLI plus a target plus a window: "99% of checkouts complete under 300ms over 28 days". The target is not an aspiration — it is a commitment about how much unreliability you are willing to pay to avoid, and 100% is always the wrong answer.

Symptom · Reliability discussions have no shared vocabulary. One person says the service is "pretty stable", another says it is "constantly breaking", and both are looking at the same graph. Every reliability-versus-features argument restarts from zero.
SLAs: The Promise With Money Attached

An SLA is a contract: a reliability promise to a customer with a defined consequence when you break it. It is deliberately looser than your internal SLO, it is written by people who are not on call, and alerting on it means you find out you owe refunds at the same moment as your legal team.

Symptom · The SLA says 99.9%, so the team sets the internal target to 99.9%, and now every breach of the internal objective is simultaneously a contractual breach with financial consequences. There is no room left to be merely degraded.
Error Budgets: Unreliability You Are Allowed to Spend

If the objective is 99.9%, then 0.1% of failure is not a defect — it is a budget. Treating it as something to spend rather than something to avoid turns every reliability-versus-velocity argument into an arithmetic question, which is the only version of that argument anyone ever wins.

Symptom · The reliability conversation is a standoff. Engineering wants to stop feature work and harden the system; product wants to ship. Both positions are asserted with conviction and neither is supported by a number, so the argument is settled by whoever is more senior.
Alerts Worth Waking Someone For

An alert is a claim that a human should stop what they are doing right now. `CPU > 80%` is not that claim — it is a fact about a machine that may or may not correspond to a user having a bad time. Alert on symptoms users feel; keep causes on dashboards where they belong.

Symptom · The on-call phone fires four times a night. Three of the four resolve themselves before anyone opens a laptop. The fourth is a real outage that arrives among the noise and gets acknowledged twenty minutes late.
Alert Fatigue: The Page Nobody Reads

Alert fatigue is not a morale problem, it is a detection failure. Every page that turns out to be nothing raises the probability that the next real one is acknowledged late, investigated slowly, or dismissed entirely — and the arithmetic that produces it is measurable.

Symptom · Nobody looks at the alerts channel. Pages are acknowledged and closed without investigation. When a genuine outage arrives, the first response is "is that another flapping one?" — and the answer takes ten minutes to establish.
Burn-Rate Alerts: How Fast Is the Budget Going?

Alerting on "error rate above 1%" picks a threshold with no relationship to what you promised. Alerting on burn rate asks a better question: at the current failure rate, how long until the error budget is gone? Fast burn pages, slow burn tickets, and two windows each stop the alert from lying.

Symptom · Either the alert fires constantly on brief error spikes that consume a negligible slice of budget, or it stays silent through a week of low-grade degradation that quietly consumes all of it. Both come from the same cause: a fixed threshold that knows nothing about the objective.
Dashboards Built Around Questions

A dashboard is not a place to put metrics — it is a tool for answering a specific question under time pressure. Two hundred charts is not thoroughness, it is an unindexed archive, and at 03:00 the difference between eight charts in the right order and two hundred in no order is the difference between five minutes and fifty.

Symptom · During an incident the responder scrolls through a wall of charts looking for the one that matters. Everyone has a personal favourite panel. The dashboard is comprehensive and nobody can use it, so debugging happens in ad-hoc queries typed from memory.