Burn-Rate Alerts: How Fast Is the Budget Going?
Alerting on "error rate above 1%" picks a threshold with no relationship to what you promised. Alerting on burn rate asks a better question: at the current failure rate, how long until the error budget is gone? Fast burn pages, slow burn tickets, and two windows each stop the alert from lying.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Burn rate defined
Burn rate is a ratio with a clean interpretation: it is the observed failure rate divided by the failure rate your objective permits. A burn rate of 1 means you are consuming budget at exactly the pace that would exhaust it precisely at the end of the window — sustainable, by definition. A burn rate of 10 means you will exhaust a 28-day budget in 2.8 days. A burn rate of 0.5 means you are running comfortably ahead and could afford more risk.
The advantage over a fixed threshold is that burn rate is *derived* from the objective rather than guessed. error_rate > 1% means something completely different for a 99% SLO (burn rate 1, entirely sustainable) than for a 99.99% SLO (burn rate 100, budget gone in hours). One expression works for every service in the organisation, and it stays correct when the objective changes — a fixed threshold has to be re-derived by hand and, in practice, is not.
It also puts urgency and severity on the same scale. Burn rate answers "how long do we have", which is exactly the question that decides whether to wake someone. A 2x burn will exhaust a month's budget in two weeks: worth a ticket. A 100x burn exhausts it in seven hours: worth a page, now.
SLO = 99.9% over 28 days budget = 0.1% of events may fail burn_rate = observed_error_rate / 0.001 observed burn budget exhausted in response --------- ----- ------------------------- ----------------- 0.05% 0.5x never (recovers) nothing 0.10% 1.0x exactly 28 days nothing — sustainable 0.60% 6.0x 4.7 days ticket 1.44% 14.4x ~1.9 days page 10.00% 100.0x ~6.7 hours page, escalate The canonical thresholds are derived against a 30-day window, which is where the round budget fractions come from: burn 14.4x for 1 hour consumes 2% of a 30d budget burn 6x for 6 hours consumes 5% of a 30d budget burn 3x for 1 day consumes 10% of a 30d budget burn 1x for 3 days consumes 10% of a 30d budget On the 28d window above, each is ~7% larger (14.4x/1h = ~2.1%).
Two windows, because one always lies
A single-window burn-rate alert fails in one direction or the other, and which direction depends entirely on the window length. A short window (5 minutes) reacts fast but fires on every transient blip — a 30-second dependency hiccup produces a huge instantaneous burn rate that consumes a trivial fraction of the budget. A long window (6 hours) is stable but slow: it takes hours to notice a total outage, and then hours more to reset after recovery, during which the alert keeps firing at a system that is already fixed.
Multiwindow alerting resolves this by requiring both a long and a short window to exceed the threshold at once. The long window establishes that the burn is sustained and material; the short window confirms it is *still happening right now*. A transient spike fails the long window. A recovered outage fails the short window, so the alert resets promptly instead of hanging around. The convention is a short window roughly one twelfth of the long one.
Layering two such pairs gives good coverage of both failure shapes. A fast-burn pair (1h and 5m, threshold 14.4x) catches acute outages within minutes — at that rate 2% of the budget is gone in an hour. A slow-burn pair (6h and 30m, threshold 6x) catches the grinding low-grade degradation that never trips an acute threshold but consumes the budget over days, and it routes to a ticket rather than a page because there is time to respond in working hours.
These specific numbers — 14.4x/1h and 6x/6h — are conventions, popularised via the Google SRE workbook, chosen so each pair corresponds to a round fraction of budget (2% and 5%) over a convenient window. They are a good default and not a law. A service with different traffic volume or a different tolerance for detection lag should recompute them from the budget fraction it actually wants to react to.
| Tier | Long window | Short window | Threshold | Budget consumed when it fires | Route |
|---|---|---|---|---|---|
| Fast burn | 1 hour | 5 minutes | 14.4x | ~2% of a 30d budget (~2.1% on 28d) | Page |
| Slow burn | 6 hours | 30 minutes | 6x | ~5% of a 30d budget (~5.4% on 28d) | Ticket |
| (optional) Very slow | 3 days | 6 hours | 1x | ~10% of a 30d budget (~10.7% on 28d) | Ticket / review |
Where burn-rate alerting struggles
Low traffic is the honest limitation. Burn rate is a ratio of rates, and at low event counts the denominator is tiny and the ratio is wild. A service handling 10 requests a minute against a 99.9% objective permits 0.01 failures per minute — so a single failed request in a 5-minute window produces a burn rate around 20x and pages you for one error. The mitigations are all imperfect: lengthen the short window, require a minimum absolute event count before the rule can fire, or accept that below some traffic volume SLO-based alerting is not the right instrument and use absolute error counts instead.
The second gap is that burn-rate alerts are deliberately symptom-based and therefore tell you nothing about cause. That is the correct trade — it is why they catch failures nobody anticipated — but it means the page must be paired with a dashboard and a runbook that make the next step obvious (Dashboards Built Around Questions). An alert that says "budget is burning at 30x" and links nowhere leaves the responder to start from zero at 03:00.
Third, they only cover what the SLI measures. Anything outside the SLI is invisible: asynchronous work, batch pipelines, background jobs, and the queue whose oldest message is quietly ageing (Depth Is Not an Emergency; Age Is). A request-availability SLI can be perfect while a pipeline has processed nothing for six hours. Burn-rate alerting on well-chosen SLIs plus explicit freshness alerts covers both; either alone leaves a hole.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Burn rate, 5m window | 48x | Something is badly wrong right now | smoking gun |
| Burn rate, 1h window | 19x | Sustained, not a blip — fast-burn pair fires, page | smoking gun |
| Budget remaining | 54% | Still recoverable if this stops soon | suspect |
| Burn rate, 5m (30 min later, post-fix) | 0.2x | Recovered; short window clears so the alert resets | normal |
| Burn rate, 1h (30 min later) | 11x | Still elevated by history — alone it would keep paging a healthy system | suspect |
Key points
- Burn rate = observed failure rate / permitted failure rate; 1x exhausts the budget exactly at the window's end, so it is sustainable by definition.
- It is derived from the objective rather than guessed, so one expression works across services and stays correct when the SLO changes.
- Require a long and a short window to exceed the threshold together: the long one proves it is material, the short one proves it is current and lets the alert reset.
- Two tiers cover both shapes: fast burn (1h/5m at 14.4x) pages for acute outages, slow burn (6h/30m at 6x) tickets grinding degradation.
- At low traffic the ratio is noise — one failed request can read as 20x. Use minimum event counts, longer windows, or absolute thresholds instead.
Progressive depth
Overview
Burn rate answers "how fast is the error budget draining?". 1x means it lasts exactly to the end of the window. 10x means it is gone in a tenth of the time.
Practical
Two tiers, each requiring a long and a short window simultaneously: fast burn (1h + 5m at 14.4x) pages, slow burn (6h + 30m at 6x) tickets. The short window is roughly one twelfth of the long one.
Advanced
The thresholds are derived, not given: pick the budget fraction you want to react to, divide by the window fraction the alert covers. Add a minimum-event guard for low traffic, and pair every rule with a runbook since burn rate is deliberately cause-agnostic.
Internals
Burn rate is a ratio of a measured ratio to a constant, so its variance is dominated by the numerator's event count. At n events per window with true failure probability p, the observed count is roughly binomial: the standard deviation of the observed rate is sqrt(p(1−p)/n), which the division by (1 − SLO) amplifies directly. For a 99.9% objective and a 5-minute window at 10 req/s (n = 3,000), a single spurious failure moves the observed rate by 0.033% — a third of the entire permitted rate, i.e. 0.33x of burn. That is why short windows on low-traffic services are unusable, and why the minimum-event guard is not optional there.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Deploy → SLI: error ratio rises from 0.02% to 1.44% for the checkout journey.
- 2SLI → burn rate: against a 99.9% objective that is 14.4x, so roughly 2.1% of the 28-day budget is consumed per hour.
- 3Burn rate → windows: the 1h window crosses 14.4x and the 5m window confirms it is still happening.
- 4Alert → on-call: fast-burn pair pages with a runbook link; budget still reads 54% remaining, so there is room to respond.
- 5Rollback → recovery: the 5m window falls to 0.2x within minutes and the alert clears, even though the 1h window stays elevated by history.
- • Reading a high short-window burn rate alone as an emergency; a 30-second blip produces a huge instantaneous rate and negligible budget spend.
- • Reading a long-window rate alone as current state — it stays elevated for the length of the window after recovery.
- • Treating 14.4x and 6x as standards rather than as conventions chosen to correspond to 2% and 5% of a 28-day budget.
- • Applying burn-rate alerting to a low-traffic service and interpreting single-request noise as a real signal.
- • Assuming budget-burn coverage implies coverage of asynchronous work, which the request SLI never sees.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Compute burn rate as `error_ratio / (1 − SLO)` over each configured window, from the same SLI the objective is defined on.
- • Evaluate long and short windows together in one expression so the alert cannot fire on either alone.
- • Track how much budget each firing corresponded to, so thresholds can be tuned against actual consumption rather than intuition.
- • Require a minimum event count in the short window before the rule may fire, and record how often that guard suppresses a firing.
- • Replace fixed error-rate thresholds with multiwindow burn-rate rules derived from each service's objective.
- • Configure two tiers — fast burn to page, slow burn to ticket — and route them differently on purpose.
- • Add a minimum-event-count guard so low-traffic windows cannot fire on a single failure.
- • Attach a runbook and a dashboard link to every burn-rate rule, since the alert is deliberately cause-agnostic ([[alerting]]).
- • Add explicit freshness and age alerts for asynchronous work that the SLI cannot observe ([[queue-age]]).
- • Back-test the rules against the last quarter of SLI data: confirm they fire for real incidents and not for transient blips.
- • Inject a controlled elevated error rate and confirm the fast-burn pair fires within the expected window and clears promptly after recovery.
- • Check that each firing corresponded to a meaningful budget slice by comparing recorded consumption against the tier's design target.
- • Multiwindow rules are harder to read and debug than a single threshold, and a misconfigured window pair fails in ways that are not obvious from the expression.
- • Fast-burn detection is bounded by the short window — you will not detect a total outage faster than a few minutes, by design.
- • Burn-rate alerts say nothing about cause, so they raise the dependence on good dashboards and runbooks during response.
- • Re-derive thresholds whenever the SLO target or window changes — burn-rate rules encode the objective and silently go stale otherwise.
- • Review firings quarterly against budget consumed, and retune tiers that consistently fire above or below their design fraction.
- • Monitor traffic volume against the minimum-count guard: a service that shrinks can drift into the noise regime without anyone noticing.
Accuracy
Performance numbers are conditional. These are the conditions.
- ESTIMATEDBurn-rate arithmetic is exact given the stated objective and window: 14.4x for 1 hour consumes 1/720 of a 30-day window × 14.4 ≈ 2%; 6x for 6 hours ≈ 5%; 3x for 1 day ≈ 10%. Substituting a 28-day window changes these by roughly 7%.
- ILLUSTRATIVEThe 14.4x/6x threshold pairs are a widely used convention popularised by the Google SRE workbook, chosen for round budget fractions. They are a starting point to recompute per service, not a standard.
- WORKLOAD-SPECIFICEverything here assumes enough events per short window for the ratio to be meaningful. Below roughly a few hundred events per short window, burn rate is dominated by noise and absolute counts are the better instrument.