Time-Series Anomaly Detection
Forecast the series, compare the actual to the forecast, and flag when the residual leaves a band. The band is a threshold with a false-alarm cost, the baseline must know about seasonality, labels are scarce, and an alert with no owner is noise.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
How do you flag that a series has done something it should not have, without paging someone every Monday morning — and what makes an alert worth sending?
A payments platform wants to know within minutes when transaction volume for a merchant drops unexpectedly — an integration break, a fraud block gone wrong. The last attempt fired forty alerts a day, mostly at midnight and on weekends, and the on-call team muted it within a fortnight.
Flag any window whose count is more than a fixed number of standard deviations below the merchant's mean. It is simple, it needs no labels, and a real outage is a big deviation.
A merchant whose mean is dominated by daytime volume looks anomalous every night, because the mean and standard deviation do not know it is night. Forty alerts a day, mostly at midnight, most of them "the shop is closed".
- A merchant whose mean is dominated by daytime volume looks anomalous every night, because the mean and standard deviation do not know it is night. Forty alerts a day, mostly at midnight, most of them "the shop is closed".
- A fixed threshold across twenty thousand merchants is wrong for all of them: too tight for the noisy small ones, too loose for the steady large ones, and it fires on a Sunday for every merchant that does not trade on Sundays.
- The threshold was set to catch every incident in the ticket history, with no cost assigned to the alerts it would also raise on the far larger set of normal windows. Recall was optimised, precision was never measured, and the page was muted (Threshold Selection).
- Each alert went to a shared channel with no owner. Nobody was responsible for acting on one, so nobody did, and the ones that were real were lost among the ones that were not (Model Monitoring).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Flag a (merchant, five-minute window) as anomalous when its transaction count is far below what the merchant's history predicts for that time. There is no ground-truth label for most windows; a few hundred confirmed incidents exist from past tickets.
- The decision is a page to the on-call integrations engineer, so the useful output is a ranked, deduplicated list of merchants whose drop is both large and unexplained.
- Five-minute transaction counts per merchant for eighteen months, for around twenty thousand merchants. Strong daily and weekly cycles that differ by merchant, timezone and sector; many merchants are quiet at night and on Sundays by design.
- A ticket history of a few hundred confirmed outages with start times, and no record of the far larger number of windows that were fine (Anomaly Detection).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The forecast-residual approach: forecast the count for the window from the merchant's own history — a seasonal model that knows this merchant at this hour on this weekday — and compute the residual, actual minus forecast. An anomaly is a residual outside a band. The forecast carries the seasonality, so "the shop is closed" is the forecast, not an anomaly.
- The band is a threshold, and it has the same structure as any threshold on a score: wider means fewer alerts and more missed incidents, tighter means the reverse. It should scale with the merchant's own residual variability — a band in units of that merchant's typical forecast error, not a global constant — and it can be asymmetric, since a drop is the incident and a spike usually is not.
- Labels are scarce and one-sided: confirmed incidents exist, confirmed non-incidents do not, and the incidents are the ones someone noticed. So the detector cannot be trained as a classifier in the ordinary way; it is a forecast plus a rule, tuned against the incidents for recall and against alert volume for precision, and the alert volume the on-call team can absorb is a hard constraint (Alert Fatigue: The Page Nobody Reads).
The forecast is the expectation; the residual is the signal
A merchant at 3 a.m. on a Sunday does not trade. A mean-and-standard-deviation detector does not know that and flags it. A forecast that reads this merchant's last four Sundays at 3 a.m. expects nothing, sees nothing, and stays quiet. The seasonality lives in the forecast so that it is not in the residual.
So the detector is two parts: an expectation per window per merchant from the merchant's own recent history, and a rule on the difference. The expectation is a small forecasting problem with a one-step horizon and a seasonal naive baseline that is usually good enough. The rule is a threshold, and a threshold is a cost decision.
- 1Expect
Forecast this merchant's count for this five-minute window from the same window in recent weeks — a seasonal naive or a small seasonal model.
fails by The merchant changed hours last week; the expectation is stale until the baseline window rolls past the change.
- 2Compare
Residual = actual − expected, scaled by the merchant's typical residual spread so that a noisy small merchant and a steady large one are comparable.
fails by The count itself is late because the ingestion pipeline is behind; the residual measures the pipeline, not the merchant.
- 3Persist
Flag only when the scaled residual is below the band for k consecutive windows.
fails by k is set too high and a short outage is missed; too low and single-window noise pages.
- 4Route
Deduplicate across windows, rank by size and merchant importance, send to a named owner with the runbook, log the outcome.
fails by No owner is named; the alert lands in a channel and the outcome is never logged.
Every step has a stale-baseline or a pipeline-delay failure. The detector is downstream of ingestion, and the first question on any alert is whether the data arrived.
The band is a threshold with a budget
The on-call team can triage perhaps a dozen alerts a day. That number is the constraint, and the band is set so the detector emits about that many on a normal day. Then recall on the ticketed incidents is measured at that band, and the result is what the detector can promise. Tightening the band to raise recall past the budget does not raise recall; it raises the alert count until the pager is muted, at which point recall is zero.
A false positive costs an engineer's attention and, cumulatively, the pager's credibility. A false negative costs the outage's duration until a customer reports it. Both are real; they are not equal; and the band should be set from their ratio and the budget, not from the wish to catch everything.
The counts are illustrative. The point is the ratio: three hundred false alarms against forty real drops in a month is ten a day, which a team can carry; at a tighter band it is forty a day, which it cannot. The true negative count is what makes a tiny false-positive rate into a real pager load.
An alert with no owner is noise
The last attempt fired forty a day into a shared channel. Nobody was on the hook for any single one, so nobody triaged, so no outcome was logged, so the detector never learned which of its alerts were real, so it could not be tuned, so it was muted. The chain is causal and it starts at ownership, not at the model.
The operational rule is that an alert exists only if a named person is expected to act on it within a stated time, with a runbook that says what to check first (did the data arrive?), and with an outcome recorded. That record is also the only supply of labels the detector will ever get, and it is what turns a scarce-label problem into a slowly growing one.
Every alert the detector emits reaches a named owner who is expected to triage it within the runbook's time and record whether it was real, so that the alert has an effect and the detector accumulates labels.
holds when The alert volume is within the team's stated capacity; the rota names an owner for the alert at all hours; the runbook's first step is a data-arrival check; the outcome field is required to close the alert.
breaks when Volume exceeds capacity after a baseline goes stale; the owner rotates off and the rota is not updated; the outcome field is optional and fills with "unknown"; a platform-wide event floods the channel.
respond Widen the band or add persistence to bring volume under capacity before touching anything else; a detector that is being ignored has no recall to protect. Then fix the rota and the outcome field, and re-derive the band from the outcomes.
How to build it
Most important first.
- Use a seasonality-aware baseline per merchant: the same window last week and the week before, or a simple seasonal model, as the forecast. This alone removes most of the midnight and Sunday alerts, because the forecast for those windows is low (Trend and Seasonality).
- Scale the band by the merchant's own historical residual spread, and set it from the cost of an alert — the on-call team's capacity per day — rather than from the desire to catch everything. Then measure what recall that buys on the ticketed incidents.
- Require persistence: an anomaly is a residual outside the band for two or three consecutive windows, not one. A single five-minute dip is noise; three in a row is a pattern. This trades a few minutes of detection delay for most of the false alarms.
- Route every alert to a named owner with a runbook, and log the outcome — real, benign, unknown — so the detector accumulates the labels it did not have (Human Oversight).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Alerts per day against the on-call team's stated capacity, and the fraction of alerts marked real when triaged. This is the number that decides whether the detector survives its second week.
- Recall on the ticketed incidents at the chosen band, and time-to-detect from incident start; a detector with high recall and a forty-minute delay is a different product from one that fires in ten.
- Do not measure "anomalies detected". Without a denominator of windows that were normal and a cost per false alarm, the count of flags is a count of pages, not of value.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Each merchant's seasonal pattern is stable over the baseline window, so the forecast for a given hour and weekday is a fair expectation; a merchant that changes hours, timezone or sector breaks it.
- The on-call capacity the band was set against is real and stays roughly constant; a team half the size at the same band is a muted alert in a fortnight.
- The ingestion of transaction counts is itself timely, so a drop in the count is a drop in transactions and not a delay in the pipeline that reports them (The Freshness SLO).
- Offline: replay the last six months through the detector and count alerts per day and recall on ticketed incidents, per band width; choose the band from the curve and the capacity, and report both numbers.
- Online: for the first month, run the detector in shadow and have the on-call engineer triage a daily sample of what it would have paged, recording real / benign / unknown.
- Over time: track the alerts-per-day and the real fraction weekly; a rising alert count with a falling real fraction means a baseline going stale or a merchant population changing, and it is the signal to re-derive the bands before the pager is muted.
What can go wrong
- A merchant changes its opening hours; the seasonal baseline still expects last month's pattern and fires every morning for a week until the history rolls over (Concept Drift).
- A platform-wide outage drops every merchant at once; twenty thousand alerts fire, the pager melts, and the one alert that mattered — the platform, not any merchant — was never modelled (From Symptom to Root Cause).
- The outcome log fills with "unknown" because triage takes longer than the on-call shift; the labels the detector needed never arrive, and the band is re-tuned against the same few hundred incidents (Ground-Truth Delay).
- A per-merchant seasonal forecast is twenty thousand small models to maintain, or one global model with merchant features; either is more than a mean and a standard deviation, and either is the difference between a detector and a noise generator.
- A persistence requirement adds minutes to detection; an integration break that lasts three minutes will not be caught, and that is a chosen cost.
- Requiring a named owner per alert means some alerts have no owner and are not sent, which is the correct outcome and feels like a gap.
- "We should lower the threshold so we never miss an outage." The pager has a budget. Below it, the detector catches more and is muted; the marginal alert is not free, and a muted detector has zero recall.
- "The detector needs labels; we should label a big dataset." The labels arrive from triage, one at a time, if triage is designed to produce them. A labelling project for windows that were fine is expensive and mostly confirms that most windows are fine.
- "Anomaly detection is unsupervised, so we can't evaluate it." It has a few hundred incidents and a daily alert budget. Recall on the incidents at the budget is an evaluation, and it is the one the on-call team lives with.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe forecast-residual structure — expectation, residual, band, owner — applies to any monitored series, from merchant volume to CPU load; the choice of baseline and band is where it becomes specific.
- DOMAIN-SPECIFICMerchant volume has strong per-entity seasonality and a one-sided incident (drops), so the band is asymmetric and per merchant; infrastructure metrics often have symmetric incidents (spikes and drops), shared seasonality and a much smaller entity count, which changes the baseline strategy.
- CONTESTEDA strong position holds that forecast-residual detectors are a hand-tuned relic and that learned detectors — autoencoders, isolation forests over windows, density models — find anomalies no seasonal baseline would. That is true for anomalies of shape rather than level; the reply is that those models are harder to explain to the engineer being paged, need the same alert budget, and on a merchant-volume drop the seasonal residual is both sufficient and legible.
Where the depth lives
This domain teaches the model and hands the rest off by name.