Pipeline Metrics
Rows processed, bytes processed, duration, failures, retries and lag — what each one detects, what moves it for boring reasons, and what none of them can see.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Which six numbers, recorded per run, would let me tell a broken pipeline from a busy one without opening a log?
The on-call engineer choosing where to look first, and the platform owner deciding which pipeline to invest in next quarter. Both need a small, stable set of numbers with history; neither can use a metric that changes meaning when the schedule changes (Data Platform Anti-Patterns).
One metric point per task attempt per logical period. Two dimensions, and confusing them is the classic error: a metric keyed only by wall-clock time cannot express a backfill, where twelve logical periods are processed in one afternoon and every duration looks anomalous.
Record duration and success. They come free with the orchestrator, they chart nicely, and duration genuinely does detect a whole class of problems — a job that suddenly takes twice as long is usually worth looking at.
Duration collapses when a run has nothing to do. The fastest run of the month is the one that read an empty partition, so the chart that should scream is at its calmest (Missing Rows).
- Duration collapses when a run has nothing to do. The fastest run of the month is the one that read an empty partition, so the chart that should scream is at its calmest (Missing Rows).
- Duration doubles because the cluster was busy, not because the data changed. Without a rows-processed number beside it, the two are indistinguishable (Saturation: The Reading Utilization Cannot Give You).
- Rows processed looks stable while bytes processed triples, because a free-text column started carrying a serialised payload. The cost moved and the row count could not see it.
- The failure count is zero and the retry count is eleven, so a persistent upstream fault has been absorbed silently for a week (Retries in Pipelines).
- A backfill runs and every metric for that day is meaningless, because twelve logical periods were processed in one wall-clock window and nothing recorded which was which (Backfills).
What is actually happening
- These six are chosen because each is a different *kind* of quantity. Rows and bytes are volume; duration is time; failures and retries are outcomes; lag is a distance between two clocks. A monitoring set built from one kind can only detect one kind of problem.
- Rows processed and bytes processed diverge whenever the average record size changes, and that divergence is itself the signal. Constant rows with growing bytes is a payload change; falling rows with constant bytes is usually deduplication or a filter that started matching.
- Duration is a composite of queue wait, input volume, engine behaviour and cluster contention, which is why it is a poor primary signal and a good secondary one. Normalising it — duration per million rows — removes the volume component and leaves the part that indicates a real change (Measure Before You Optimize).
- Retries and failures are not the same quantity. Failures are runs that ended badly; retries are attempts that ended badly and were hidden. A platform with zero failures and a rising retry rate is degrading, and only one of those numbers says so.
- Lag is the only one of the six that is a *data* metric wearing a pipeline metric's clothes. Consumer lag on a partitioned log, replication lag on a source, and the gap between the newest source record and the newest processed record are all distances between two positions, and each bounds the freshness of everything downstream (Offsets and Commits).
The six, and what moves each for boring reasons
Every metric has a set of causes that are genuinely interesting and a set that are not, and an alert that cannot separate them becomes noise. The fourth column below is the one to read first: it is why duration alerts get widened and why row-count alerts fire on public holidays.
The pairing matters as much as the individual numbers. Rows without bytes cannot see a payload change; failures without retries cannot see a degrading upstream; duration without volume cannot distinguish a busy cluster from a broken input. Each of the three pairs answers a question neither member can answer alone.
A note on lag, which behaves differently from the other five. It is the only one that can be sampled without the pipeline running, which makes it the only one that keeps reporting during an outage. That property makes it the best single number to put on a status board.
| Stage | One row is | Breaks if |
|---|---|---|
| `pipeline_run_facts` | One task attempt for one logical period. | You count rows and call it runs. A task that retried three times contributes four rows and one run. |
| Metric series point | One value of one metric for one label combination at one timestamp. | The label set changes between versions, so the same series name silently becomes a different population. |
| Lag sample | One observation of the distance between two positions at a sampling instant. | It is aggregated by averaging. The mean of a lag that spent an hour at zero and an hour at a day is not a number anyone can act on (The Average Was Fine and Users Were Not). |
| Check result | One check evaluated against one dataset for one period. | Results are stored keyed by evaluation time rather than by evaluated period, so a re-check after a backfill overwrites the record of the original failure. |
| Alert event | One notification delivered to one destination. | It is treated as one incident. A single stale root dataset can produce forty alerts through lineage and remains one incident (Impact Analysis). |
Four of these five are routinely aggregated across their grain by dashboards that were built quickly. Each aggregation is legitimate; each also destroys the distinction that made the metric useful.
| Metric | What it counts | What it detects | What moves it for a boring reason |
|---|---|---|---|
| Rows processed | Records read and records written, per stage, per logical period. | Partial loads, empty inputs, fan-out joins, filters that started matching, deduplication that stopped. | Weekends, holidays, marketing campaigns, and any genuine change in business volume (Volume Anomalies). |
| Bytes processed | Bytes read and written, and — on a distributed engine — bytes shuffled. | Payload growth, an encoding change, a schema addition, and skew when reported per task. | Compression settings, file compaction, and the arrival of a new column nobody mentioned (File Compaction). |
| Duration | Wall-clock time from task start to task end. | Hung runs, a query plan that changed, an engine that stopped pruning partitions. | Cluster contention, queue wait, a larger day, and a backfill processing twelve periods at once (Partition Pruning). |
| Failures | Runs that ended in a failed state. | The loud class: raised exceptions, timeouts, credentials expiring, an unreachable source. | A source's planned maintenance window, and any deliberate fail-on-bad-data check doing its job (Contract Enforcement). |
| Retries | Attempts that failed and were re-run automatically. | A degrading upstream, an intermittent network path, contention on a shared resource — all before they become failures. | An aggressive retry policy on a task that is not idempotent, where the retries are themselves the problem (Idempotent Data Pipelines). |
| Lag | The distance between the newest available source record and the newest processed one. | A stalled consumer, a stopped connector, a schedule that no longer keeps up with volume — and it keeps reporting while everything else is silent. | A quiet source, which makes lag look excellent, and a backfill, which makes it look terrible (CDC Failure Modes and the Retention Deadline). |
Recording them so that a backfill is legible
The design decision that separates a metric set you can debug with from one you cannot is keying by logical period as well as by wall-clock time. Every incremental pipeline processes a period that is not now, and every backfill processes many of them at once.
With both keys present, "how long does this pipeline take for one day of data" and "what was happening on the platform on Tuesday afternoon" are two different queries and both are answerable. With only wall-clock time, the first is unanswerable and the second is misleading whenever a backfill ran.
The second decision is to record ratios rather than only absolutes. Rows out over rows in is dimensionless, so it survives volume growth, schedule changes and seasonality — which makes it the one number whose alert threshold does not need to be re-tuned every quarter.
1-- Rows in / rows out per stage, per logical period.2with stage_ratio as (3 select4 task,5 logical_period,6 rows_read,7 rows_written,8 case when rows_read > 09 then rows_written::numeric / rows_read10 end as ratio11 from pipeline_run_facts12 where status = 'success'13),14baseline as (15 -- the same weekday, over the preceding four weeks16 select17 task,18 extract(dow from logical_period) as dow,19 percentile_cont(0.5) within group (order by ratio) as median_ratio20 from stage_ratio21 where logical_period between current_date - 35 and current_date - 122 group by 1, 223)24select25 s.task,26 s.logical_period,27 s.rows_read,28 s.rows_written,29 round(s.ratio, 4) as ratio_today,30 round(b.median_ratio, 4) as ratio_typical,31 round(s.ratio / b.median_ratio, 3) as relative32from stage_ratio s33join baseline b34 on b.task = s.task35 and b.dow = extract(dow from s.logical_period)36where s.logical_period = current_date - 137 and (s.ratio > b.median_ratio * 1.2 or s.ratio < b.median_ratio * 0.8);The bands are placeholders — they are chosen per pipeline from its own observed variance, not copied. What transfers is the shape: a dimensionless ratio compared against the same weekday, so growth and seasonality do not require re-tuning.
Observability has a cost shape of its own
It is easy to write about monitoring as though it were free. It is not: every check is a query that runs forever, every label multiplies a series count, and every retained sample is storage that nobody will ever delete. Platforms that instrument uniformly discover this around the time the metrics backend needs its own capacity plan.
The drivers below are ordered by how often they dominate, not by how often they are blamed. Cardinality is first because it is multiplicative rather than additive — one badly chosen label does not add a series, it multiplies the series count by the label's distinct values — and because the failure it produces looks like a monitoring outage rather than like a design mistake (Cardinality: The Label That Took Down Monitoring).
The counter-pressure is real and worth naming. Under-instrumenting is also expensive; it is just that the cost arrives as a longer incident rather than as a line on an invoice, and only one of those two gets discussed in a planning meeting.
Multiplicative: series count is the product of distinct label values. A partition-key label on a large table can outgrow every other driver combined, and it degrades the backend rather than showing up as a bill.
A distribution check over a wide table is a scan. Running it hourly on a large partitioned dataset can cost more than the pipeline that built it (Scan Cost).
Grows with tasks times periods times attempts and is never deleted by anyone. Daily aggregates keep the baselines and cost a fraction of the raw detail.
Driven by verbosity rather than by data size, so it grows when someone adds debug logging and never comes back down (The Log Bill and What It Is Buying).
Small in absolute terms, and it grows with the number of column-level edges rather than with data volume — so it is stable under data growth and jumps when a wide model is added (Column-Level Lineage).
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights for a warehouse-centric platform, shown to establish an ordering rather than a magnitude. The ordering is the teaching: the thing people optimise first is usually the last line, and the thing that actually breaks is usually the first.
How to build it
Most important first.
- Key every metric by logical period as well as by wall-clock time. This single decision is what makes backfills legible and makes "the same period last week" a computable comparison (Incremental Processing).
- Record rows *in* and rows *out* separately per stage. The ratio between them is the highest-value derived signal in a pipeline: a join that suddenly emits more rows than it read has fanned out, and no absolute count would have shown it (Grain: What Does One Row Represent?).
- Normalise duration by volume before alerting on it. Alert on duration per million rows, and keep raw duration for capacity planning where the absolute number is what matters (Capacity Planning: Traffic to Machines).
- Count retries as a first-class metric with the same prominence as failures, and alert on retry *rate* rather than on individual retries.
- Keep label cardinality deliberately small — dataset, task, status. Adding partition key or user id as a label is how a metrics backend becomes the most expensive component in the platform (Cardinality: The Label That Took Down Monitoring).
- Store the metric history as a table in the warehouse, not only in a time-series database. The interesting questions are joins against the catalog — "which tier-1 datasets have degrading row ratios" — and a metrics backend cannot do joins (Metadata: Technical, Operational and Business).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- These metrics guarantee an accurate description of what the pipeline did. They make no claim about what it should have done, and the gap between those two is where every incident lives.
- Rows written guarantees that many rows were written, not that they were the right rows or that they were written once. Uniqueness and completeness remain separate questions (Duplicate Rows).
- Lag guarantees a bound on freshness only if the lag measurement itself is fresh. A consumer that has stopped committing offsets reports a lag that stops moving, which reads as healthy on a chart that plots the difference rather than the age of the measurement.
- None of the six detects a value-level error. All of them are dimensionally incapable of it: they count and time, and correctness is neither a count nor a time (Two Dashboards, Two Numbers).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that earns its place: rows out divided by rows in, compared with the same ratio for the same weekday historically, per stage. It catches fan-out joins, filters that started matching, deduplication that stopped working and casts that dropped rows — a wide class from one cheap number.
- It misses any change where input and output move together, including the case where the input itself is half missing. The ratio is perfectly normal when the whole pipeline is fed half a day.
- It also misses value errors entirely, which is the point of pairing it with reconciliation against the source (Reconciliation).
- Metrics are emitted when a run finishes, so their freshness is bounded by the run interval. A daily pipeline's metrics are a daily signal, which is why lag — which can be sampled continuously against the source — is the one worth polling independently.
- Alerting on a metric that only appears at run completion means a hung run produces no data point at all. Absence has to be alerted on separately from a bad value, and absence is the harder alert to write (Gauges: Blind Between Scrapes).
- Aggregation windows blur short events. A five-minute rollup will not show a thirty-second spike, which is usually fine for data pipelines and is exactly the trap that Observability & Performance documents for request latency (Percentiles: Which One, and How Many Users Is That?).
- Metric definitions must be versioned like schemas. Changing what "rows processed" counts — say, from input records to output records after deduplication — silently breaks every historical comparison and every alert threshold derived from it.
- Adding a stage to a pipeline adds a row-count boundary, and old comparisons across the pipeline as a whole stop lining up. Record the stage set with the metric so that a change of shape is visible rather than confusing.
- When a schedule interval changes, per-run metrics change magnitude for a reason that has nothing to do with data. Store the interval alongside the metric so future you can normalise (Dataset Documentation).
- Metric history is itself a dataset and can be backfilled from run records if those were retained. Recovering it from logs is possible and unpleasant; recovering it from nothing is not possible at all, which is an argument for retaining run facts longer than feels necessary.
- When a backfill runs, mark the metric points it produced. Otherwise every baseline computed afterwards includes a day where twelve periods were processed at once, and the baseline is quietly poisoned (What Backfills Break).
- After a metric definition change, keep both definitions running for one full baseline window rather than rewriting history. Rewritten history is indistinguishable from history, which makes it worse than a discontinuity you can see.
What can go wrong
- A metric emitted only on success, so failures are invisible in the volume chart and the chart shows a smooth line through an outage.
- Label cardinality explosion — one series per partition per dataset — which degrades the metrics backend until the monitoring is the incident (Label Sets That Survive a Year).
- Duration alerting that has been widened so many times it can no longer fire.
- Lag measured but never aged, so a stopped consumer reports a frozen and healthy-looking number.
- Metrics keyed only by wall-clock time, which makes every backfill look like an anomaly and every real anomaly look like a possible backfill.
- "Duration is the health metric." A broken run is frequently the fastest run of the week. Duration without volume beside it is close to uninterpretable.
- "Zero failures means the pipeline is healthy." Zero failures and a rising retry rate means the pipeline is failing repeatedly and recovering, which is a different and more expensive kind of unhealthy.
- "Bytes processed is just rows times a constant." Average record size moves whenever a payload, an encoding or a compression setting changes, and the divergence between the two is the signal (Dictionary, Run-Length, Delta and Bit Packing).
- "Lag is a streaming concern." Every incremental batch pipeline has a lag — the gap between the newest source record and the newest processed one — and it is the most direct measure of freshness there is (The High-Water Mark).
Operating it
- Rows in and rows out per stage per logical period, and their ratio, on one chart. The single most diagnostic view in a data platform (Debugging a Data Incident).
- Duration per million rows, trended, with raw duration available separately.
- Failures and retries as two distinct series, never summed.
- Lag as an *age*: how old is the newest record this consumer has processed, sampled on a clock that is independent of the consumer (Replication Lag: Reads That Are Correct and Stale).
- At ten times the pipelines, the metric set must be generated by a shared wrapper rather than written per job, or coverage becomes a function of which engineer wrote which DAG.
- At a hundred times, dashboards stop working as a discovery mechanism — nobody scrolls four hundred charts. The interface becomes a query over the metric table plus alerts on derived ratios (Dashboards Built Around Questions).
- Metric volume grows with tasks times periods times labels, and periods grow when schedules get finer. Moving from daily to hourly multiplies every series by twenty-four before a single new pipeline is added.
- The metrics themselves are tiny; their cardinality is not. Cost scales with the number of distinct label combinations multiplied by retention, and a single well-intentioned label can multiply it by the number of partitions in your largest table (Cardinality: The Label That Took Down Monitoring).
- Computing rows and bytes is free when the engine already reports them and expensive when it requires a second pass over the output. Prefer engine-reported counters, and accept an approximate number over an exact one that costs a scan.
- Long retention of raw per-attempt points is rarely worth it. Keep daily aggregates for seasons and raw detail for weeks — the baselines need the former and the debugging needs the latter (Storage Lifecycle).
- A rich metric set is code in every pipeline and a second system to operate. The alternative — inferring health from the data alone — is cheaper to build and much slower to localise a fault, because it tells you a table is wrong without telling you which hop broke.
- Normalised metrics are better for alerting and worse for capacity planning; keeping both doubles the series count. That is usually the right call and it should be a deliberate one.
- Low cardinality keeps the metrics backend healthy and makes some questions unanswerable there. Answer those in the warehouse against the metric table instead, accepting minutes of latency for unlimited dimensionality.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe six quantities are properties of any pipeline that reads and writes records, batch or streaming. What differs is which the engine reports for free: some report input and output row counts per operator, others only wall-clock duration.
- ENGINE-SPECIFICDistributed engines report per-stage and per-task counters including shuffled bytes, while a single-node engine reports one number for the whole query — so the same "bytes processed" metric localises a skew problem in one and cannot see it in the other (Data Skew).
- SCALE-SPECIFICLabel cardinality is harmless below a few thousand series and is the dominant operational cost above a few million. The dataset-plus-task labelling advised here is chosen for the second regime and is unnecessarily austere in the first.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why a lag measurement taken on a consumer and a lag measurement taken on the broker can legitimately disagree, and what a consistent position across partitions would even mean.