Data Observability
Pipeline health and data health are two different systems. A platform that watches only the first finds almost none of the incidents anyone cares about.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Every task has been green for a month and finance says the quarter is wrong. What should have been watching, and what was it watching instead?
Everyone who reads a number without being able to see how it was made — an analyst, a finance close, an experiment readout, a training set, a retrieval index. None of them will open your DAG. Their only question is whether today's number can be believed, and the observability layer exists so that a machine answers it before a human has to (Trusting Data).
The unit is the dataset-period: one table, one partition, one refresh. Every signal in this module is measured per dataset and per period, because a platform-wide roll-up is an average, and an average over four hundred tables cannot show you the one that stopped updating on Friday.
Alert on orchestrator task failure and on run duration. It is what the tool ships with, it costs nothing to enable, it catches genuine outages, and for the first year of a small platform it is a defensible amount of monitoring.
An upstream API changed its pagination and the extract returned zero rows. The load succeeded, the transform succeeded, the publish succeeded, and yesterday is now an unusually quiet day in every dashboard (The Pipeline Succeeded. The Data Is Wrong.).
- An upstream API changed its pagination and the extract returned zero rows. The load succeeded, the transform succeeded, the publish succeeded, and yesterday is now an unusually quiet day in every dashboard (The Pipeline Succeeded. The Data Is Wrong.).
- A dimension gained duplicate keys, so the fact join fans out and revenue is reported several times over. The run is *faster* than usual because nothing raised, and a number that is too high is questioned far less often than one that is too low (Duplicate Rows).
- The producer started sending
amountas a string. The cast produced null rather than an error, so the row count reconciles perfectly and every measure is zero (Breaking Schema Changes). - A task was removed from the DAG during a refactor. There is now no task to fail, no alert to fire, and a table that quietly stopped being built (When a Task Fails Mid-DAG).
- The one alert that does fire — a five-minute duration threshold on a job that occasionally takes six — fires twice a week for a year, so when a real one arrives nobody looks (Alert Fatigue: The Page Nobody Reads).
What is actually happening
- An orchestrator observes processes: exit codes, durations, retries, dependency state. That is a complete description of whether the code ran and an empty one about what the code produced. The two are only correlated when failures raise, and this domain's characteristic failures do not raise (Scheduler vs Orchestrator).
- Data observability instruments the dataset instead: how recent is it, how many rows arrived, what shape is it, does it still match its declared schema, and where did it come from. Those five questions are answerable by querying the data itself, which is why they work even for datasets no pipeline of yours produces.
- Each signal is a *derivative* of the table rather than of the job.
max(event_time)does not care which tool wrote the rows; a row count against the same weekday last month does not care whether the DAG was refactored. That independence is the point — the monitor must not share a failure mode with the thing it monitors. - The signals are cheap because they are aggregates, not scans of values. Counting rows, taking a max timestamp and computing a null rate touch metadata or a single column. Verifying that every value is *correct* is a different and much more expensive class of work, which is why it lives in Data Quality and is applied selectively.
- What ties them together is Data Lineage. A freshness alert on one table is a fact; a freshness alert plus the graph of what that table feeds is an incident with a blast radius, which is the difference between a signal and a response.
Two planes, and only one of them is usually watched
Draw your platform twice. The first drawing is the control plane: a scheduler, its tasks, their dependencies, their exit codes. The second is the data plane: the datasets those tasks produce, their partitions, their rows, their columns. Almost every platform monitors the first drawing and calls it monitoring.
The two planes fail independently in both directions. A task can fail while the data is perfectly fine — a flaky network call on a re-runnable job. Much more importantly, the data can be badly wrong while every task succeeds, because the failures that matter here (missing rows, duplicates, late arrivals, nulled casts, wrong logic) are all *successful* executions of code.
That independence is why the answer is not "better alerts on the DAG". It is a second set of signals, computed from the tables rather than from the jobs, on a schedule of their own. The strongest version of this rule: the monitor must not share a failure mode with the thing it monitors — which is exactly why a freshness check that reads the pipeline's own status table is worth very little.
- Control-plane signals — task success, duration, retry count, queue wait, dependency state. Owned by the orchestrator, useful, and blind by construction to everything below (Pipeline Observability).
- Data-plane signals — freshness, volume, schema conformance, distribution, null and uniqueness rates. Computed against the dataset, independent of who wrote it (The Dimensions of Data Quality).
- The connecting signal — lineage. It turns "this table is stale" into "these nine dashboards and one training set are stale", which is the difference between a fact and an incident (Lineage Debugging).
- The meta-signal — coverage. Which datasets have no signals at all. It is the only one that detects the platform growing past its instrumentation.
The five signals, and what each one still cannot see
A useful data observability layer is small. Five signals cover the overwhelming majority of detectable incidents, and the reason to keep the list short is that every additional signal costs attention rather than compute.
Read the misses column as the design brief. Each signal is chosen because it is cheap and broad; none of them is chosen because it is thorough. Their blind spots overlap deliberately — volume catches what freshness cannot, distribution catches part of what volume cannot, and reconciliation against the source is the only one that sees the whole journey at once (Reconciliation).
Notice what is absent from the list: correctness. No signal here asserts that a value is right. That is a much more expensive question, answered selectively with tests written by someone who knows the business meaning of the column (Data Tests).
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Freshness — age of the newest complete record | The dataset is recent enough for the decisions it drives. | A stopped pipeline, a stopped source, a removed task, a silently empty upstream, a publish that never happened. | Data that is perfectly fresh and completely wrong. It also fires falsely on any period where the source genuinely produced nothing, which is what eventually gets it muted. |
| Volume — rows written per period against that period's own history | Today looks like a normal day for this dataset. | Partial loads, an extract window that closed early, a broken filter, a source outage, a schema change that nulled a column used in a WHERE clause. | Every error that preserves row count — which is most value-level bugs — and slow drift, because each day resembles the one before it. |
| Schema — column set and types against the declared contract | The producer is still sending what it agreed to send. | Added, removed, renamed and retyped fields before they reach a cast that silently nulls them. | Semantic change with an unchanged schema: the same column, the same type, a different meaning. No type system will ever see it (Semantic Changes). |
| Distribution — null rate, cardinality and category shares against history | The contents look like the contents usually look. | A nulled column, a category that disappeared, a join that lost a segment, a unit change large enough to move the mean. | Any error that preserves the shape while changing every value inside it, and genuine business change, which it reports as an anomaly with total confidence. |
| Lineage — the edges between datasets, emitted as they are built | We can say what feeds this and what this feeds. | Nothing on its own. It converts every other signal into a blast radius and an upstream walk. | Anything that happens outside the tools that emit it — a hand-written export, a notebook, a BI-layer join, a spreadsheet someone maintains (Data Lineage). |
Five signals, five different blind spots, and one of them (lineage) detects nothing by itself. That combination is the design: broad and cheap, then deliberately supplemented with narrow and expensive tests where a consumer would actually notice.
Where this ends and Observability & Performance begins
Both disciplines use the word observability and share almost none of its content. Observability & Performance asks *why is this slow, where is the bottleneck, what is saturated* — a question about the behaviour of a running system, answered with latency distributions, traces and queueing theory (Latency Is a Distribution, Not a Number, Queueing: Why Systems Get Slow Before They Get Broken).
This module asks *is the data correct, complete and fresh* — a question about the contents of a table, answered by querying it. A perfectly healthy system, at every percentile, with no saturation anywhere, can be publishing a revenue number that is double the truth. Conversely a badly saturated warehouse with a slow query is a performance incident even though every row in it is correct.
The practical rule for authoring and for on-call: if the symptom is a duration, a rate or a saturation level, it belongs to that domain and its playbooks apply unchanged. If the symptom is a *value* — missing, duplicated, stale, misshapen — it belongs here. The one genuine overlap is lag on a stream, which is a queue depth to one domain and a freshness bound to the other, and both readings are correct (The Backlog Arithmetic: Four Levers and a Drain Time).
| The question | Observability & Performance owns | Data Observability owns |
|---|---|---|
| What is the unit of concern? | A request, a span, a worker, a queue. | A dataset-period: one table, one partition, one refresh. |
| What does the primary signal measure? | Duration, rate, error ratio, saturation — see The Four Golden Signals and RED: Rate, Errors, Duration. | Age, row count, schema conformance, distribution, lineage edges. |
| What does a healthy reading mean? | The system is serving within its budget (SLOs: A Target, a Window, and a Reason, Error Budgets: Unreliability You Are Allowed to Spend). | The table is recent, normally sized and normally shaped. It says nothing about whether the values are right. |
| How is an incident diagnosed? | Narrow from symptom to signal, then to a trace and a span (From Symptom to Root Cause, Distributed Tracing). | Walk upstream dataset by dataset until a period stops being complete (Debugging a Data Incident). |
| What is the characteristic failure? | Tail latency that averages away (Tail Latency: Why p50 Being Fine Does Not Help, The Average Was Fine and Users Were Not). | A green pipeline that published wrong rows (The Pipeline Succeeded. The Data Is Wrong.). |
| Where do they genuinely meet? | Consumer lag on a partitioned log is queue depth and a saturation signal. | The same lag is an upper bound on end-to-end freshness for every downstream table (The Freshness SLO). |
How to build it
Most important first.
- Instrument the dataset, not only the job. Every serving table gets freshness, volume, schema conformance and at least one distribution signal, emitted on a schedule that does not depend on the pipeline that builds it.
- Declare a tier per dataset and let the tier decide the response. A tier-1 table that finance closes on earns a page; a tier-3 exploratory table earns a row on a dashboard and nothing else. Uniform alerting across four hundred tables produces four hundred ignored alerts (Quality Alerting).
- Make freshness a stated commitment rather than an observation — a per-dataset target that a consumer can read, with a breach that means something (The Freshness SLO).
- Emit lineage from the transformation tool as it runs, so the graph is generated rather than maintained. A lineage graph written by hand is out of date the first week and dangerous by the second (Data Lineage).
- Give every alert an owner and a runbook entry before it is enabled. An alert with no named owner is a notification, and notifications are the raw material of Alert Fatigue: The Page Nobody Reads rather than of incident response (Data Ownership).
- Record the checks' own results as a dataset. "Which checks failed on which datasets over the last quarter" is the question that tells you where to invest, and it is unanswerable if results only ever went to a chat channel (The Data Quality Dashboard).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A green orchestrator guarantees that a process exited zero within its timeout. It guarantees nothing about row counts, values, completeness or meaning, and treating it as evidence of correctness is the single most expensive mistake in this domain.
- A passing freshness check guarantees the table contains a recent record. It does not guarantee that the record is right, that the period is complete, or that nothing older was silently dropped.
- A passing volume check guarantees today looks like a normal day in count. It guarantees nothing at all about values, which is where most real bugs live (Volume Anomalies).
- No combination of these signals guarantees correctness. They bound how *wrong* a dataset can be while looking normal, and that bound is what you are buying.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check on the checks: assert that every tier-1 dataset has a freshness signal, a volume signal and an owner, and alert on datasets that have none. Coverage gaps are invisible by construction — an unmonitored table produces no alerts, which looks exactly like a healthy one.
- It misses whether the thresholds are meaningful. A dataset with a freshness threshold of thirty days passes coverage and detects nothing.
- It also misses the datasets nobody registered. A table created by hand during an incident and never removed is monitored by nothing and consumed by someone (Data Discovery).
- Observability adds latency of its own: a check that runs after the publish detects a problem after consumers can already read it. Running the same checks *before* the publish trades freshness for safety and is usually the right trade for tier-1 datasets (Atomic Publish).
- Detection latency is its own number and worth stating separately from data latency. A dataset that is one hour fresh but whose failures are noticed the next morning has a twelve-hour incident, whatever the pipeline's schedule says.
- Checks that run on a fixed clock detect a stalled hourly table quickly and a stalled monthly one absurdly late. Match the check interval to the dataset's own cadence rather than to a platform default.
- Signals must survive schema change. A volume check written as
count(*)keeps working when a column is added; a distribution check pinned to a column that was renamed starts failing for a reason that has nothing to do with data quality (Schema Evolution). - The most useful schema signal is not "the schema changed" but "the schema changed and here is who reads the affected column" — which requires column-level lineage to be anything other than noise (Column-Level Lineage).
- Semantics change without the schema changing, and no signal in this module detects that. A
revenuecolumn that moved from gross to net is fresh, complete, well-typed, normally distributed and wrong (Semantic Changes).
- Observability itself must be re-runnable. Store check results with the period they evaluated, not only the time they ran, so a backfill can re-evaluate a historical period and produce a comparable answer (Validating a Backfill Before You Publish).
- Keep the raw signal history. Deciding whether today's volume is anomalous requires a baseline, and a monitoring system that retains only a week cannot detect anything seasonal.
- After an incident, the recovery step people skip is adding the signal that would have caught it. An incident review whose only output is a fixed pipeline has thrown away most of its value (Data Incidents).
What can go wrong
- The monitor shares a failure mode with the monitored. A freshness check that reads the same metadata table the pipeline writes goes quiet exactly when the pipeline does.
- Coverage that decays. New datasets arrive faster than checks are written, so the fraction of the platform actually monitored falls every quarter while the dashboard of green checks looks unchanged.
- Thresholds tuned to stop paging rather than to detect. Every widening is locally reasonable and the sequence ends with a check that cannot fire.
- Alerting on every signal for every dataset, which trains everyone to filter the channel and is functionally identical to having no alerts (Alert Fatigue: The Page Nobody Reads).
- Observability metadata that itself leaks — a null-rate alert quoting example values from a column of email addresses (PII in Pipelines).
- "We have observability — we use Grafana." That is a metrics backend. Observability is a question about whether you can answer *why this dataset looks like this* without adding new instrumentation, and a dashboard of task durations cannot (Observability Is Not a Dashboard).
- "All the checks are green, so the data is correct." The checks bound the failures you thought of. Every incident in this module's later lessons passed several checks on its way to the dashboard.
- "Monitoring is the data team's job." Detection can be centralised; the obligation to produce correct data cannot. A team that owns a dataset and not its signals will always learn about its own incidents last (Who Owns Data Quality).
- "Data observability is observability with a different noun." The signals share vocabulary and almost nothing else: Observability & Performance asks why a request is slow, and this asks whether a table is complete. A latency histogram cannot answer the second and a null-rate check cannot answer the first (Data Engineering and Observability).
- Alerts, samples and profiles are copies of the data. A check that includes offending rows in its message has just written production values into a chat log with a completely different access model (Data Masking, Tokenisation & Encryption).
- Lineage and catalog metadata are themselves sensitive: a column-level graph tells a reader exactly where personal data lives and which extracts carry it, which is useful to an engineer and equally useful to an attacker (Data Classification).
Operating it
- Freshness per serving dataset, with its target on the same axis. One number per table that a consumer can read without asking anyone (Freshness Monitoring).
- Row count per dataset per run, compared with the same weekday historically rather than with a fixed threshold (Volume Anomalies).
- Check-result history as a queryable table: dataset, check, period, passed, detail. This is the dataset that tells you which parts of the platform are actually fragile (The Data Quality Dashboard).
- Monitoring coverage as a percentage of tier-1 datasets, trended. It is the only signal that detects the platform outgrowing its own instrumentation.
- At ten times the number of datasets, hand-written checks stop being viable and the leverage moves to defaults — every table registered in the catalog gets freshness and volume automatically, with opt-in for anything more expensive.
- At a hundred times, the bottleneck is triage rather than detection. Grouping alerts by lineage — one incident for the root dataset instead of forty for its descendants — is what keeps the channel readable (Impact Analysis).
- Consumer count scales the *communication* problem. Fifty dashboards on one broken table means fifty people forming their own theory unless there is one place that says which datasets are currently degraded.
- Signal cost is dominated by how often each check queries the data and how much it has to read. A freshness check is a
max()on a clustered column and is close to free; a distribution check over every column of a wide table is a scan, and running it hourly on a large table is a real line item (Scan Cost). - Cardinality is the other driver. One metric per dataset is cheap; one metric per dataset per partition per column is a combinatorial explosion in the metrics backend, which is exactly the trap Observability & Performance documents under Cardinality: The Label That Took Down Monitoring.
- Retention of signal history is a deliberate choice, not a default: baselines need seasons, so a year of daily aggregates is usually the right shape and a year of per-run raw detail is not.
- Every signal added is a query that runs forever, a threshold that will need tuning, and a potential false positive. Instrumenting everything uniformly costs more than it detects, and the cost is paid in attention rather than in compute.
- Checking before publish makes data later and safer; checking after publish makes it faster and occasionally wrong. Neither is universally right and the answer differs per dataset, which is why tiers exist.
- Buying a data observability product buys coverage quickly and couples your incident response to a vendor's idea of what a dataset is. Building it buys fit and costs an ongoing engineering commitment that nobody budgets for.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe five signals — freshness, volume, schema, distribution, lineage — are properties of datasets rather than of tools, and can be computed with plain SQL against any store that answers queries. What varies is how much of it a platform gives you for free.
- TOOL-SPECIFICOrchestrators differ in what they even model: a task-centric scheduler can only tell you a process ran, while an asset-centric one records which dataset a run materialised and when — so the same alert is a code signal in one and a data signal in the other.
- ORG-SPECIFICTiering, ownership and escalation are organisational rather than technical. In a three-person team the tier of a dataset is remembered; above roughly thirty datasets it must be written down or alerting degenerates into a channel nobody reads.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns how the monitoring configuration itself is versioned, reviewed and deployed. Checks defined by hand in a UI drift from the models they guard within a quarter; that domain is being built separately.
- — Distributed Systems owns why a signal computed on one machine can disagree with the same signal computed on another, and what a consistent snapshot of a distributed dataset even means.