Stale Dashboards
A complete, plausible, internally consistent number for a day that ended two days ago. The failure mode that looks most like health.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
The dashboard renders a full set of numbers for "yesterday" and nothing on the page says when the data was last updated. How would anyone know it is Tuesday's?
Anyone reading a number to decide something now: an on-call engineer checking whether a release moved a metric, an ops team watching a queue, a trader-style decision on the day. Staleness hurts in proportion to how time-sensitive the decision is, and not at all for a monthly close.
The unit is one serving dataset and its clock. Freshness is a per-dataset property, and there are at least three clocks in play: when the event happened, when it arrived, and when the dataset was last published. Averaging them, or publishing one and calling it the freshness, is the mistake (Freshness Monitoring).
Trust that a dashboard which renders is a dashboard which is current. There is no error, the layout is complete, every tile has a value, and the chart runs right up to the edge. Nothing on the screen suggests otherwise, which is exactly the problem.
The transformation raised and the run was abandoned, so nothing was published and the serving table still holds the previous period. The orchestrator is red, and nobody looks at the orchestrator (When a Task Fails Mid-DAG).
- The transformation raised and the run was abandoned, so nothing was published and the serving table still holds the previous period. The orchestrator is red, and nobody looks at the orchestrator (When a Task Fails Mid-DAG).
- The schedule was paused during an incident three weeks ago and never resumed. Every task is green because no task ran (Orchestration).
- The upstream source stopped sending. The pipeline runs on schedule, processes zero rows successfully, and publishes an empty increment on top of yesterday's data (Ingestion Failure & Recovery).
- The model is current and the mart it feeds refreshes on a slower schedule, so the tile is an hour behind the table an analyst would query directly and neither of them says so (Data Marts).
- The BI tool cached the query result and is serving it from cache. Every dataset downstream of the warehouse is fresh and the tile is not (Caching as a Contract Clause).
- The chart's last point is the current, incomplete period, so it always dips at the right-hand edge. Everyone learns to ignore the last point, and then a genuine drop is ignored too (Late-Arriving Data).
- A
max(updated_at)freshness check reads the *source's* clock, which the source still updates, so the check is green while the pipeline that copies those rows has been dead for a day.
What is actually happening
- Staleness is the absence of an event, and absence is structurally harder to detect than presence. Every other failure in this module produces something — wrong rows, missing rows, extra rows — that a query can find. Staleness produces nothing at all, so the only way to see it is to compare against a clock (Freshness Checks).
- A dashboard renders whatever the serving table contains. It has no concept of when that content was written, and it will present a two-day-old figure with the same visual confidence as a two-minute-old one.
- There are three distinct clocks and conflating them is the most common freshness bug. Event time is when the thing happened. Arrival time is when it reached you. Publish time is when the serving dataset was last written. A consumer needs the third and usually gets a query against the first (Event Time, Processing Time).
- End-to-end freshness is the sum of every hop's delay, dominated by the coarsest schedule in the chain. One fast hop does not make a platform fresh, and describing it as real-time because the ingestion is streaming misleads every consumer who hears it (Batch vs Streaming Ingestion).
- A period that has not closed is not stale — it is incomplete, which looks identical and requires the opposite response. Widening a lateness window fixes one and does nothing for the other (Watermarks).
- In the in-repo model, an abandoned transform is the loudest fault available: nothing is published, so completeness, freshness, distribution and reconciliation all fail together. The genuinely dangerous version of staleness is the one where a run *does* publish — an empty or unchanged increment — and only the freshness check has anything to say (The Pipeline Succeeded. The Data Is Wrong.).
Stale is the failure that looks most like health
Every other failure in this module leaves a trace in the data. Missing rows leave a shortfall; duplicates leave repeats; a broken transform leaves a number that disagrees with the source. Staleness leaves a dataset that is internally perfect — complete, unique, well-typed, correctly distributed — and simply old.
That is why it needs a clock rather than a query. No amount of inspecting the rows will reveal that they are yesterday's, because yesterday's rows were correct yesterday and have not decayed. The only evidence is a comparison between a publish time and now, and if nobody recorded the publish time there is no evidence at all.
Work down the table looking at the symptom column. Four of the six produce a completely green orchestrator, and one of them produces a green *freshness check* as well. That last row is the one that costs a platform its credibility, because the monitoring existed, ran, and reported success throughout.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| The transformation raises and the run is abandoned. | The serving table still holds the previous period. The orchestrator is red — and this is the good case, because something is loud (When a Task Fails Mid-DAG). | Nothing was published. The failure is genuine, attributable and already alerting. | Fix and re-run. Verify what was published rather than that the task turned green; then ask why the consumer noticed before the alert did. |
| A schedule was paused during a previous incident and never resumed. | Perfect silence. No failures, no alerts, no runs. The dataset simply stopped moving. | A paused DAG emits nothing to fail on, and most status pages render "no recent failures" identically to "healthy" (Orchestration). | Alert on the absence of a successful publish within the expected interval, not on the presence of failures. Track pause state as a monitored fact. |
| The upstream source stops producing. | Every run succeeds, quickly, having processed zero rows. Volume drops to a flat line that nobody has a chart for. | An empty input is a valid input. The pipeline is behaving exactly as designed (Ingestion Failure & Recovery). | Treat zero rows as suspicious. Assert a minimum expected volume per run for datasets where zero is not a normal outcome (Volume Anomalies). |
| A mart refreshes on a slower schedule than the model it derives from. | The tile and a direct query of the model disagree for part of every day, and both are correct. | The consumer-visible freshness is the sum of every hop, and the mart added one that nobody published (Data Marts). | Publish freshness per dataset, including marts. Where the gap is not acceptable, align the schedules or point the dashboard at the model. |
| The BI layer serves the query result from its own cache. | Every dataset is fresh, every check is green, and the tile is hours old. Refreshing the browser changes nothing. | A caching layer with an independent lifetime sits after every guarantee your platform makes (TTL and Expiry). | Render the underlying dataset's publish time on the dashboard rather than the query time, so the cache cannot disguise itself as freshness. |
The freshness check reads a source-maintained updated_at column. | The freshness monitor is green while the pipeline has been dead for a day. This is the worst row in the table. | The check measures whether the *source* is alive. The source is fine; the copy is not, and the check never looked at the copy. | Measure publish lag on the serving dataset — the write time of the newest complete period — and audit every existing freshness check for which clock it reads. |
Three clocks, and which one the consumer needs
Freshness confusion is almost always clock confusion. An event happened at one time, arrived at another, and was published in a dataset at a third. A consumer looking at a tile is asking about the third, will usually be shown the first, and has no way to tell them apart.
The timeline below makes the gap concrete. Events A through D all belong to the 10:00–10:05 window. Three of them arrive in time for the run that closes it; one arrives afterwards. The dataset is published at 10:07 and the dashboard is read at 10:20. At that moment the newest event time in the data is 10:04, the publish time is 10:07, and one real event from the window is missing entirely — so the number is thirteen minutes old *and* incomplete, and the tile communicates neither (Event Time, Late-Arriving Data).
Note what a max(event_time) freshness check would report here: 10:04, sixteen minutes of lag, which looks acceptable and is measuring the wrong thing. If the pipeline had died at 10:07 and the check ran at 14:00, the same query would still return 10:04 — because that is the newest event *in the table*, and a dead pipeline stops the table's clock along with everything else. The check would show four hours of lag, which is right by accident. Move the source to a system that keeps its own timestamps and the same check reads the source's clock and goes green while the copy rots (Freshness Checks).
| Event | Happened | Arrived | Lands in |
|---|---|---|---|
| A | 10:01 | 10:02 | W1 Normal path. Event time and arrival time differ by the transport delay only. |
| B | 10:03 | 10:04 | W1 Also normal. This is the newest event time the dataset will contain for W1. |
| C | 10:04 | 10:06 | W1 Arrived after the window ended but before it closed at 10:06. Counted, because closing is what matters, not the window boundary. |
| D | 10:02 | 10:14 | dropped Happened inside W1 and arrived long after it closed. Counted nowhere. The window looks quiet rather than incomplete (Missing Rows). |
| E | 10:07 | 10:08 | W2 Belongs to the next window, which is still open when the dashboard is read — so W2 is incomplete rather than stale. |
Read at 10:20, the tile shows W1. Publish lag is thirteen minutes; newest event time is sixteen minutes old; one event that belongs in the window is absent. Three different honest answers to "how fresh is this", and the consumer needs the first one.
Measuring the right clock
A freshness check is three decisions: which clock, which target, and what to do when the source legitimately produces nothing. Getting the first wrong makes the check useless; getting the third wrong makes it ignored, which amounts to the same thing by a slower route.
The target should come from the consumer. "This dataset drives a decision that is made each morning, so it must be published by 07:00 with the previous day complete" is a statement someone can agree to and be held to. "Freshness under fifteen minutes" is a number picked from what the pipeline currently achieves, and it will be adjusted whenever the pipeline gets slower (The Freshness SLO).
And put the answer where the consumer is. A monitoring system that knows a dataset is stale while the dashboard renders it without comment has located the problem and failed to deliver the finding. The last-updated stamp next to the number is worth more than the alert, because it reaches the person about to act (Dashboards Built Around Questions).
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Publish lag: now minus the write time of the newest complete period | The pipeline has produced a finished period recently enough for the decisions this dataset drives. | Crashed runs, paused schedules, stalled marts, dead sources — every mechanism where nothing new was written. | A run that published on time and published nothing, or published the same rows again. Punctuality is not progress (Volume Anomalies). |
max(event_time) in the serving table versus now | The newest thing the dataset knows about is recent. | A completely dead pipeline, eventually, because the table's newest event stops advancing with it. | The distinction between a stalled pipeline and a quiet source — and if the column is maintained by the source and copied faithfully, the check reports the source's health rather than yours. |
| Heartbeat: time since the last successful publish that changed something | The dataset is not merely being processed, it is being updated. | Empty increments, no-op runs, a source that stopped while the schedule kept firing (Pipeline SLOs). | A run that changes rows and changes them wrongly. It asserts motion, never direction. |
| Expected-period assertion before publish | The period this run was supposed to produce actually exists in the output. | A partial publish where some partitions landed and others did not, which reads as a quiet day rather than a broken run (Atomic Publish). | A period that exists and is empty, and any staleness introduced after publish — the BI cache is downstream of every assertion you can make. |
| Consumer-visible last-updated stamp on the dashboard | The person reading the number can see how old it is without asking anyone. | Everything, from the only vantage point that matters — including the BI cache, which no server-side check can see (The Data Quality Dashboard). | Nothing about correctness, and it depends entirely on the consumer looking at it. It is a control, not an alert, and it works because it is always on screen. |
The second row is included because it is the check most platforms actually have, and its blind spot — reading a clock the source maintains — is the specific reason so many teams believe they monitor freshness and do not.
How to build it
Most important first.
- Put a last updated timestamp on the dashboard itself, next to the numbers, sourced from the serving dataset's publish time rather than from a source column. A consumer who can see the clock does not need a monitoring system to catch this (The Data Quality Dashboard).
- Define a freshness target per dataset, derived from the decisions it drives rather than from what the pipeline currently achieves, and alert against the target (The Freshness SLO).
- Measure freshness as publish lag: now minus the time the newest complete period was written. Not
max(event_time), which the source keeps advancing, and not task success time, which advances even on empty runs (Freshness Monitoring). - Alert on the absence of a successful publish within the expected interval — a heartbeat — rather than on the presence of a failure. A pipeline that stopped being scheduled emits no failures at all (Pipeline SLOs).
- Make incomplete periods visually distinct: shade the current partial period, or exclude it by default. Training consumers to ignore the last data point trains them to ignore real drops too.
- Treat an empty increment as suspicious rather than as success. Zero rows is occasionally correct and usually means the source stopped (Volume Anomalies).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- An orchestrator guarantees that it attempted the tasks it was scheduled to attempt. It guarantees nothing about tasks whose schedule was paused, and a paused DAG is indistinguishable from a healthy idle one on most status pages.
- A successful run guarantees the code completed. It does not guarantee that anything was published, that what was published differs from what was already there, or that the period it covered is complete (The Pipeline Succeeded. The Data Is Wrong.).
- A freshness check guarantees only the specific clock it reads. A check on a source-maintained column guarantees the source is alive; a check on publish time guarantees your pipeline is (Freshness Checks).
- A BI tool guarantees to render the result of the query it ran, which may have been served from a cache with its own independent lifetime (TTL and Expiry).
- Nothing guarantees that fresh data is correct. Freshness and correctness are orthogonal, and a fast pipeline publishing wrong numbers publishes them promptly (The Dimensions of Data Quality).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check is publish lag against a per-dataset target: now minus the write time of the newest complete period, compared with a number someone agreed to. It catches paused schedules, crashed runs, dead sources and stalled marts.
- It misses data that is perfectly fresh and completely wrong — every other failure in this module. It also fires falsely on any period where the source genuinely produced nothing, which is why weekends and holidays are where freshness alerting loses its credibility (Alert Fatigue: The Page Nobody Reads).
- Pair it with a volume check so an empty-but-punctual run is distinguishable from a healthy one, and with a heartbeat so a dataset that stopped being scheduled is noticed at all (Volume Anomalies, Pipeline SLOs).
- The consumer-facing question is never "how fresh is the platform" but "what decision does this number drive and how stale can it be before that decision changes". Most dashboards that demand minutes are read once a day (Cost vs Freshness).
- Freshness must be published per dataset. One platform-wide number hides the single table that has not updated since Friday, which is always the one somebody is about to use.
- The gap between the freshest hop and the coarsest is where consumer expectations go wrong. If ingestion is continuous and the mart rebuilds nightly, the honest freshness of the mart is a day, and saying "we stream our data" is technically true and practically a lie (Batch vs Streaming Ingestion).
- Renaming or re-pointing a dataset silently orphans the freshness monitor, which keeps watching the old object and keeps reporting green. Monitors need the same review as consumers when a model moves (Impact Analysis).
- Adding a mart between a model and a dashboard adds a hop and therefore lengthens the consumer-visible freshness, without any single component becoming slower. This is the most common way a platform gets staler without anything breaking.
- Changing a schedule from hourly to daily is a contract change for every consumer, and it will be made in a pull request that mentions cost (The Freshness SLO).
- Resume the schedule or re-run the failed run, then verify what was actually published rather than that the task turned green — an empty successful run looks identical to a recovered one on the orchestrator (Reprocessing vs Retrying).
- If several periods were missed, backfill them in order and validate each before publishing, rather than running one enormous catch-up job that competes with the current period for the same compute (Planning a Backfill, What Backfills Break).
- Announce the gap to consumers. A dashboard that silently jumps from Tuesday to Friday causes a second incident when someone compares a screenshot taken during the gap to the corrected chart (Data Incidents).
- Where the source itself stopped, catching up may be impossible if its retention window has passed — the same unrecoverable case as a connector gap (Retention and Replay).
What can go wrong
- The freshness check reads a source-maintained timestamp column, so it stays green while the pipeline is dead. This is the single most common way freshness monitoring fails.
- Freshness alerts fire every weekend on a dataset with no weekend traffic, are silenced, and the silence outlives the reason for it (Alert Fatigue: The Page Nobody Reads).
- The dashboard shows a last-updated timestamp taken from the query execution time, which is always now, and therefore always reassuring.
- The alert routes to a channel nobody reads because the dataset's owner is a team that no longer exists (Data Ownership).
- The catch-up backfill overwrites the current partition mid-day and the dashboard briefly shows a half-built period (Atomic Publish).
- Only the final serving table is monitored, so an upstream model that stopped is invisible until its staleness propagates and by then the blast radius is every dataset downstream.
- "The DAG is green, so the data is current." A paused schedule is green. An empty successful run is green. A run that published nothing is green (The Pipeline Succeeded. The Data Is Wrong.).
- "We monitor freshness." Check which clock. A large fraction of freshness monitors read a column the source keeps updating and therefore monitor the source's health, not the pipeline's.
- "The numbers look right, so the data is fresh." Stale data looks completely right. It was right, two days ago, and nothing about it has degraded since (The Dimensions of Data Quality).
- "Making it fresher is a scheduling change." It is a cost change, a completeness change and often a correctness change, because a shorter window catches fewer late events (Late-Arriving Data).
- "The last point on the chart always dips." That is a modelling failure being normalised. Once consumers are trained to ignore the edge, a genuine drop is ignored for exactly as long as it takes someone else to notice.
Operating it
- Publish lag per serving dataset, with its target drawn on the same axis, on one page. This is the single most useful data-platform panel there is (The Freshness SLO).
- A heartbeat per scheduled dataset: time since the last *successful publish that changed something*, which is a different and better signal than time since the last successful task (Pipeline Metrics).
- Rows written per run, so an empty increment is visible as a flat line rather than as an ordinary green tick (Volume Anomalies).
- The dashboard's own last-updated field, rendered next to the numbers, so the consumer is the first line of detection rather than the last (The Data Quality Dashboard).
- Schedule state — which DAGs are paused — as a monitored fact rather than a UI detail, because a paused schedule generates no failures to alert on (Orchestration).
- At 10x datasets, per-dataset freshness targets stop being maintainable by hand and must be derived from a declared classification — every dataset in this tier gets this target — or they will be set once and never revisited (Data Contracts).
- At 100x, the freshness dashboard itself becomes unreadable and must be reduced to exceptions: datasets currently outside their target, ranked by how many consumers they have (Impact Analysis).
- Consumer growth makes staleness more expensive rather than more likely. The same one-day gap costs nothing with three consumers and generates a dozen support requests with three hundred (Who Actually Consumes This Data).
- Freshness monitoring is nearly free: one metadata query per dataset per interval. It is the cheapest check in this domain and the one most often absent (Data Observability).
- Making a dataset fresher is not free at all — a shorter interval multiplies the number of runs and, for full rebuilds, multiplies the compute by the same factor for no additional data (Compute Waste, Incremental Processing).
- The real cost trade is freshness against completeness. Publishing sooner means publishing before late events arrive, and then either living with an incomplete number or paying to revise it (Cost vs Freshness).
- A tight freshness target buys confidence and generates alerts on every ordinary variation. A loose one is quiet and lets a real stall run for hours. The target belongs to the consumer, not to the pipeline, and should be argued in terms of decisions rather than minutes.
- Showing a last-updated timestamp on every dashboard makes staleness self-evident and makes every ordinary delay visible to executives. That transparency is correct and it will generate questions you did not previously have to answer.
- Excluding the incomplete current period makes charts honest and removes the very signal some consumers use for intraday monitoring. Shading it is usually the better compromise than dropping it.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALStaleness as an absence rather than a presence, and the three-clock confusion, hold in every stack. What differs is where the stall happens: batch platforms stall at the scheduler, streaming platforms stall at a stuck consumer whose lag grows silently.
- TOOL-SPECIFICOrchestrators differ in whether a paused schedule is visible as an alertable state or only as a UI toggle, and in whether a run that processed zero rows is distinguishable from one that processed millions. Where it is only a toggle, the pause outlives the incident that caused it.
- SIMULATEDThe claim that an abandoned transform trips completeness, freshness, distribution and reconciliation together — while a published-but-empty run trips far fewer — comes from the model in
src/de/sim/pipeline.ts, asserted inscripts/de-sim.test.ts, and is a property of that teaching model.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the alerting discipline this depends on — what makes a page actionable, why a silenced alert outlives its reason, and how a paused schedule should show up on a status page.