Late-Arriving Data
An event that happened on Tuesday and arrived on Thursday, after Tuesday was already computed, published and read.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A record belonging to a partition you closed three days ago has just landed. Where does it go, and who has to be told that Tuesday changed?
Anyone who reads a period after it has been published and assumes it is final. A finance team closing a month needs to know the date after which the number stops moving; a daily operational report needs the opposite — the freshest available answer, revised as often as necessary. The same dataset usually has both consumers, and they want incompatible things.
The unit is the (event, partition) assignment: one arriving record and the partition it belongs to by event time, which may not be the partition that is currently open. Everything in this lesson follows from those two times being different.
Partition on arrival. Whatever lands today goes into today's partition, the day closes when the clock does, and every partition is complete the moment it is written. This is simple, requires no state, and is correct for any source where arrival and occurrence are the same event — which is fewer sources than anyone expects.
A mobile client buffers events while offline and uploads them two days later. Under arrival partitioning, Tuesday's purchases are counted on Thursday, so Tuesday looks quiet, Thursday looks strong, and both numbers are wrong in ways that cancel at the month level and mislead at the day level.
- A mobile client buffers events while offline and uploads them two days later. Under arrival partitioning, Tuesday's purchases are counted on Thursday, so Tuesday looks quiet, Thursday looks strong, and both numbers are wrong in ways that cancel at the month level and mislead at the day level.
- A payment settles four days after the order. The order is in Monday's partition and its settlement is in Friday's, so any model that joins them within a partition finds neither (Stream Joins).
- A CDC connector is down for four hours and catches up afterwards. Every change it emits is late by construction, and a pipeline that closed the window on arrival has assigned them all to the wrong period (CDC Failure Modes and the Retention Deadline).
- The batch closes its window strictly on arrival time, so events that happened inside the period but arrived outside it are counted nowhere at all — the period looks merely quiet rather than incomplete, which is the exact behaviour the pipeline model in
src/de/sim/pipeline.tsproduces under its late-events fault. - A corrective record arrives for a period that has already been reported externally, so the choice is between a number that is right and a number that matches what was published — a business decision that arrives disguised as an engineering one.
- Late data is handled by re-running the affected partition every night for the last thirty days, which works and quietly makes the nightly job thirty times as expensive as it needs to be (Compute Waste).
What is actually happening
- Every record carries at least two times: when it happened and when it arrived. Partitioning by event time is what makes a period mean something to a consumer; partitioning by arrival time is what makes it cheap to write. Late data is the gap between them made visible (Event Time).
- A partition is not finished when its period ends. It is finished when you decide it is, and that decision is a policy with a cost on each side: decide early and the number is wrong, decide late and the number is unavailable (Processing Time).
- There are exactly three ways to accommodate a record that belongs to a closed partition. Update the partition — go back and recompute it. Merge incrementally — apply the record into the existing partition without a full rebuild. Delay finalisation — do not close the partition until the lateness you actually observe has passed.
- The stream-processing counterpart of this problem is watermarks and allowed lateness, which decide window membership continuously rather than per batch. The mechanism is genuinely the same question at a different cadence, and the depth lives in Late Events and Watermarks rather than here.
- Lateness is a distribution, not a constant. Most records arrive promptly, a tail arrives within hours, and a thin tail arrives for weeks. Any policy is a cut point on that distribution, and the honest way to choose it is to measure the distribution rather than to pick a round number (The Dimensions of Data Quality).
- Whatever policy you choose, a partition that is revised after publication is a restatement and inherits every concern from Backfills — including that consumers who already read it need to be told.
Two times, one partition
The whole problem fits on one timeline. Events happen at one moment and arrive at another; a partition covers a range of the first and is written at a moment on the second. Every strategy in this lesson is a different answer to the question of when to stop waiting.
The timeline below shows a single day's partition and four records. Two arrive promptly and are uncontroversial. One arrives after the batch's cut-off but inside a seven-day revision window, so it lands in the right period and changes a number somebody may already have read. One arrives after the window and has nowhere to go.
The last record is the important one, because it is the one whose handling reveals the policy. Silently dropped, it is a permanent, invisible shortfall in that period. Quarantined, it is a measurement of the window being too narrow. Applied by a deliberate backfill, it is a restatement with a reason attached to it.
| Event | Happened | Arrived | Lands in |
|---|---|---|---|
| e1 | Tue 09:14 | Tue 09:14 | Tuesday partition The ordinary case: a server-side event, stored within a second of happening. Nothing here needs a policy. |
| e2 | Tue 23:52 | Wed 00:36 | Tuesday partition Arrived on Wednesday, happened on Tuesday. Correct under event-time partitioning; counted on the wrong day under arrival partitioning, which is the most common quiet error in this lesson. |
| e3 | Tue 18:20 | Thu 11:05 | Tuesday partition, by revision A mobile client that was offline. Inside the seven-day revision window, so Tuesday is recomputed and its published number moves — which is correct, and which somebody has already screenshotted. |
| e4 | Tue 16:40 | Tue +11d | Nowhere, unless quarantined Beyond the revision window. Dropped silently it is an invisible permanent shortfall; quarantined it is evidence that the window is too narrow; applied by hand it is a restatement of a month that was closed. |
Clock labels on a teaching timeline, not measurements. The shape is the point: promptness is the common case, and the policy exists entirely for the tail.
Three ways to accommodate a late record
Once a record belongs to a partition that is already published, there are only three things you can do with it, and each buys a different property. They are not exclusive — most mature platforms use delayed finalisation for the freshest periods and partition updates for the tail — but they should be chosen rather than inherited.
The criteria that decide are the lateness distribution of the source, the cost of rewriting a partition, and how much a consumer cares about a number being final. That third one is the one engineers under-weight, and it is usually the one the business cares about most.
Whichever is chosen, the same precondition applies: the partition write has to converge under repetition. Every one of these strategies re-writes a period that has already been written, which is an unsafe operation in an append-mode pipeline no matter how the record got there (Upserts and Merges).
What does this dataset owe its consumers — the freshest answer, the final answer, or the cheapest one?
when Lateness is measurable and bounded, partitions are cheap to rewrite, and consumers accept that recent periods are revised.
cost Rewriting whole partitions on a cycle, so cost scales with the revision window rather than with the number of late records. Simple, stateless and the most common answer (Full Refresh vs Incremental).
when Volume makes full partition rewrites unaffordable, and the target supports a merge on a business key.
cost State and complexity: a key that must be unique, more small writes, and a compaction obligation afterwards. Buys revision at fine grain and continuous cost (File Compaction).
when Consumers need a number that does not move more than they need it early — month-end close, regulatory reporting, anything published externally.
cost Availability. The period is unavailable or explicitly provisional for the length of the delay, and no delay makes it truly complete — it only makes what is missing small (Cost vs Freshness).
when The record is beyond the revision window and the period has been reported externally.
cost A side table nobody reads unless prompted, plus a periodic decision about whether to restate. Correct, honest, and it fails quietly if nobody ever looks at it (Data Incidents).
when The record is genuinely negligible, the window is calibrated against a measured distribution, and the loss is counted and reported.
cost A known, quantified, permanent shortfall. This is a legitimate choice only when the count is monitored — dropping silently is not a policy, it is the absence of one (Missing Rows).
What one row means changes when a period reopens
Late data quietly changes the grain question. A partition that can be revised is not "the orders of Tuesday" — it is "the orders of Tuesday as known at the time you read it", and those are different datasets with different valid uses.
That distinction is invisible in the table, which is why it has to live in the contract and in whatever a consumer sees next to the number. A downstream model that snapshots a revised partition and one that recomputes from it will disagree, and both will be internally consistent.
The last row is the trap that catches careful teams: a report generated during the revision window is a legitimate artefact that will never match the finalised data, and reconciling them later is an investigation into a difference that is not an error.
| Stage | One row is | Breaks if |
|---|---|---|
| Raw arrival file | One record as delivered, stamped with both event time and arrival time. | Only arrival time is retained. Lateness then becomes unmeasurable and every policy below is guesswork (The Raw Landing Zone). |
| Event-time partition, before cut-off | One event that happened in this period and has arrived so far. | It is read as "all events of this period". It is a prefix, and the share missing is exactly the lateness tail. |
| Event-time partition, published | One event in this period, as known at cut-off. | A consumer treats it as final. Under any revision policy it is not, and nothing in the table says so (Data Contracts). |
| Event-time partition, after revision | One event in this period, as known now — a different set from the published one. | Two consumers read the same partition on different days, get different totals, and one of them opens an incident. |
| Finalised partition | One event in this period, frozen. Changes only through an explicit backfill. | Finalisation was never declared, so nothing is ever frozen and no number can be cited (Backfills). |
| A report generated during the window | One event as known on the day the report was run — a snapshot of a moving dataset. | It is reconciled against the finalised data later and the difference is investigated as an error rather than recognised as revision. |
Revision does not change what a row is. It changes what the *set* of rows is, which is the thing every aggregate depends on and the thing no schema records.
How to build it
Most important first.
- Partition on event time and carry arrival time as a column. That single decision makes the period mean what consumers think it means, and makes lateness measurable rather than invisible.
- Measure the lateness distribution per source before choosing a policy. Sources differ enormously — a server-side event stream and a mobile SDK have nothing in common here — and a single platform-wide rule will be wrong for both (Ingestion Sources).
- Make the partition write idempotent so that revisiting a period is a merge rather than an append. Every late-data strategy reduces to re-writing a closed partition, which is only safe when the write converges (Upserts and Merges).
- Restrict the revision window explicitly: the last N periods are open to revision, older ones are frozen and only change through a deliberate backfill. Bounded state, bounded cost, and a clear answer to "is this final".
- Publish the finalisation policy as part of the dataset's contract — "revised for seven days, final after that" — so a consumer can decide when to snapshot rather than discovering the number moved (Data Contracts).
- Expose a per-partition completeness or revision indicator alongside the data, so a dashboard can show that the last three days are provisional instead of presenting them with the same authority as last month (The Data Quality Dashboard).
- Track records that arrive beyond the revision window rather than dropping them silently. They are a measurement of the policy being wrong, and dropping them destroys the only evidence (Missing Rows).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Event-time partitioning guarantees that a record is counted in the period it belongs to, provided you eventually process it. It guarantees nothing about when that period becomes complete.
- A bounded revision window guarantees that a partition older than the window will not change without an explicit backfill. That is the guarantee finance actually needs, and it is the one most platforms have never stated.
- Delayed finalisation guarantees a more complete number and explicitly does not guarantee a *complete* one — there is no cut point past which nothing arrives, only one past which very little does.
- Nothing guarantees a late record is correct. A record arriving three days late is as likely to be a duplicate of one you already have as it is to be new, which is why deduplication and lateness handling are the same subsystem (Deduplication).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check is a reconciliation of a closed period against the source, run after the revision window has passed rather than immediately. It catches the events that arrived too late to be included and the ones that were dropped (Reconciliation).
- Alongside it, a monitor on the lateness distribution itself: the share of records arriving beyond the window, per source, over time. It catches a policy that was right last quarter and is no longer.
- Both miss the events that never arrive at all. A record lost in transit is indistinguishable from a record that was very late until the day the source is queried directly (CDC Failure Modes and the Retention Deadline).
- This is the freshness lesson where the trade is explicit: every hour a partition is held open buys completeness and costs availability. The right cut point comes from what the number is for, not from what the pipeline can do (Cost vs Freshness).
- A dataset with a revision window has two freshness numbers — how recent the newest data is, and how long ago the newest *final* period was — and consumers need the second one far more often than they are given it.
- Reprocessing recent partitions nightly gives good completeness with a coarse revision cadence. A merge on arrival gives fine-grained revision at the cost of touching the target continuously, which competes with the queries reading it (Incremental Processing).
- If the producer changes how it stamps event time — client clock to server clock, local time to UTC — the lateness distribution changes shape overnight and the policy calibrated to the old one is silently wrong (Semantic Changes).
- A new source with a different lateness profile added to an existing dataset inherits the dataset's revision window whether it fits or not, and that is usually where a platform-wide rule first hurts.
- Adding event-time partitioning to a dataset that was arrival-partitioned is a full rebuild of history, because every record needs re-assigning. It is the migration people postpone until the numbers are wrong enough to force it (Full Refresh vs Incremental).
- A late record that missed its window is recovered by reprocessing its partition from raw — which works if raw retained it, and confirms once more that the raw layer is the recovery position for everything in this module (Keeping Raw History: The Recovery Position and the Liability).
- A period that was finalised too early and then corrected is a restatement, and goes through the backfill path with validation and an announcement rather than being quietly rewritten (Planning a Backfill).
- Records that arrived beyond the revision window and were quarantined rather than dropped can be applied later in a single deliberate operation. Records that were dropped cannot be recovered from anything except the source.
What can go wrong
- Arrival-time partitioning silently moving activity between periods, so both periods are wrong and the total is right.
- A window closed strictly on arrival, counting late events nowhere — the period looks quiet rather than incomplete.
- A revision window shorter than the real lateness tail, so a stable share of every period is permanently missing.
- A revision window so long that nothing is ever final, and finance snapshots the data themselves to get a number that stops moving.
- Reprocessing the last thirty days nightly to be safe, which works and multiplies the daily cost by thirty (Compute Waste).
- Late records deduplicated against a window that has already expired, so a re-delivered event three days late is inserted as new — the mitigation failing rather than the operation (Deduplication).
- "Late data is a streaming problem." It is a batch problem with a longer cycle, and it is worse in batch because the window is closed by a schedule rather than by a watermark that can wait (Late Events).
- "We partition by ingestion date, which is close enough." It is close enough exactly when lateness is negligible, and the way to know that is to measure it rather than to assume it (Ingestion Time).
- "Yesterday's number is final." Yesterday's number is final when the revision policy says so. Without a stated policy every number is provisional and nobody knows which ones (The Freshness SLO).
- "Just reprocess the last thirty days every night to be safe." That is a real strategy with a real cost, and choosing it without measuring the lateness distribution means paying thirty times over for a window that is probably three (Compute Waste).
Operating it
- The distribution of arrival minus event time, per source, as a percentile chart over time. This one chart determines the policy and detects every change to it (Freshness Monitoring).
- Rows added to already-published partitions, per day. A partition that keeps growing after publication is either healthy revision or an ingestion problem, and the volume tells you which (Volume Anomalies).
- Count of records rejected for arriving beyond the revision window, by source. It should be small, stable and non-zero; any of those changing is a signal.
- Per-partition finalisation state exposed to consumers, so "is this number final" is answerable without asking an engineer (Dataset Documentation).
- At 10x volume the nightly rebuild of a wide revision window becomes the dominant cost in the platform, and the answer is to narrow the window using the measured distribution rather than the assumed one.
- At 100x, per-partition rebuilds stop fitting the schedule entirely and the strategy has to become an incremental merge into open partitions, with all the state and complexity that implies (Incremental Processing).
- More sources multiply the problem rather than adding to it, because each has its own lateness profile and the dataset's window must accommodate the worst — which is an argument for per-source policies at scale (Ingestion Sources).
- Revisiting closed partitions costs a rewrite of each, so the cost scales with the revision window times the per-partition write cost, every cycle. This is the term that makes an over-wide window expensive (File Compaction).
- Delayed finalisation costs nothing in compute and costs availability, which is charged to consumers rather than to the platform and therefore rarely appears in a cost review.
- Merging late records individually costs more small writes and more small files, which degrades read performance until compaction runs (File Size and the Small-Files Problem).
- Event-time partitioning is correct and makes writes more expensive: every batch touches several partitions instead of one, producing more, smaller files.
- A long revision window buys completeness and costs both compute and the ability to call any recent number final. A short one is cheap, decisive and quietly incomplete.
- Exposing provisional-versus-final state to consumers is honest and complicates every dashboard that shows a recent period. Most teams choose not to, and then explain it verbally, once per consumer, forever.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALEvent time and arrival time diverge in every system where the producer is not the storer. What varies is the magnitude: a server-side database CDC stream is late by seconds, a mobile SDK with offline buffering is late by days, and one policy cannot serve both.
- SOURCE-SPECIFICMobile clients buffer while offline and can arrive days late with clock skew on top; a payment provider settles on its own schedule and is late by design rather than by failure; a database CDC stream is late only when the connector is behind. The distributions have different shapes, so the same revision window means different completeness for each.
- SIMULATEDThe "counted nowhere" behaviour comes from the pipeline model in
src/de/sim/pipeline.ts, which filters records whose arrival falls outside the closed window and reports what the dashboard shows versus what is true. It is a deterministic teaching model, not a measurement.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why event time and arrival time cannot be reconciled in general: there is no global clock, and the ordering a receiver observes is not the ordering in which things happened.