The question this answers
The window already emitted its answer and a record for it just showed up. Now what?
With an allowed-lateness setting of L, records arriving within L of a window’s end are incorporated and the window’s result is updated or emitted late. Records arriving after L are not incorporated by that window and their fate is whatever you configured — dropped, or routed to a side output. Nothing guarantees a record arrives within L, because arrival delay has no upper bound.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The processor knows this record’s event time is older than the current watermark, so it belongs to a window it has already closed or is about to. It does not know whether more such records are coming, whether the downstream consumer of the earlier result has already acted on it irreversibly, or whether this record is late because of a transient blip or because a whole source has been offline for a day. All three call for different handling and are indistinguishable at the moment of arrival.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The three options, and who pays for each
Once a window has fired, a record belonging to it can be handled in exactly three ways. There is no fourth, and every framework offers some combination of these.
Drop it. Simple, bounded memory, no downstream complication — and silent, quiet under-counting. The 10:00 hour is permanently missing revenue that genuinely occurred. This is acceptable for sampled telemetry and unacceptable for anything financial, and the danger is that it is usually the default, so it gets chosen by not choosing.
Update the result. The window re-fires with a corrected value and downstream must handle a restatement: an update rather than an append. This is correct and it pushes real complexity outward — every consumer of that result must be idempotent and must accept that a number it already showed can change. A dashboard can handle this; an email that said "your daily total was $412" cannot.
Side-output it. Route late records to a separate stream for inspection, reconciliation, or batch correction. The main pipeline stays simple and bounded, and someone still has to look — which makes this the A Dead-Letter Queue Is a Workflow, Not a Bin pattern in a new costume, with the same failure mode: a side output nobody reads is a slower way of dropping records.
The honest framing is that lateness is not a bug to be eliminated. It is a budget: how long you are willing to wait for completeness before committing to an answer, and what you do about the tail beyond it.
| Option | Result correctness | Downstream burden | Memory | Right for |
|---|---|---|---|---|
| Dropprotocol | Silently under-counts | None | Bounded | Sampled telemetry, best-effort metrics |
| Update / re-firetypical | Eventually correct | Must handle restatements idempotently | Held for allowed lateness | Dashboards, derived views, anything upsertable |
| Side outputtypical | Main result under-counts; correction possible | Someone must process the side stream | Bounded in the main job | Billing, audit, anything requiring an explanation |
| Allowed lateness = 0protocol | Under-counts by the whole late tail | None | Minimal | Low-latency alerting where completeness is not the point |
| Allowed lateness = very largeassumption | Nearly complete | Restatements arrive long after the fact | Key cardinality × many open windows | Rarely — usually a batch job is the better answer |
Restatement is a contract, not an implementation detail
Choosing to update means every downstream consumer must be built for it, and this is the part that is usually discovered late. A window result is no longer an event ("the 10:00 hour was $12,400") but a fact that may be revised ("the 10:00 hour is currently believed to be $12,400"). Those are different contracts and they require different consumer code.
Concretely: outputs must be keyed by window so a later value overwrites rather than adds — an upsert, not an insert. Consumers must be idempotent, because the same window may be emitted several times. Anything irreversible triggered by the result — an email, a payment, an alert page — must either wait until the window is final or be capable of correction. And "final" must be defined and communicated, or downstream cannot tell a provisional number from a settled one.
The pattern that works is to emit results with an explicit status: provisional while the window can still change, final once allowed lateness has passed. Downstream can then choose — display provisional numbers, but only bill on final ones. Without that flag, every consumer must guess, and they will guess differently.
Why records are late, and why the reasons matter
Lateness has distinct causes with very different distributions, and a single allowed-lateness number is being asked to cover all of them.
Structural lateness is inherent to the source: mobile clients that batch uploads, devices that go offline, partners that deliver hourly files. This tail is measured in hours or days and is not going away. Operational lateness comes from your own pipeline: consumer lag, a rebalance, a backlog drain. This is usually minutes and is under your control. Pathological lateness is a bug or a wrong clock — a record dated last year, or a producer that was buffering for a week.
Sizing allowed lateness from the observed p99 of skew covers operational lateness and the near tail of structural lateness, and misses the far tail by construction. That is generally the right trade, because covering the far tail means holding window state for days. But it means the far tail must go *somewhere* — which is the argument for a side output over a drop, in any pipeline where those records represent money or obligations.
One caution: lateness distributions are not stationary. A new mobile app version with different upload batching, a new partner integration, or a change in retry policy shifts the distribution, and an allowed-lateness value calibrated last quarter silently starts dropping more. Monitor the fraction of records arriving after the threshold, not just the threshold.
event-time skew (processing_time - event_time), last 7 days
p50 1.2s
p90 4.8s
p99 47s
p99.9 19m <-- mobile clients reconnecting
max 3d4h <-- one device offline over a long weekend
allowedLateness = 1h covers ~99.97% of records
~0.03% are side-outputted, ~40k records/day
those 40k are 0.4% of revenue -- too much to drop,
so they go to a reconciliation job, not /dev/null.Triggers: emitting more than once on purpose
The choice is not only "when does the window fire" but "how many times". A trigger decides when a window emits, and a window may emit repeatedly as more data arrives — early and provisional, then again on the watermark, then again for each late record within allowed lateness.
This dissolves the apparent conflict between latency and completeness. You do not have to choose between a fast wrong answer and a slow right one; you can emit an early estimate at 10:01 for the operations dashboard, the watermark-triggered result at 11:00, and corrections until 12:00. Each emission is labelled with its status, and each consumer picks the point on the curve it needs.
The cost is entirely downstream. Multiple emissions per window mean consumers must be keyed and idempotent, storage sees more writes, and anyone reading a number must know which emission they have. That is a real burden, and it is worth paying only where both low latency and completeness are genuinely required. Where they are not, one trigger at the watermark is simpler and easier to reason about — and the honest default.
Key points
- A late record has exactly three fates: dropped, incorporated with a restated result, or routed to a side output.
- Dropping is the usual default and is silent under-counting; choose it deliberately or not at all.
- Updating makes the window result a revisable fact, which every downstream consumer must be built to handle — keyed, idempotent, and aware of provisional versus final.
- Allowed lateness is a budget balancing completeness against window state; size it from the observed skew distribution and monitor the fraction beyond it.
- Triggers let one window emit early, on time, and corrected — trading downstream complexity for both low latency and completeness.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A record arrives whose event time falls in a window whose end has already passed the watermark.
- • The processor compares the record’s event time against the window end plus allowed lateness.
- • Within allowed lateness: the record is added to the still-retained window state and the window re-fires with an updated result.
- • Beyond allowed lateness: the window state no longer exists, so the record is dropped or emitted to a side output according to configuration.
- • When the watermark passes window end plus allowed lateness, the window is finalised and its state is released.
- • A late record is dropped silently and an aggregate under-counts with no signal.
- • A restatement reaches a consumer that appends instead of upserting, double-counting the window.
- • An irreversible downstream action consumes a provisional result.
- • Allowed lateness is raised to improve completeness and window state exhausts memory.
- • The lateness distribution shifts after a client change and the drop rate rises without anyone noticing.
- • A side output exists but has no consumer, so late records accumulate and expire.
- • Silent under-count: the operator sees hourly revenue consistently 0.4% below the ledger. There are no errors, the pipeline is healthy, and the gap is late records being dropped by an allowed-lateness default nobody chose.
- • Double-counted restatement: after enabling window updates, downstream totals are roughly double for the affected windows because the consumer inserts rather than upserts. Both the pipeline and the consumer report success.
- • Invoice correction storm: billing acted on provisional window results and late records restate them. The operator sees a stream of credit notes and a support queue, caused by a missing final-versus-provisional distinction.
- • Memory exhaustion after a lateness increase: allowed lateness is changed from 1 hour to 24 to reduce drops, and the job OOM-loops at the next peak because open windows increased 24-fold.
- • Drift in the lateness tail: a mobile release changes upload batching and the fraction of records beyond allowed lateness rises from 0.03% to 3%. No alert exists on that number, and the aggregate quietly degrades over weeks.
- • Abandoned side output: a late-records stream has 11 million records and no consumer. It was created during the original design as the responsible choice and never wired to anything.
- • None to handle an individual late record — it is a local decision by one operator against its own watermark.
- • Real coordination appears downstream: a restated result must reach every consumer, and consumers must agree on what "final" means. Without an explicit status flag, they will not.
- • Deciding to finalise a window is committing to an answer under uncertainty, which is the same shape as any No Heartbeat Does Not Mean Dead timeout — you act because waiting forever is not an option, and you accept a bounded chance of being wrong.
- • Records arriving within allowed lateness are incorporated, and results converge to the correct value for that portion of the tail.
- • Records beyond it are never incorporated by the window, regardless of retries, restarts or replays of the live pipeline.
- • A replay from the log *can* recover them, because the records still exist there — which is why the log’s retention is the real backstop for the lateness decision.
- • Detect: alert on the rate of records arriving beyond allowed lateness, expressed as a fraction of volume and as a fraction of value.
- • Contain: for a sudden spike, raise allowed lateness only after checking window-state memory headroom; the fix can cause a worse outage than the problem.
- • Recover: reprocess the affected event-time range from the log, where all records — late ones included — are present and correctly timestamped.
- • Reconcile: compare recomputed windows against emitted results and publish corrections through the restatement path.
- • Verify: drop rate back to baseline, side output being consumed, and recomputation matching the ledger.
- • Late-record rate, split into within-allowed-lateness and beyond, as counts and as business value.
- • The full skew distribution over time, so a shift is visible before it becomes a discrepancy.
- • Window re-fire count per window, which measures how much restatement downstream is absorbing.
- • Side-output depth and age, with an owner — the same discipline a DLQ needs.
- • Open window count and state size, which bounds how much allowed lateness you can afford.
- • Any source with genuinely delayed delivery — mobile, IoT, partner feeds, third-party webhooks — where dropping the tail is materially wrong.
- • Pipelines whose consumers can absorb restatements: dashboards, materialised views, search indexes.
- • Any aggregate that will later be compared against a system of record, where a systematic under-count is discovered by someone else.
- • Low-latency alerting, where waiting for completeness defeats the purpose and a small under-count is irrelevant.
- • High-cardinality keys with long lateness, where window state dominates the cost of the job.
- • Downstream systems that cannot accept a revised number — anything that emails, bills, or pages on a provisional result.
- • Allowed lateness of zero plus a nightly batch recomputation over stored records: the streaming job stays simple and correctness is restored out of band. Frequently the best overall design.
- • Side output plus a reconciliation job, keeping the main pipeline bounded and making the correction explicit and auditable.
- • Ingestion-time windows, which have no late records by construction — you have redefined the question rather than answered it, and for some metrics that is legitimate.
- • Emit provisional and final results as distinct, clearly labelled outputs, letting each consumer choose its point on the latency/completeness curve.
The window already fired, and a record for it just showed up
What people believe, and what is true
Late events are rare.
They are structural for any mobile or IoT source. The far tail is measured in days, and it is the tail that contains the interesting records.
A bigger allowed lateness is strictly better.
It multiplies open windows and therefore memory, and it delays finality for everything downstream. It is a resource and latency decision, not only a correctness one.
The window fired, so the answer is final.
Only if allowed lateness has also passed. Firing and finality are different events, and downstream needs to be told which one it is looking at.
We side-output late records, so nothing is lost.
Only if something consumes the side output. An unread side output is a slower drop with a better name — the same trap as an unowned DLQ.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
An event shows up after its window closed. You can ignore it, correct the answer, or set it aside for later. Ignoring is the usual default and quietly under-counts.
Practical
Set allowed lateness from the measured p99 of skew and monitor the fraction beyond it. Side-output the far tail with an owner and a reconciliation job. If you restate, key outputs by window, make consumers upsert, and label results provisional or final so nothing irreversible fires on a provisional number.
Advanced
Firing a window is committing to an answer under permanent uncertainty, which makes allowed lateness the same kind of parameter as a failure-detector timeout: it trades the probability of being wrong against the delay before acting, and no value eliminates either. That is why the durable designs do not try to be complete — they make results revisable and label their confidence, so the pipeline emits a converging sequence of statements rather than a single claim of truth. Consumers then choose their own point on the trade, which is the only structure that lets one aggregate serve both an alerting dashboard and a billing run.
Apply it
- 🔧 Measure the skew distribution of a real topic and compute what fraction of records — and what fraction of revenue — falls beyond each candidate lateness value.
- 🔧 Implement provisional/final labelling on window output and demonstrate a downstream consumer that bills only on final and displays provisional.
- ⚡ Hourly revenue is consistently 0.4% below the ledger and nothing errors. Find the cause and choose between the three options with a justification.
- ⚡ A mobile release changes upload batching and your drop rate rises 100-fold. Which metric should have caught it, and what do you change first given memory constraints?
- 💬 An event from 10:00 arrives at 10:05 and the window already fired. What are your options and who pays for each?
- 💬 How would you choose an allowed-lateness value, and what would make you change it later?
- 💬 Your window results are restated. What must every downstream consumer do differently?