Watermarks
An estimate of how far event time has progressed, derived from the data itself. It decides when a window may be emitted and what counts as late — and it is a claim, not a measurement.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
How does a system that can never know whether more data is coming decide that a period is complete enough to publish?
Every consumer of a windowed result, whether they know it or not. The watermark decides when their number appears and how much of the period it covers; a stalled one means no number at all, and an over-eager one means a number missing its stragglers (The Freshness SLO).
A single scalar per stream: the event-time position the system believes it has passed. One number, derived from the maximum event time observed minus a chosen allowance, and the whole correctness-versus-timeliness trade of the module is compressed into it.
Emit a window when its end time passes on the wall clock. It always works, it never stalls, and it is one line of code.
A twenty-minute upstream stall means every window in that period emits empty on schedule, and the data arrives afterwards to find its windows already closed and purged (Late Events).
- A twenty-minute upstream stall means every window in that period emits empty on schedule, and the data arrives afterwards to find its windows already closed and purged (Late Events).
- A replay of historical data emits every window instantly and empty, because the wall clock is far past all of the event times being replayed (Replay from the Log).
- One partition is slower than the others, so windows emit before its records arrive and a consistent fraction of every window is missing — a systematic undercount that looks like a business trend (Missing Rows).
- The job is restarted and catches up, processing an hour of backlog in minutes. Every window in the backlog emits on wall-clock schedule with whatever fraction had arrived, which is a different fraction each time (Deterministic Replay: Making the Schedule Reproducible).
- Two jobs computing the same metric emit at different moments and therefore include different subsets, and the two numbers never agree (Two Dashboards, Two Numbers).
What is actually happening
- A watermark is a claim the system makes: "I do not expect to see any more events with an event time before *T*". It is derived from the data — typically the maximum event time observed so far, minus a configured out-of-orderness allowance (Event Time).
- Because it is derived from event times, it advances when *later data arrives*, not when time passes. An idle stream produces no new event times, so the watermark stops, and every open window stays open indefinitely (Processing Time).
- It is monotonic by construction: a watermark never moves backwards. An event arriving with an old event time does not pull it back — it is simply late, and the watermark is unmoved.
- A window may emit when the watermark passes its end, and it is purged when the watermark passes its end plus the allowed lateness. Between those two points a late record still updates it (Windows).
- In a job with several input partitions or operators, the effective watermark is the minimum across them, because the job can only claim progress that all of its inputs support. One lagging or idle partition therefore holds the whole job back (Topics and Partitions).
- It is an estimate, and it can be wrong in both directions. Too conservative and everything waits; too aggressive and real data is classified as late. A single future-dated record makes it maximally aggressive instantly, which is the most destructive single-record failure in the module.
A claim, not a measurement
src/de/sim/stream.ts; scripts/de-sim.test.ts asserts that the trace never decreases, that the late record leaves the watermark unchanged, and that its value at that point is exactly 10:06 — the highest event time seen so far.The watermark answers a question that has no correct answer: has everything for this period arrived? Nothing in a distributed system can know that. A producer might be offline, a partition might be lagging, a retry might be in flight, and no amount of waiting proves that nothing more is coming (Ordering Guarantees: Four Levels, Four Prices).
So the system substitutes a claim it can actually make: "the highest event time I have seen is *T*, and I do not expect anything more than *d* behind it, so I will act as though event time has reached *T − d*". That is the whole mechanism. It is a heuristic with a configurable safety margin, and calling it an estimate rather than a fact is the difference between using it correctly and being surprised by it.
The timeline below traces the watermark through the canonical scenario. The rows to read carefully are d and late. Event d advances the watermark past the first window's end and thereby *closes* it — the window ended because later data arrived, not because time passed. Event late carries an old event time and so does not move the watermark at all, which is exactly what makes it late.
| Event | Happened | Arrived | Lands in |
|---|---|---|---|
| a | 10:01 | 10:01 | 10:00–10:05 Watermark → 10:01. W1 ends at 10:05, so it stays open. |
| b | 10:02 | 10:02 | 10:00–10:05 Watermark → 10:02. |
| c | 10:04 | 10:04 | 10:00–10:05 Watermark → 10:04. Still short of 10:05; W1 has not emitted. |
| d | 10:06 | 10:06 | 10:05–10:10 Watermark → 10:06, which passes W1's end. W1 emits and, with no allowed lateness, is purged. A later event closed an earlier window. |
| late | 10:00 | 10:07 | 10:00–10:05 (dropped at this setting) The watermark stays at 10:06 — an old event time never pulls it back. W1 is already purged, so this record is late and dropped. |
| e | 10:08 | 10:08 | 10:05–10:10 Watermark → 10:08. W2 is still open; nothing has advanced past 10:10. |
Final watermark after the last arrival. Two properties are visible: it is monotonic — the late record does not move it backwards — and it advances only when a record with a higher event time arrives, which is why an idle stream freezes it entirely.
Where it comes from, and the three ways it goes wrong
A watermark is generated at the source, per input, and combined downstream by taking the minimum. That rule is correct — an operator cannot claim more progress than its slowest input supports — and it is the reason a single idle or lagging partition stops the whole job.
The three failures below are distinct and demand opposite responses, which is why diagnosing them correctly matters. A stall means the watermark is not advancing and the fix is idleness handling or finding the lagging input. A poisoned watermark means it advanced far too much and the fix is validation upstream. A systematically tight watermark means it is slightly too aggressive and the fix is a wider allowance.
The declaration below is how the claim is usually made in SQL. Note what it actually says: not "data is complete", but "treat event time as having reached the maximum seen minus five seconds". Every consequence in this lesson follows from that one subtraction.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A partition or source goes idle | Output stops entirely. Consumer lag reads zero, task status is green, error rate is zero. | The effective watermark is the minimum across inputs, and an input producing no events produces no new event times. | Enable idleness detection so a quiet input stops holding the minimum back — accepting that its eventual records will then be late (Late Events). |
| A single future-dated event time | A sudden mass of records classified as late; output for the affected period collapses; recovery does not happen on its own. | The watermark took the maximum event time seen, which is now far in the future, and it cannot move backwards. | Validate event time against ingestion time before the watermark generator. Recovering already-dropped records requires a batch recomputation from raw storage (Ingestion Time). |
| The allowance is smaller than the real lateness tail | A small, steady, non-zero drop count that looks like normal operation. | The out-of-orderness allowance was chosen from a sample that did not include the tail, or the producer population changed. | Size the allowance from the measured distribution and re-measure when sources change; route the residual tail to a side output rather than widening indefinitely (Percentiles: Which One, and How Many Users Is That?). |
| A new, slower source added to an existing job | Freshness of every existing output degrades with no change to those pipelines. | The minimum-across-inputs rule now includes an input with a much larger delay. | Generate watermarks per source and, where profiles differ sharply, split into separate jobs feeding a common downstream model (Model Layering). |
| Watermark generated after a shuffle instead of at the source | Out-of-orderness looks larger than the sources actually exhibit, forcing a wider allowance than necessary. | Records were reordered by the shuffle before the watermark was derived, so the measured disorder includes the job's own doing. | Generate at the source, per partition, and let the minimum rule combine them downstream (The Shuffle). |
1CREATE TABLE orders (2 order_id STRING,3 account_id STRING,4 amount_minor BIGINT,5 occurred_at TIMESTAMP(3),6 -- THE CLAIM: "act as though event time has reached7 -- (max occurred_at seen) - 5 seconds".8 -- Bigger interval = fewer records classified late, later emission,9 -- more windows held open.10 -- Smaller interval = faster emission, more records dropped as late.11 WATERMARK FOR occurred_at AS occurred_at - INTERVAL '5' SECOND12) WITH (13 'connector' = 'kafka'14 -- ... source configuration15);16 17-- The validation that must happen BEFORE this table's watermark18-- generator sees a record. One future-dated occurred_at advances19-- the watermark past all real data and drops everything after it.20--21-- reject when occurred_at > ingested_at + <small tolerance>22--23-- Expressed upstream, in the parsing stage, where a rejection is a24-- routed record rather than a job-wide catastrophe.The interval is the entire correctness-versus-timeliness trade, written as one number. It should come from the measured lateness distribution of this specific source, and it should be revisited whenever that source's producers change.
Watermark declaration syntax, idleness configuration and whether allowed lateness is a separate knob from the watermark delay all vary by engine and have changed across versions. Verify the exact form and the defaults in the documentation for your version — particularly the default idleness behaviour, which decides whether a quiet partition stalls your job or silently makes its data late.
Choosing the delay
There is no correct watermark delay, only a defensible one, and defending it requires two things you may not have: the measured lateness distribution of the source, and a statement of what the consumer needs. Without both, the number is a guess that will be tuned reactively during incidents.
The options below are the real positions, and note that the last is a combination rather than a compromise. A modest delay plus a side output plus a periodic batch correction covers a long tail without holding windows open for it — which is almost always better than any single delay, because it decouples "how long we wait" from "how much we eventually capture".
The one genuinely wrong answer is to widen the delay until the drop counter reaches zero. That optimises one metric by degrading state size, restore time and time-to-answer simultaneously, and it eventually produces a job that cannot be restarted within an acceptable window — trading a small, measured, recoverable loss for a large, unmeasured, operational one (Streaming State).
What does the lateness distribution actually look like, and what does the consumer need — a fast number, a complete one, or both eventually?
when The source genuinely emits in event-time order per partition, such as a CDC stream reading a database log (CDC Ordering and Transaction Boundaries).
cost Any disorder at all, including from a producer retry or a partition change, becomes a dropped record. Safe only where ordering is a real property of the source, not an observation.
when Server-side producers on synchronised clocks, low transport delay, and freshness matters more than the tail.
cost The tail is dropped. Acceptable only with a drop counter and a side output, so the loss is measured rather than assumed (Late Events).
when A mixed producer population whose distribution you have plotted and whose tail you can afford to hold in state.
cost Every window is delayed by that amount and held open for it, multiplying state by the delay divided by the window size (Streaming State).
when Almost never in streaming. Occasionally right for a source with genuine multi-hour buffering, such as an intermittently-connected device fleet.
cost Enormous state, long restores, and an answer so delayed that a batch job would have produced it sooner and more cheaply (Batch vs Streaming Ingestion).
when The tail is long but the common case is fast — which describes most real platforms.
cost Three mechanisms to build and own instead of one parameter to set, and a second implementation of the metric that must agree with the first (Batch and Streaming Unification).
How to build it
Most important first.
- Derive the out-of-orderness allowance from the measured lateness distribution per source, not from a default. It is the same number that sizes your window state, so it deserves the same scrutiny (Percentiles: Which One, and How Many Users Is That?).
- Validate event times before they reach the watermark generator. Reject anything more than a small tolerance ahead of ingestion time, because a single future timestamp advances the watermark past all real data and causes mass dropping (Ingestion Time).
- Configure idleness handling explicitly. A partition that produces nothing must be marked idle so it stops holding the minimum back, and that behaviour must be a deliberate choice — because marking it idle also means its eventual records will be late (Consumer Groups and the Parallelism Ceiling).
- Generate watermarks per source, not globally, when sources have different lateness profiles. A single strategy across a web stream and a mobile fleet is wrong for at least one of them (Event Time).
- Alert on watermark lag — the gap between the watermark and wall clock — as a first-class signal. It is the only thing that distinguishes "no data is arriving" from "data is arriving and event time has stopped advancing" (Pipeline Observability).
- Keep the watermark delay and the allowed lateness as separate decisions where the engine allows it: the first controls when results first appear, the second controls how long they can still be corrected (Late Events).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Monotonicity: the watermark never decreases. Anything built on it can assume progress is one-directional.
- It guarantees that the system *believes* no more data before *T* will arrive. It does not guarantee that none will — that is what late events are, and the guarantee is deliberately a belief rather than a fact (Late Events).
- For a multi-input operator, the emitted watermark is the minimum of its inputs, which guarantees a downstream operator never sees a claim stronger than its weakest upstream supports.
- What is explicitly not guaranteed: that the watermark advances at all; that it reflects wall-clock time; that two jobs over the same stream compute the same watermark; or that it is correct — it is an estimate whose quality depends entirely on the timestamps in the data.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Alert on watermark lag exceeding a threshold, per job. This is the check that catches the stall, and it is the one signal in the module that cannot be derived from anything else (Freshness Monitoring).
- Count records classified as late by the watermark, per source. A sudden jump usually means a future-dated record moved the watermark, not that the world changed (Distribution Tests).
- Both miss a watermark that is systematically slightly too aggressive: a small constant fraction of records is dropped every window, the counters are non-zero but stable, and it reads as normal operation. Only a reconciliation against a batch recomputation quantifies it (Reconciliation).
- The watermark delay is a direct, explicit subtraction from freshness. Every unit of out-of-orderness allowance is a unit of delay before any window can emit, applied uniformly to every window.
- This is the module's central trade made numeric: a larger delay means more records included before first emission and a later first emission. The pair moves together and no configuration separates them.
- The stall case is the freshness failure that surprises people, because it is not a slowdown but a stop: the watermark freezes, no window emits, output ceases entirely, and consumer lag reads zero throughout because the job is fully caught up on offsets (The Backlog Arithmetic: Four Levers and a Drain Time).
- Changing the watermark delay changes the completeness of every result the job produces, with no schema change and no marker in the output. It is a metric redefinition disguised as a tuning parameter (Semantic Changes).
- Adding a new source with a different lateness profile to an existing job changes the effective watermark for everything, because the minimum across inputs now includes the slower one. A new input silently reshapes the freshness of the existing outputs (Data Contracts).
- Moving watermark generation from the source operator to a later stage changes what "out of order" means, because records have been shuffled in between. Where the watermark is generated is part of the job's definition, not an implementation detail.
- A replay regenerates watermarks from the replayed event times, so the same records produce the same watermark progression and the same window emissions. This is why event-time watermarking makes replay reproducible and wall-clock triggering does not (Deterministic Replay: Making the Schedule Reproducible).
- Recovering from a poisoned watermark — one advanced by a future-dated record — usually requires restarting the job with fresh watermark state after purging the offending record, because the watermark cannot be moved backwards by design.
- The records dropped while the watermark was poisoned are gone from the stream's output and must be recovered from raw storage by a batch recomputation for the affected range (Reprocessing vs Retrying).
What can go wrong
- An idle partition holding the minimum watermark back, so the whole job stops emitting while every other signal looks healthy. The most confusing incident in this module.
- A single future-dated record advancing the watermark past all real data, after which genuine records are classified as late and dropped en masse.
- A watermark delay chosen from a small sample, so a fat-tailed source loses a growing share of records as its fleet grows (Percentiles: Which One, and How Many Users Is That?).
- Idleness detection configured to keep the watermark moving, which fixes the stall and quietly guarantees that the idle source's eventual records are late (Late Events).
- The mitigation failing: widening the delay to stop dropping records, which multiplies the number of open windows and turns a correctness problem into a state and restore-time problem (Streaming State).
- "The watermark tells you the data is complete up to that point." It tells you the system *believes* so. It is an estimate derived from observed timestamps, and every late event is a counter-example to it (Late Events).
- "The watermark is behind, so we are lagging." It may be, or event time may have stopped advancing because the source went quiet. Watermark lag and consumer lag are different signals with different causes and different fixes (The Backlog Arithmetic: Four Levers and a Drain Time).
- "A watermark advances with time." It advances with *event time*, derived from records. Nothing about the passage of wall-clock time moves it, which is why an idle stream freezes it completely (Processing Time).
- "One bad record cannot do much damage." A single future-dated timestamp advances the watermark past all real data and causes every subsequent genuine record to be dropped as late. It is the highest-leverage single bad record in the module (Event Time).
Operating it
- Watermark lag:
wall_clock − watermark, per job and per source. The single most important streaming metric that most platforms do not have. - The watermark per input partition, not just the job minimum, because the minimum tells you there is a problem and the per-partition view tells you which one.
- Records dropped as late, per source, correlated with watermark movement — a spike in both at the same instant is the future-timestamp signature (Debugging a Data Incident).
- Time between window end and window emission, which is the watermark delay as consumers actually experience it rather than as configured (Freshness Monitoring).
- At 10x volume, watermark behaviour is unchanged. It is a function of timestamps, not of throughput.
- At 10x partitions, the chance that at least one is idle or lagging rises accordingly, so the minimum-across-inputs rule makes stalls more likely as the topic grows. Idleness handling stops being optional (Topics and Partitions).
- At 100x sources, the lateness distribution becomes so heterogeneous that a single watermark strategy cannot be right, and the architecture usually splits into per-source jobs with per-source policies feeding a common downstream model (Model Layering).
- The watermark itself is a scalar and costs nothing. Its *delay* costs state, because it is the delay that decides how many windows are open at once (Streaming State).
- Idleness detection costs a timer per partition and buys liveness. It is one of the few things in this module that is nearly free and materially improves operability.
- The expensive failure is the stall, whose cost is not compute but a complete cessation of output that may go unnoticed until a consumer complains (Data Incidents).
- A larger out-of-orderness allowance buys completeness at first emission and costs latency for every window plus state for every window held open. The two cannot be separated.
- Idleness detection buys liveness — windows continue to emit when a partition goes quiet — and costs correctness for that partition, because its eventual records will now be late by construction.
- Generating watermarks per source buys an appropriate policy for each and costs a more complex job with several watermark strategies to reason about, and results whose completeness differs by source in ways consumers must be told about.
Watermark lab — one late event
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
| Window | Events | Sum | Fired |
|---|---|---|---|
| 10:00–10:05 | a b c | 30sim | 10:05 |
| 10:05–10:10 | d e | 20sim | open |
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe idea of a progress estimate over event time, monotonic and derived from observed data, is common to every engine that supports event-time processing; the minimum-across-inputs rule follows from the semantics rather than from any implementation.
- ENGINE-SPECIFICFlink generates watermarks per source with a pluggable strategy and supports explicit idleness detection, advancing per record; Spark Structured Streaming derives a single watermark from a declared column and advances it per micro-batch, taking the minimum across inputs; Kafka Streams uses stream time per task rather than a separately-configured watermark. The same intent therefore has different failure behaviour on each.
- SIMULATEDThe timeline here comes from
src/de/sim/stream.ts, where the watermark is the maximum event time seen minus the allowed lateness and a window closes once it passes the window end.scripts/de-sim.test.tsasserts monotonicity and that a late record does not advance it. The model deliberately collapses watermark delay and allowed lateness into one knob where real engines expose two.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why a progress claim over event time is the best available answer rather than a workaround — there is no mechanism by which a receiver can distinguish a slow sender from a silent one, so completeness must be estimated rather than determined.