Windows
A window is a rule that turns an infinite stream into a set of finite groups you are allowed to aggregate — and choosing the rule decides your state size, your latency and what questions you can answer.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
How do I compute an aggregate over an input that never ends, and what does the boundary I choose commit me to?
Everyone who asks for "per hour", "in the last five minutes", "during the session" or "rate over a rolling window". Each of those phrases is a different window definition with a different cost, and the person asking has almost never distinguished between them.
One key-and-window pair: the unit of both the output row and the state entry. Getting this wrong is the classic grain error of streaming — a sliding window emits the same event's contribution in several rows, so summing across windows double counts (Grain: What Does One Row Represent?).
Aggregate as records arrive and emit the running total. There is no boundary to choose, no state to expire, and the number is always current.
The running total only grows, so it answers "how much ever" and no consumer wanted that. The moment somebody asks "per hour" a boundary is needed and there is none (Fact Tables).
- The running total only grows, so it answers "how much ever" and no consumer wanted that. The moment somebody asks "per hour" a boundary is needed and there is none (Fact Tables).
- The aggregate is keyed by an unbounded key space and nothing ever expires, so state grows for as long as the job runs (Streaming State).
- A restart replays and the running total jumps, because the sink appended the recomputed value alongside the old one instead of replacing it (Duplicate Rows).
- Someone adds a bucket by truncating the arrival timestamp, which works until a client buffers and its events all land in the wrong bucket (Event Time).
- A sliding window is introduced for a smoother chart, and downstream someone sums across windows to get a daily total — which counts every event as many times as it appeared in overlapping windows (Sliding Windows).
What is actually happening
- A window assigner is a pure function from an event's event time (and, for session windows, its key and neighbours) to a set of window boundaries. Tumbling assigners return exactly one window; sliding assigners return size ÷ slide of them; session assigners return one whose boundaries depend on the data (Event Time).
- A trigger decides when a window may be emitted. The default is "when the watermark passes the window end", which is what makes emission a consequence of event-time progress rather than of the clock (Watermarks).
- State is held per key and per open window, so the number of live state entries is keys × open windows. Everything expensive about windowing follows from that product (Streaming State).
- A window is purged once no further update is possible — its end plus the allowed lateness. Until then it can be updated and re-emitted; afterwards it cannot be updated at all (Late Events).
- Windows are computed independently per key. Two keys' windows for the same period are separate state entries and separate output rows, which is why key cardinality multiplies everything.
- A global window with a custom trigger is the escape hatch: no time boundary at all, emission driven by a count, a punctuation record or an external signal. It is the right answer for "every hundredth event" and the wrong answer for anything time-shaped.
A window is a rule, not a clock
runStream in src/de/sim/stream.ts with the default five-minute tumbling configuration; scripts/de-sim.test.ts asserts that every event lands in exactly one window and that assignment ignores arrival time entirely.The useful definition is mechanical: a window assigner is a function that takes an event and returns the set of window boundaries it belongs to. Tumbling returns one. Sliding returns several. Session returns one whose edges are decided by the neighbouring events for that key. Nothing else about windowing changes between the three.
The timeline below shows the simplest case — five-minute tumbling windows — and the property to read off it is that placement depends only on the event's own timestamp. Event d arrived after event c and lands in a later window because it *happened* later, not because it arrived later; if their arrival order were reversed the placement would be identical.
That is the invariant worth carrying into every other lesson in this module: assignment is by event time, always. Arrival time never decides where a record lands. It decides only whether the window it belongs to is still open when the record shows up, which is a different question with a different mechanism (Late Events).
| Event | Happened | Arrived | Lands in |
|---|---|---|---|
| a | 10:01 | 10:01 | 10:00–10:05 |
| b | 10:02 | 10:02 | 10:00–10:05 |
| c | 10:04 | 10:04 | 10:00–10:05 The last on-time member of W1. The window still cannot emit — nothing has told it that 10:05 has passed. |
| d | 10:06 | 10:06 | 10:05–10:10 This record is what closes W1: it advances the watermark past 10:05. A window closes because later data arrived, not because time passed. |
| e | 10:08 | 10:08 | 10:05–10:10 |
Watermark after the last arrival. Note the mechanism in event d: the boundary is crossed by the arrival of an event with a later event time, which is why an idle stream leaves the last window open forever (Watermarks).
Three families, and the question each one answers
Consumers do not ask for windows; they ask questions in English, and the translation is where the design happens. "Revenue per hour" is tumbling. "Failed logins in the last five minutes, checked every minute" is sliding. "How long was the visit" is session. "Every thousandth event" is a global window with a count trigger and has nothing to do with time at all.
Getting the translation wrong is not a performance problem, it is a wrong answer. A sliding window used where tumbling was meant produces overlapping results that will eventually be summed; a tumbling window used where sliding was meant produces a rate that jumps at boundaries and misses a burst straddling one.
The costs below are the ones that matter operationally, and they are dominated by the state term. Note that the session row has no bound at all in the general case — its cost is decided by the data rather than by the configuration, which is what makes it the hardest of the three to operate (Session Windows).
What shape is the consumer's sentence — a fixed period, a rolling view, an activity burst, or a count?
when "Per hour", "per day", "each five-minute bucket". Fixed, contiguous, non-overlapping periods that will be compared with each other.
cost Lowest state of the three: one open window per key, plus however many the lateness allowance holds. Results are additive across windows (Tumbling Windows).
when "In the last ten minutes, updated every minute". A rolling view that must respond to a burst without waiting for a boundary.
cost State and output multiplied by size ÷ slide. Results are not additive, and summing them double counts by exactly that factor (Sliding Windows).
when "During a visit", "in one burst of activity". Boundaries defined by inactivity rather than by the clock.
cost Unbounded in principle — a key that never goes idle holds one growing entry forever. Windows also merge retroactively when a late event bridges a gap (Session Windows).
when "Every hundredth event", "when a control record arrives". The boundary is a count or a signal, not a time.
cost You now own the trigger logic, including what happens when the count is never reached — which is a stall with no time-based escape.
when The consumer wants a rolling number and the data volume makes sliding-window state uncomfortable.
cost An extra hop and a small loss of granularity, in exchange for a fraction of the state. This is the option most often overlooked and most often correct (The Metrics Layer).
What a window emits, and when
The lifecycle is five steps and each one has a distinct failure. Reading them in order is the fastest way to diagnose a windowed job, because the symptom maps cleanly onto the stage: no output at all is a trigger problem, wrong placement is an assignment problem, growing memory is a purge problem, and duplicated output is a sink problem.
The step that is most often misunderstood is the separation between trigger and purge. A window can emit and continue to exist: it emits when the watermark passes its end, and it is destroyed only when the lateness allowance expires. Between those two moments, a late record still updates it and causes another emission (Late Events).
That interval is why an updatable sink is a precondition rather than a nicety. Every re-emission carries the same window key with a new value, so an appending sink accumulates versions of the same period and a consumer summing the table gets a number that grows with the number of corrections rather than with the data.
- 1Assign
Maps the event's event time to one or more window boundaries.
guarantees Deterministic given event time; identical on every replay.
fails by Being fed arrival time, so a buffered client's events land in the wrong period and a replay produces different windows entirely (Event Time).
- 2Accumulate
Updates the per-key, per-window aggregate in state.
guarantees Durable across restarts if state and offsets are checkpointed together (Checkpointing).
fails by Storing the contributing records rather than the aggregate, so state grows with volume instead of with keys.
- 3Trigger
Decides the window may emit — by default when the watermark passes the window end.
guarantees The emitted value includes everything assigned so far. Nothing about what has not arrived.
fails by A stalled watermark: no trigger fires, output stops, and lag stays at zero throughout (Watermarks).
- 4Emit
Writes
(window_start, window_end, key, aggregate)to the sink.guarantees Whatever the sink promises. An upsert keyed on window and group is idempotent; an append is not (Upserts and Merges).
fails by Appending, so replays and late corrections accumulate as extra rows for the same period (Duplicate Rows).
- 5Update on late arrival
Accepts records arriving before the allowance expires and re-emits.
guarantees The published value converges towards the truth for as long as the window is held open.
fails by Downstream consumers that snapshot the first value they see and never observe the correction.
- 6Purge
Drops the window's state once no further update is possible.
guarantees State is bounded by keys × (allowance ÷ size + 1) × value size.
fails by Never purging — a global window, an unbounded key space, or a session that never goes idle (Streaming State).
Trigger and purge are separate events separated by the allowed lateness. Everything confusing about windowed output happens in the gap between them.
How to build it
Most important first.
- Start from the consumer's sentence and translate it precisely. "Per hour" is tumbling; "in the last five minutes, updated every minute" is sliding; "during a visit" is session; "since the account was created" is not a window at all and needs a different design (Who Actually Consumes This Data).
- Choose the smallest window that answers the question. Larger windows hold more state and delay the answer, and a consumer who says "hourly" often means "a number that updates hourly", which a five-minute window plus a downstream rollup serves better (The Metrics Layer).
- Compute the state estimate before writing the job: keys × (allowance ÷ window size + 1) × value size for tumbling, multiplied by the overlap factor for sliding. If it is uncomfortable, change the window, not the memory limit.
- Prefer tumbling plus a downstream rolling aggregate over a sliding window when both would work. The rolling sum over emitted tumbling results costs a fraction of the state and gives the same chart (Sliding Windows).
- Emit the window boundaries as explicit columns in the output —
window_startandwindow_end— so a consumer can tell the grain of a row without reading the job. An aggregate whose period is implicit will eventually be summed by someone (Grain: What Does One Row Represent?). - Decide the lateness policy at the same time as the window, because they are one decision. A window definition without a stated lateness allowance is half a specification (Late Events).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Assignment is deterministic given event time, so a replay places every record in exactly the same windows. This is the property that makes windowed results reproducible (Deterministic Replay: Making the Schedule Reproducible).
- Emission guarantees that the result includes every record that had arrived and been assigned when the trigger fired. It says nothing about records that had not arrived.
- Tumbling windows guarantee that each event contributes to exactly one window, so results are additive across windows. Sliding windows guarantee the opposite: they are not additive, and summing them double counts by the overlap factor.
- What is explicitly not guaranteed: that a window's result is final when first emitted; that all keys' windows for a period emit at the same moment; or that a window ever emits at all, since a stalled watermark stalls emission indefinitely (Watermarks).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Assert that the sum of a measure across tumbling windows for a period equals the same measure computed directly from raw events for that period. It catches misassignment, dropped late records and boundary errors in one query (Reconciliation).
- For sliding windows, assert instead on a *single* window against a direct recomputation, because summing across overlapping windows is expected to over-count and would make the check meaningless.
- Both miss the case where the window definition itself is wrong — an hourly window in the wrong timezone reconciles perfectly against a recomputation that uses the same wrong timezone (Semantic Changes).
- A windowed result cannot be fresher than the window end plus whatever the trigger waits for. Asking for a fresher number than the window allows is asking for a different window, not for faster hardware.
- A processing-time trigger over an event-time window gives early provisional results and a late final one, which is usually the right shape: consumers get something immediately and something trustworthy eventually (Processing Time).
- The most-recent window is always incomplete and always looks like a drop on a chart. Either exclude it from the visualisation or mark it, because every viewer will otherwise read a partial period as a decline (Stale Dashboards).
- Changing a window size or slide invalidates existing state and changes the meaning of every historical row. This is one of the few changes in the domain that is simultaneously a state migration, a metric redefinition and a schema change to the output grain (Grain: What Does One Row Represent?).
- The safe path is a new output dataset at the new definition, run in parallel, with an explicit switchover date recorded — not an in-place change to a job whose old output stays in the same table (Model Layering).
- Adding
window_startandwindow_endto an output that lacked them is a compatible addition and worth doing before anything else, because it makes every subsequent change discussable (Dataset Documentation).
- Replaying a range through a windowed job reproduces the same windows exactly, so a corrected job produces directly comparable output — provided assignment is by event time (Replay from the Log).
- The publish is the risk, not the computation: a replay re-emits every window in the range and the sink must upsert on
(window_start, key)or the period will hold two answers (Upserts and Merges). - A backfill through a windowed job writes into historical windows rather than into today, which is what makes a bounded correction possible at all (Planning a Backfill).
What can go wrong
- State multiplied unexpectedly by an overlap factor or by a wide lateness allowance, so a change to one line of window configuration causes a memory incident weeks later (Streaming State).
- A stalled watermark, so no window ever triggers and output stops while every operational metric looks healthy (Watermarks).
- Sliding-window results summed downstream, producing totals inflated by exactly the overlap factor and looking entirely plausible (Sliding Windows).
- A session window that never closes because a key never goes idle, holding one unbounded state entry (Session Windows).
- The mitigation failing: emitting early provisional results to improve freshness, and a downstream consumer snapshotting the first value it sees, so the correction is computed, emitted and ignored.
- "A window is a time range." A window is a *rule for assigning events to groups*. Session windows have boundaries derived from the data, and a global window with a count trigger has no time in it at all.
- "Smaller windows mean fresher data." Only up to the trigger. If the trigger waits for the watermark plus an hour of allowed lateness, a one-minute window is no fresher than a ten-minute one (Watermarks).
- "I can sum the windows to get a daily total." True for tumbling, false for sliding, and meaningless for sessions. The window type decides whether the output is additive, and nothing in the schema records which type produced it (Grain: What Does One Row Represent?).
- "The window closed, so the data is complete." The window closed because the watermark passed its end, and the watermark is an estimate. Completeness was assumed, not verified (Late Events).
Operating it
- Open window count per operator, alongside key count. The ratio between them is the overlap-and-lateness multiplier, made visible instead of inferred from configuration.
- Windows emitted per interval and windows re-emitted per interval, which distinguishes normal progress from a correction storm.
- Time between a window's end and its first emission, which is the number consumers actually experience as freshness (Freshness Monitoring).
- The most-recent window's completeness, exposed as a flag on the output row, so downstream tools can exclude partial periods rather than plotting them (The Data Quality Dashboard).
- At 10x volume with the same keys, window state is unchanged — the aggregate absorbs the extra records without growing.
- At 10x keys, everything scales linearly: state, output rows and sink writes. This is the axis that matters and it is not the one people plan for.
- At 100x, skew dominates: one key's windows hold most of the state on one instance, and the standard remedies are a two-stage aggregation or a salted pre-aggregation followed by a merge (Salting a Skewed Key).
- State is keys × open windows × value size, and each of those three factors is chosen by a different decision: the key, the window definition plus lateness, and what you store per window.
- Output volume scales with keys × windows emitted, which for a sliding window is multiplied by the overlap factor — a sliding window emits far more rows than a tumbling one over the same data (Sliding Windows).
- Sink write cost follows output volume and gets worse with corrections, because each re-emission is a rewrite rather than an append. On a lake table format that can mean rewriting files (Open Table Formats).
- A smaller window buys freshness and a smaller state per window, and costs more output rows and more sink writes for the same period.
- A larger lateness allowance buys completeness and costs state and time-to-final directly. There is no setting that improves both, and pretending otherwise is how streaming jobs become unoperable.
- Sliding windows buy a smooth, always-current view and cost the overlap factor in state, output volume and the permanent risk that somebody sums them (Sliding Windows).
Window lab — tumbling, sliding, session
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
| id | happened | arrived | counted |
|---|---|---|---|
| a | 10:01 | 10:01 | yes |
| b | 10:02 | 10:02 | yes |
| c | 10:04 | 10:04 | yes |
| d | 10:06 | 10:06 | yes |
| e | 10:08 | 10:08 | yes |
| late | 10:00 | 10:07 | dropped |
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALAssigner, trigger, state-per-key-per-window and purge are the four parts of windowing in every engine, and the additivity properties of each window family follow from set theory rather than from any implementation.
- ENGINE-SPECIFICFlink exposes assigner, trigger, evictor and allowed lateness as four separate configurations; Spark Structured Streaming exposes a window function plus a watermark and folds triggering into the output mode and processing trigger; Kafka Streams expresses suppression as a separate operator. Expressing "emit once, when complete" therefore looks completely different in each.
- SIMULATEDThe timeline here is produced by
src/de/sim/stream.ts, whose assignment rules are asserted inscripts/de-sim.test.ts. Clock labels are teaching positions rather than measured latencies, and the model exposes one lateness knob where engines expose two.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why the boundary has to be decided locally with incomplete information, rather than by asking whether every producer has finished sending for a period.