Sliding Windows
Overlapping windows of fixed size, advancing by a smaller step. Every event lands in size ÷ slide of them, which is exactly the factor by which state, output and the risk of double counting all multiply.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
When is a rolling, overlapping view worth multiplying my state and output volume by the overlap factor?
Alerting and monitoring, mostly. "More than five failed logins in any ten-minute period" cannot be answered by tumbling windows, because a burst straddling a boundary is split in two and neither half crosses the threshold. A rolling view sees it (Quality Alerting).
One row per (window_start, key), exactly as with tumbling — and this is the trap. The output *looks* identical to a tumbling aggregate, so the fact that a single event contributes to several rows is invisible in the data and destroys any sum across windows (Grain: What Does One Row Represent?).
Ask for a ten-minute window that updates every minute, because that is what the alert needs and it is one configuration change from the tumbling version.
State multiplies by ten, because every key now has ten windows open at once instead of one. Checkpoints slow, restores lengthen, and the connection to the one-line change is not obvious a month later (Streaming State).
- State multiplies by ten, because every key now has ten windows open at once instead of one. Checkpoints slow, restores lengthen, and the connection to the one-line change is not obvious a month later (Streaming State).
- Output volume multiplies by ten as well, so the sink absorbs ten times the writes and a lake table gains ten times the files (File Size and the Small-Files Problem).
- Somebody builds a daily total by summing the window rows, and it comes out roughly ten times too large — a number so wrong that it is usually caught, unlike the many that are only slightly wrong (Two Dashboards, Two Numbers).
- The alert fires ten times for one burst, once per overlapping window that contains it, and the on-call engineer learns to ignore it (Alert Fatigue: The Page Nobody Reads).
- A late record updates every window that contains it, so one straggler causes ten re-emissions and ten sink writes (Late Events).
What is actually happening
- The assigner returns every window whose half-open range contains the event time. With size *S* and slide *L*, that is ⌈S ÷ L⌉ windows — the overlap factor — and it applies to every single event (Windows).
- When the slide equals the size, the overlap factor is one and a sliding window degenerates into a tumbling window. Tumbling is therefore the special case, not a separate mechanism.
- Live state is keys × overlap factor × (lateness allowance ÷ slide + 1). The overlap factor multiplies the largest cost term in a streaming job, and it is chosen by a ratio in the window definition rather than by anything about the data (Streaming State).
- Output is not additive. The same event appears in several windows by design, so summing a measure across windows over-counts by the overlap factor. There is no schema-level marker distinguishing this output from tumbling output.
- Emission still happens per window when the watermark passes its end, so a sliding job emits at the slide interval rather than at the size interval — the freshness is set by the slide, and the coverage by the size.
- A common alternative computes the same rolling view from tumbling windows plus a downstream running aggregate over the last *n* emitted buckets. It is approximate at the boundaries, uses a fraction of the state, and is often the better trade (The Metrics Layer).
Every event counted more than once, on purpose
runStream(events, { kind: 'sliding', sizeMin: 10, slideMin: 5 }) in src/de/sim/stream.ts; the test file asserts that one event yields two windows and an emitted sum of twice its value, which is the non-additivity property expressed as an assertion.A ten-minute window sliding by five minutes means a new window starts every five minutes and each one covers ten. Consequently every instant is inside exactly two windows, and every event contributes to exactly two output rows. That is the mechanism, and it is not a side effect — it is what the consumer asked for when they said "rolling".
The timeline makes it concrete. Three events, four windows, six placements. Nothing has gone wrong; the same value has legitimately been counted twice because it belongs to two overlapping periods. The failure only occurs downstream, when somebody adds those rows together and gets twice the truth.
The overlap factor generalises: size divided by slide, rounded up. Ten over five is two. Sixty over one is sixty. That number multiplies state, output rows, sink writes and late-record correction work simultaneously, which makes it the single highest-leverage figure in the job and the one least likely to appear in a design review.
| Event | Happened | Arrived | Lands in |
|---|---|---|---|
| x | 10:02 | 10:02 | 09:55–10:05 and 10:00–10:10 Two placements. Sum both windows and this event has been counted twice. |
| y | 10:07 | 10:07 | 10:00–10:10 and 10:05–10:15 Two placements again — coverage is uniform, which is the property that makes a boundary-straddling burst visible. |
| z | 10:12 | 10:12 | 10:05–10:15 and 10:10–10:20 Three events, six placements. The overlap factor is size ÷ slide = 10 ÷ 5 = 2. |
Three events and six contributions. Nothing here is wrong, and nothing in the output schema distinguishes these rows from a tumbling aggregate that would be safe to sum.
The overlap factor is the whole cost
A sliding window has one number that decides everything about its operability, and it is not the volume, the key count or the throughput. It is size divided by slide, and it appears as a multiplier in four separate places at once.
This matters because the ratio is chosen casually. A consumer asks for "a ten-minute view, refreshed every ten seconds" and nobody notices that the request has an overlap factor of sixty — sixty times the state, sixty times the output rows, sixty times the sink writes, and sixty re-emissions for every late record.
The escape is almost always the same: keep the size, coarsen the slide, and if the consumer genuinely needs a fast refresh, compute a coarse tumbling aggregate and maintain the rolling sum downstream over the last *n* buckets. That gives an approximation at the boundaries and reduces the multiplier to one (Tumbling Windows).
The dominant term. Every key holds one aggregate per open window, so the ratio in the window definition is a direct multiplier on the job's memory and checkpoint size.
A new window emits every slide interval rather than every size interval, so both the row count and the write rate scale with the overlap factor.
One straggler updates every window containing it, so each late record causes overlap-factor re-emissions and the same number of sink writes (Late Events).
More frequent emission means more, smaller files per partition, which moves cost onto the read side and forces a compaction schedule (File Compaction).
Each record is assigned to several windows instead of one. Real but small — the arithmetic is trivial compared with holding the results.
Deliberately near the bottom again: throughput adds records to existing aggregates without adding state, exactly as with tumbling windows.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights for a keyed sliding aggregate, given to establish an ordering rather than a magnitude. The ordering is the teaching: one ratio in the window definition dominates four separate cost lines, and record volume is not one of them.
When you actually want one
The case for a sliding window is specific and real: a threshold that must be evaluated over a period regardless of where that period starts. "Five failed logins in any ten minutes" is genuinely unanswerable with tumbling windows, because an attacker producing three logins at 10:04 and three at 10:06 crosses no bucket's threshold while obviously crossing the real one.
The case against it is that most requests phrased as "rolling" do not have that property. A chart that updates every minute does not need overlapping windows; it needs frequent tumbling windows and a rolling sum applied at read time, which is arithmetically almost the same picture for a fraction of the operational cost.
The distinction to test is whether the boundary matters *to the decision*. If crossing a threshold has consequences — locking an account, paging a human, refusing a transaction — then the exact placement of the window matters and sliding earns its cost. If the output is a line on a chart, it does not.
The job keeps ten windows open per key at all times, emits ten times as many rows as the tumbling equivalent, and re-emits ten windows for every late record. Every value is exact for its exact range.
The job keeps one window open per key, emits one row per key per minute, and a downstream model sums the last ten emitted rows to produce the rolling view.
The two produce the same series except at sub-minute granularity, where the tumbling version quantises the window edges to minute boundaries. State drops by the overlap factor and so do output rows and sink writes. The sliding version is worth its cost only when a threshold must be evaluated at an arbitrary boundary and the sub-minute placement changes the decision — which is true for an abuse detector and false for a dashboard.
Naming here is genuinely inconsistent between engines: what Flink SQL calls HOP (taking slide before size) some libraries call a sliding window, while Kafka Streams distinguishes hopping windows from a differently-defined sliding window. Confirm which construct your engine's "sliding" refers to and in which order it takes its arguments before trusting an overlap-factor estimate.
How to build it
Most important first.
- Justify the overlap factor explicitly. Size ÷ slide is a multiplier on state, output rows and sink writes simultaneously, and it should be a decision with a written reason rather than a pair of intervals that sounded right.
- Prefer tumbling plus a downstream rolling sum when the consumer will tolerate boundary granularity. A rolling sum over the last twelve five-minute buckets approximates a one-hour rolling window at a tenth of the state (Tumbling Windows).
- Mark the output as non-additive in the dataset documentation and, better, in the column names —
amount_in_windowreads differently fromamountand stops one class of misuse at the point of reading (Dataset Documentation). - Deduplicate alerts downstream on the underlying condition rather than on the window, so one burst produces one alert rather than one per overlapping window (Quality Alerting).
- Keep the value small. The overlap factor multiplies value size along with everything else, so a large per-window payload is the one design decision that compounds worst here (Projection Pushdown).
- Choose the slide from how quickly the consumer must react, and the size from the period the question is about. They are two independent requirements and conflating them is what produces accidental overlap factors of sixty (Windows).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Coverage: every point in time is covered by the same number of windows, so a burst is fully contained in at least one window regardless of where it starts. This is the property you are buying and tumbling windows cannot provide it.
- Determinism: assignment depends only on event time, so a replay reproduces the same set of window memberships exactly (Deterministic Replay: Making the Schedule Reproducible).
- Explicitly not additive: results must not be summed across windows. This is a guarantee stated in the negative and it is the most important sentence in the lesson.
- Also not guaranteed: that overlapping windows agree with each other about a period they share, since each was emitted at a different moment with a different set of arrived records.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Check a single window against a direct recomputation over the raw events for that exact range. Never check a sum across windows, because over-counting by the overlap factor is the expected behaviour and the check would be asserting the wrong thing (Reconciliation).
- Assert the expected number of windows each event landed in — it should equal the overlap factor — as a test over a small fixture. It catches a mis-specified slide, which otherwise silently changes every number by a constant factor.
- Both miss downstream misuse. Nothing in the pipeline can prevent an analyst from summing the output, which is why the naming and the documentation are part of the quality control rather than adjacent to it (Dataset Documentation).
- The slide sets the update rate. A ten-minute window sliding by one minute produces a new answer every minute, each covering the preceding ten — which is what "rolling" means to a consumer.
- The size sets the smoothing. A larger size means each answer is steadier and responds to a change more slowly; the pair therefore controls responsiveness and noise independently, which is the real reason to use this family (The Average Was Fine and Users Were Not).
- The most recent window is always partial, exactly as with tumbling — but here the partial window overlaps complete ones, so a chart that plots all of them shows a dip at the right-hand edge even though the underlying activity is flat (Stale Dashboards).
- Changing the size or the slide changes both the meaning of a row and the overlap factor, so it is simultaneously a metric redefinition and a capacity change. Historical rows remain valid under the old definition and are not comparable with the new ones (Semantic Changes).
- Migrating from sliding to tumbling plus a downstream rollup is a common and worthwhile change, and it alters the numbers at boundaries. Run both, quantify the difference, and let consumers see it before the cutover (Model Layering).
- Adding a column that records size and slide on every row is a compatible change and makes the output self-describing — worth doing early, because the alternative is a table whose meaning lives in a job definition somewhere (Dataset Documentation).
- A replay reproduces every overlapping window identically, so a corrected run is directly comparable — but it re-emits overlap-factor times as many rows as a tumbling job would for the same range (Replay from the Log).
- The sink must upsert on
(window_start, key). With overlapping windows an appending sink accumulates errors faster than anywhere else in the module, because each late record touches several windows (Upserts and Merges). - Bounded backfills work but are wider than they look: correcting a range of event times requires recomputing every window that overlaps that range, which extends by the window size on each side (Planning a Backfill).
What can go wrong
- An overlap factor chosen accidentally — a five-minute window sliding by five seconds is a factor of sixty — turning a reasonable job into an unoperable one with no error at any point (Streaming State).
- Sliding output summed downstream, producing a total inflated by exactly the overlap factor.
- Alert storms: one condition firing once per overlapping window, training the on-call to ignore the alert (Alert Fatigue: The Page Nobody Reads).
- Sink write amplification from re-emissions, which on a lake table format means repeated file rewrites for the same period (File Compaction).
- The mitigation failing: replacing sliding windows with tumbling plus a downstream rollup, and thereby missing exactly the boundary-straddling burst the sliding window existed to catch.
- "Sliding windows are more accurate." They are more *responsive*. Each individual window is exactly as accurate as a tumbling window of the same size; what changes is that there are more of them and they overlap.
- "I can sum sliding windows to get a total." Never. The overlap factor is exactly how many times each event has been counted, and there is nothing in the output schema to warn you (Grain: What Does One Row Represent?).
- "A one-minute slide gives one-minute freshness." It gives a new answer every minute, each still describing the whole window size. A ten-minute window sliding by one minute cannot tell you what happened in the last minute alone.
- "The engine optimises the overlap away." Some engines share partial aggregates across overlapping windows for decomposable measures, which reduces the constant. State, output rows and sink writes still grow with the overlap, and non-decomposable measures get no help at all.
Operating it
- Open windows per key, which should equal the overlap factor times the lateness multiplier. Publishing it as a metric makes an accidental slide change visible immediately (Streaming State).
- Output rows per interval compared with the tumbling equivalent, which quantifies what the overlap is actually costing in sink writes.
- Re-emission count per late record, which for a sliding job is naturally the overlap factor and is a useful sanity check on the configuration.
- Alert firings per underlying incident, which is the number that tells you whether the overlap is producing signal or noise (Alert Fatigue: The Page Nobody Reads).
- At 10x volume, sliding behaves like tumbling: the aggregates absorb more records without more state.
- At 10x keys, state and output scale by keys × overlap factor, so the multiplier that was tolerable at small key counts becomes the dominant cost.
- At 100x, sliding windows are usually the first thing replaced. The standard migration is a coarser tumbling window plus a downstream rolling aggregate, accepting boundary granularity in exchange for an order-of-magnitude smaller state (The Metrics Layer).
- The overlap factor multiplies state, output rows, sink writes and re-emission work all at once. It is the single most leveraged number in a streaming job's cost, and it is set by two intervals in a configuration.
- Alternatives that approximate the same view — tumbling plus a rolling downstream aggregate, or an incrementally-maintained running total with expiry — cost a fraction of the state and give up exact boundary behaviour.
- For decomposable aggregates the engine may maintain a smaller shared structure rather than one aggregate per overlapping window, which changes the constant but not the shape: more overlap is always more work (Query Optimizers).
- Sliding buys a burst that straddles a boundary being seen, and costs the overlap factor in state, output volume, sink writes and correction work.
- A smaller slide buys responsiveness and costs the overlap factor directly. Every halving of the slide doubles everything expensive about the job.
- Approximating with tumbling plus a rollup buys most of the benefit for a fraction of the cost, and gives up exactness at boundaries — which is fine for a chart and not fine for a threshold that has compliance consequences.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALOverlapping fixed-size windows and the size ÷ slide multiplier are engine-independent arithmetic; the same relationship holds for a rolling window in batch SQL, where it appears as a range frame over an ordered set rather than as retained state.
- ENGINE-SPECIFICFlink SQL calls this a
HOPand takes the slide before the size, which is the reverse of how most people say it aloud; Spark expresses it as a window with a slide duration argument; Kafka Streams calls a related but not identical construct a hopping window and reserves "sliding" for a differently-defined operator. Reading a job written in one dialect with another's assumptions produces a silently wrong overlap factor. - SIMULATEDThe timeline comes from
src/de/sim/stream.tswith size ten and slide five;scripts/de-sim.test.tsasserts that a single event produces two window placements and that the emitted sum is therefore twice the event's value — the double counting, stated as a test.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the alerting discipline that keeps an overlapping-window detector from producing one page per overlapping window for a single underlying condition.