Session Windows
Windows whose boundaries the data draws: a session runs until a key goes quiet for longer than a gap. Per-key, data-dependent, mergeable — and the only window family with no upper bound on its own size.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
How do I group events into bursts of activity when the boundaries are defined by silence rather than by the clock?
Product analytics and anything that reasons about a visit. "How long was the session", "how many pages before checkout", "did the user abandon" — none of these have a clock-shaped answer, because a visit is defined by when someone stopped rather than by an hour boundary (Event vs Snapshot Modeling).
One row per (key, session), where the session's boundaries were computed from the events themselves. Two rows for the same key are two distinct visits separated by a gap; there is no fixed period a row corresponds to, which makes this output impossible to align with any calendar aggregate without an explicit choice about which period a straddling session belongs to.
Emit a session when a key has been quiet for the gap duration. Keep the open session in a dictionary keyed by user, extend it on every event, and flush it on a timer.
A key never goes quiet — a bot, a stuck client polling every second, a device with a reconnect loop — so its session never closes and one state entry grows for as long as the job runs (Streaming State).
- A key never goes quiet — a bot, a stuck client polling every second, a device with a reconnect loop — so its session never closes and one state entry grows for as long as the job runs (Streaming State).
- A late event arrives that falls in the gap between two sessions that have already been emitted, so those two sessions should have been one. There is nothing left to merge them with (Late Events).
- The gap is fixed at thirty minutes because that is the convention, and the product's actual usage pattern makes half of all "sessions" artefacts of the gap rather than of user behaviour (Semantic Changes).
- Session lengths are summed to get "total time on site" and the figure is meaningless, because the last event of a session has no duration and the gap is included or excluded inconsistently.
- Rescaling moves keys between instances mid-session, and an instance that never saw the earlier events emits a session that starts in the middle of a visit (Consumer Groups and the Parallelism Ceiling).
What is actually happening
- The assigner is per key and data-dependent: each event provisionally opens a window of
[event_time, event_time + gap), and any two windows for the same key that touch or overlap are merged. A run of events closer together than the gap therefore collapses into one session (Windows). - The consequence is that a session's end is
last_event_time + gap, not the time of the last event. The gap is part of the window even though no activity happened in it, which is why session durations computed from the window boundaries are systematically longer than the activity they describe. - A session closes when the watermark passes
last_event_time + gap— that is, when event time has advanced far enough to prove the key has been quiet for a gap's worth of event time. Wall clock is irrelevant (Watermarks). - Merging is the property that makes session windows structurally harder than the other families. A late event landing between two already-emitted sessions requires retracting both and emitting one — which most pipelines cannot express, so most drop it instead (Late Events).
- State per key is one open session holding either an aggregate or, if the consumer needs the sequence, every event in it. The second form is where session state becomes genuinely unbounded, because it grows with activity rather than with keys (Streaming State).
- Sessions are per key by definition, so the key choice is the definition. Keying by user, by device, by cookie or by account produces four different session datasets, all of which will be called "sessions" by somebody.
The window the data draws for itself
src/de/sim/stream.ts, where a session ends at last_event_time + gap and consecutive events merge when their separation is strictly less than the gap; scripts/de-sim.test.ts asserts both the merge and the split around that threshold, and that sessions are computed per key.Every other window family has boundaries you can compute before seeing a single event. Session boundaries cannot be computed at all until the events arrive, because the rule is about silence: a session continues while the key keeps producing events closer together than the gap, and ends when it does not.
That makes the emission condition unusual. A session closes when event time has advanced past last_event + gap — the system has to observe evidence that nothing happened, and the only evidence available is a later event from somewhere. This is why an idle stream leaves sessions open indefinitely and why the watermark matters more here than anywhere else (Watermarks).
The diagram below traces one user with a five-minute gap. Note that session A's window extends to 10:09 — five minutes past its last event at 10:04 — and that the emitted session_end therefore includes silence. Any duration computed from the window boundaries is longer than the activity by exactly one gap, and both definitions are in use in the wild.
user u1, gap = 5 minutes
events: 10:00 10:03 10:04 10:14 10:16
| | | | |
+---- session A ----+ ..... quiet ..... +-- session B --+
10:00 10:04 | 10:14 10:16
v
window ends 10:09 (last_event + gap)
emitted when the watermark passes 10:09
gap between 10:04 and 10:14 is 10 minutes >= 5 -> two sessions
gap between 10:00 and 10:03 is 3 minutes < 5 -> same session
what one row holds:
session_start 10:00 first event time
last_event 10:04 last event time
session_end 10:09 last_event + gap <- includes silence
events 3
duration? 4 min (last_event - session_start)
9 min (session_end - session_start) <- differs by the gap
the failure this shape invites:
a late event at 10:08 arrives AFTER both sessions were emitted.
10:08 is within the gap of BOTH -> A and B should have been ONE session.
correcting that means retracting two rows and emitting one.Merging is what makes them hard
The other window families have a comfortable property: a window's identity is fixed the moment it is created. A late event can change a window's *value* but never its boundaries. Session windows do not have that property, and everything difficult about them follows.
A late event that lands in the gap between two closed sessions does not update one of them — it proves that both were wrong, and that the correct output is a single merged session. Expressing that downstream requires deleting two rows and inserting one, which is a stronger requirement than upsert and which most sinks and most pipelines do not support.
Every row in the table below is survivable, and the reason to read them together is that they share a shape: the output remains completely plausible. Two sessions where there should have been one look exactly like two sessions. A truncated session looks like a short visit. This family produces no errors, only different answers.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A key never goes idle (bot, reconnect loop, stuck client) | One state entry grows without bound; state size climbs steadily with no change in event volume. | The session close condition requires a gap in event time that never occurs for that key. | Cap session length and end the session at the cap. Emit a flag recording that it ended by cap rather than by gap, so the analysis can exclude them (Hot Keys: When Aggregate Metrics Hide a Saturated Node). |
| A late event bridging two emitted sessions | Two plausible sessions in the output where the truth was one; session counts are slightly high and durations slightly low. | The bridging event arrived after both windows were purged, so there was nothing left to merge. | Widen the allowance if the tail is short; otherwise recompute sessions in batch from raw events, where the complete set is available (Late-Arriving Data). |
| The gap is inherited from convention rather than measured | The session-duration distribution has a sharp artificial mode; a large share of sessions contain exactly one event. | The gap does not match the product's real inter-event pattern, so it is manufacturing boundaries rather than finding them. | Plot inter-event intervals and choose the gap at the valley between within-visit and between-visit behaviour (Distribution Tests). |
| Duration computed two different ways by two teams | Two "average session length" figures that differ by roughly the gap, both defensible. | session_end includes the trailing gap; last_event does not. Neither definition is wrong and neither was written down. | Emit both last_event and session_end as columns and define duration once, in the metrics layer (The Metrics Layer). |
| A rescale or rebalance mid-session | Short fragment sessions appearing at a rate that correlates with deploys. | Key ownership moved between instances while a session was open, and state migration did not carry the open window as expected. | Use the engine's stateful rescaling path rather than restarting with fresh state, and treat deploys as a suspect when session counts jump (Deploys Are the First Suspect). |
The same rule, written in SQL
ROWS UNBOUNDED PRECEDING is required or implied, but the three-step shape is portable.Sessionisation is not a streaming concept — it is a gaps-and-islands calculation that predates stream processors by decades, and writing it in plain SQL is the fastest way to see exactly what the streaming operator is doing.
The query below is also the reconciliation. Because it operates on the complete set of events, it performs the merges the streaming job could not, so its session count is the true one. The difference between it and the streaming output is precisely the merge loss, which is otherwise invisible (Reconciliation).
Notice that the SQL makes the gap explicit twice: once as the threshold that starts a new island, and once as the addition that produces session_end. Those are the two places the parameter enters, and keeping them consistent between the batch and streaming implementations is the difference between a check and a second bug.
1-- Step 1: mark every event that starts a new session for its key.2WITH marked AS (3 SELECT user_id,4 occurred_at,5 CASE6 WHEN LAG(occurred_at) OVER (PARTITION BY user_id ORDER BY occurred_at) IS NULL7 THEN 18 WHEN occurred_at - LAG(occurred_at) OVER (PARTITION BY user_id ORDER BY occurred_at)9 >= INTERVAL '5 minutes'10 THEN 111 ELSE 012 END AS starts_session13 FROM raw_events14),15-- Step 2: a running sum over those markers numbers the sessions.16numbered AS (17 SELECT user_id,18 occurred_at,19 SUM(starts_session) OVER (PARTITION BY user_id ORDER BY occurred_at20 ROWS UNBOUNDED PRECEDING) AS session_seq21 FROM marked22)23-- Step 3: collapse each island. Emit BOTH end definitions so nobody24-- has to guess which duration a consumer meant.25SELECT user_id,26 session_seq,27 MIN(occurred_at) AS session_start,28 MAX(occurred_at) AS last_event,29 MAX(occurred_at) + INTERVAL '5 minutes' AS session_end,30 COUNT(*) AS events31FROM numbered32GROUP BY user_id, session_seq;Run this over the same period the streaming job covered and compare session counts. The batch version sees every event at once, so it merges what the stream could not — and the difference is a direct measurement of the merge loss.
How to build it
Most important first.
- Derive the gap from the data rather than from convention: plot the distribution of inter-event intervals per key and look for the valley between within-visit and between-visit behaviour. If there is no valley, sessionisation is imposing a structure the data does not have (Distribution Tests).
- Cap the session length regardless of activity. A hard maximum converts the unbounded case into a bounded one and turns a stuck client into several long sessions instead of one infinite state entry (Streaming State).
- Store an aggregate per session, not the event list, unless a consumer genuinely needs the sequence — and if one does, cap the list and record that it was truncated (Projection Pushdown).
- Emit
session_start,session_end,last_event_timeandevent_countas separate columns, so a consumer can compute duration either way and knows which they chose. Duration is ambiguous by exactly one gap and this makes the ambiguity explicit (Grain: What Does One Row Represent?). - Decide in advance which calendar period a session straddling midnight belongs to — start, end, or split — and write it down. Every daily session metric makes this choice and almost none document it (Dataset Documentation).
- Handle merges explicitly if late data matters: retract the two emitted sessions and emit the merged one, which requires a sink that supports deletes as well as upserts (Upserts and Merges).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Within one key, a session contains a maximal run of events with no internal gap of the configured length or longer. That is the definition and the engine enforces it exactly.
- Determinism given the complete set of events: the same events produce the same sessions on replay. Crucially, this holds only for the *complete* set — adding one late event in the middle can change the session structure retroactively.
- Sessions never overlap for a given key, so per-key session output is additive across sessions. It is not alignable with any fixed period, which is a different limitation from non-additivity (Sliding Windows).
- What is explicitly not guaranteed: that a session ever closes; that the session structure emitted is final; or that two keys' sessions are comparable in duration, since a session's length reflects both behaviour and the gap parameter.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Check the distribution of session durations and event counts rather than individual sessions. A gap that is wrong shows up as a bimodal or implausibly-shaped distribution long before anyone notices a wrong number (Distribution Tests).
- Assert an upper bound on session length and alert when it is hit. Sessions at the cap are either bots or a stuck client, and both are worth knowing about for reasons beyond data quality (Volume Anomalies).
- Both miss the merge failure: sessions that should have been one and were emitted as two look completely normal — two plausible sessions with plausible durations. Only a batch recomputation over the complete event set finds them (Reconciliation).
- A session is available only after the key has been quiet for the gap in event time, so freshness is bounded below by the gap. A thirty-minute gap means no session is ever fresher than thirty minutes plus the watermark delay.
- Long sessions are the freshest problem: an active key produces no output for as long as it stays active, so the busiest users are the ones whose data is most delayed. That is the opposite of what most consumers assume.
- Provisional emission is possible — emit the session-so-far on a processing-time trigger and correct it when it closes — and it requires consumers to tolerate a row whose
session_endmoves (Processing Time).
- Changing the gap redefines every session in the dataset. Historical rows are not comparable with new ones and there is no schema change to signal it — the most dangerous class of change in this domain (Semantic Changes).
- Changing the key — from cookie to user id, say — changes what a session *is*. It is a new dataset that happens to have the same column names, and it should be given a new name rather than a new version (Data Contracts).
- Adding a session-length cap changes the distribution of the output permanently and asymmetrically: it affects only the tail, which is exactly where bot traffic lives, so the aggregate metrics may move in ways that look like a behaviour change (Two Dashboards, Two Numbers).
- A full replay recomputes sessions correctly from complete data, including merges that the live job could not perform. This is the strongest argument for retaining raw events for a sessionised dataset (Keeping Raw History: The Recovery Position and the Liability).
- A bounded backfill is harder than for other window families, because a session can start before the backfill range and end inside it. The recompute range has to be extended by at least the maximum session length on each side (Planning a Backfill).
- Publishing a corrected sessionisation requires deleting the superseded sessions, not just upserting: a merge turns two rows into one, and an upsert alone leaves an orphan (Upserts and Merges).
What can go wrong
- A key that never goes idle, holding one state entry that grows without bound and is invisible in any per-key average (Hot Keys: When Aggregate Metrics Hide a Saturated Node).
- Two sessions emitted where one was correct, because the bridging event arrived after both had closed — plausible output, no error, no check (Late Events).
- Session duration computed as
session_end − session_start, which includes the trailing gap, compared against a figure computed aslast_event − first_event, which does not. Two teams, two numbers, both defensible. - A rescale mid-session splitting a visit across instances, producing a fragment that looks like a short session (Consumer Groups and the Parallelism Ceiling).
- The mitigation failing: a session-length cap that truncates genuinely long legitimate sessions — a video stream, a long form, a support chat — so the fix for bots becomes a systematic understatement of engaged users.
- "Thirty minutes is the standard session gap." It is a convention inherited from early web analytics, not a property of your product. A game, a support tool and a news site have completely different inter-event distributions and no shared correct gap.
- "Session duration is how long the user was there." It is
last_event − first_event, which excludes everything after the final action, orsession_end − session_start, which includes a whole gap of inactivity. Neither is "how long they were there" and the difference is exactly one gap. - "Sessions can be summed to daily totals." Sessions straddle calendar boundaries, so any daily aggregate requires a rule about which day a straddling session belongs to — and different tools pick different rules (Grain: What Does One Row Represent?).
- "Late events just update the session." Sometimes they *merge* two sessions that were already emitted separately, which requires retracting both. Most pipelines cannot express that and silently keep the wrong structure (Late Events).
- A session is behavioural data about an identifiable person, and holding open sessions means holding that behaviour in job state where no catalog, classification or retention policy can see it (PII in Pipelines).
- Keying sessions by a cookie rather than a user id does not make them anonymous — a session is a linkage structure, and linking is the operation that turns pseudonymous events into a profile. Treat the key choice as a privacy decision, not only a modelling one (Data Classification).
Operating it
- Open session count and the age of the oldest open session, per instance. The oldest-session age is the direct detector for the never-closes failure and costs nothing to emit (Streaming State).
- Distribution of session duration and events per session, tracked over time rather than checked once, because the shape drifts with product changes (Histograms: A Distribution You Can Afford to Keep Forever).
- Count of sessions ended by the length cap rather than by the gap, which separates bots and stuck clients from real behaviour.
- State bytes for the session operator specifically, since a session holding event lists rather than aggregates grows with activity and is usually the largest single state in a product-analytics pipeline.
- At 10x volume with the same keys, sessions get longer and denser rather than more numerous, so state grows only if you are storing events rather than aggregates.
- At 10x keys, active-key count grows and state grows with it, but the ratio of active to total keys usually stays low, so growth is sublinear in a way the other families' is not.
- At 100x, the tail dominates: a small number of never-idle keys hold most of the state, and a length cap plus per-key monitoring stops being an optimisation and becomes a precondition for the job running at all (Hot Keys: When Aggregate Metrics Hide a Saturated Node).
- State is one entry per active key, not per key — which is usually far smaller than a tumbling equivalent, because most keys are idle most of the time. Session windows are cheaper than they look for well-behaved data.
- The cost is entirely in the tail: value size grows with session activity if you store events, and a single never-closing session can hold more than every other key combined (Data Skew).
- Output volume is low relative to fixed windows — one row per visit rather than one row per key per period — which makes sessionised output cheap to store and query downstream (Fact Tables).
- Session windows buy boundaries that match how people actually behave, and cost an unbounded worst case, a merge problem under lateness, and output that cannot be aligned to a calendar without an arbitrary rule.
- A shorter gap buys more sessions and faster emission, and costs the splitting of genuine visits into fragments. A longer one buys fewer, more faithful sessions and costs freshness and state.
- A length cap buys a bound on the worst case and costs correctness for the longest genuine sessions — which are frequently the most valuable users, making this the most consequential trade in the lesson.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALGap-based grouping with merging is the same rule in every engine and in batch SQL, where it appears as a gap-and-island calculation over an ordered set rather than as retained state. What streaming adds is that merges may need to happen after emission.
- ENGINE-SPECIFICFlink implements session windows with genuine window merging, including merging windows whose state already exists; Spark Structured Streaming added a session window function whose late-data merge behaviour differs; Kafka Streams supports session windows over a session store with its own retention semantics. What happens to a late bridging event therefore differs materially between them.
- SCALE-SPECIFICBelow the point where a single key can stay active indefinitely — an internal tool with human users, say — the unbounded-session failure never occurs and a length cap is unnecessary ceremony. Any public-facing stream with bots or reconnecting clients needs it from the first day.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why proving that nothing happened is harder than observing that something did — a session closes on evidence of silence, and silence is exactly what a distributed system cannot distinguish from a delay.