SIMULATED

Stream Simulator

An event happened at 10:00 and arrived at 10:07. Which window does it belong to, and is it counted? Everything difficult about stream processing follows from that gap between when something happened and when you found out about it.

SIMULATEDEverything numeric on this page

Clock labels here — 10:00, 10:07 — are positions on a teaching timeline, not durations anybody measured. Lag figures come from the stream model in this repository. Which of these behaviours a given broker or engine actually exhibits varies, and the labs say where.

Two clocks. Event time is when the thing happened, and it is what every business question is about. Processing time is when your system saw it, and it is what is easy to implement. A pipeline that windows on processing time is answering a question nobody asked — and it never fails, it just quietly attributes a purchase to the wrong hour.

The unavoidable trade sits in one decision: how long to wait for late data before you emit a result. Wait longer and results are more complete and less timely. Wait less and the opposite. There is no configuration that gives you both, and a system that appears to be giving you both is discarding something without telling you.

Windows

Unbounded data cannot be aggregated without a boundary. A window is that boundary, and choosing it is a modelling decision, not a configuration one.

Window lab — tumbling, sliding, session

Move one event's event time and its arrival time independently. The first decides which window it belongs to; the second decides whether it is still open when the event turns up.

Window lab — tumbling, sliding, session
Events are assigned to windows by when they happened. When they arrived decides only whether the window was still open.
Scenario
Window kind
Six events. One of them happened at 10:00 and did not arrive until 10:07.
Windows
2sim
Events counted
5 of 6sim
Late, counted
0sim
Late, dropped
1sim
Windows emitted
10:00–10:0530 · 3 events · a b c
10:05–10:1020 · 2 events · d e
Events
idhappenedarrivedcounted
a10:0110:01yes
b10:0210:02yes
c10:0410:04yes
d10:0610:06yes
e10:0810:08yes
late10:0010:07dropped
late → 10:00–10:05: The window ended at 10:05 and the watermark had already reached 10:06 when this event arrived at 10:07. With 0 minutes of allowed lateness it is dropped.
SIMULATEDTimes are minutes on a teaching clock, not measurements. The assignment rule and the watermark arithmetic are the model's, and they are the part that transfers.
The three shapes, and what each is for
  • —Tumbling — fixed, non-overlapping. Every event lands in exactly one window, which is what makes the results additive across windows.
  • —Sliding — fixed length, advancing by less than its length. Every event lands in several windows, so the results are not additive and summing them double-counts.
  • —Session — bounded by a gap in activity rather than by the clock. The gap is a modelling parameter that changes every rate computed over it, and no source system emits one.
Session Windows →Late Events →