Stream Processing

Computation over an input that has no end, where every result is provisional, time becomes a data field, and the job is a long-lived process holding state rather than a script that finishes.

Filter, map, transform: operators whose output for a record depends only on that record. The cheapest, most restartable, most rescalable thing a stream can do — and a much narrower category than it first appears.

Counting, joining, windowing and deduplicating all require remembering something between records — which turns a job into a database you have to operate.

Where the state physically lives, what makes it grow, how it is snapshotted and restored — and why state size, not throughput, is the number that decides whether a streaming job can be operated.

The time the thing actually happened, carried in the record itself — the only clock that makes a result reproducible when you process the same data again next year.

The wall clock of the machine doing the work: always available, always monotonic, never late — and the reason a replay produces a different answer than the original run.

The moment a record entered the processing system, stamped by one clock the platform controls — the timestamp that makes lag measurable and replay stable without pretending to know when anything happened.

An event happened at 10:00 and arrived at 10:07. The 10:00–10:05 window was already emitted. What happens next is a policy decision, and most platforms have made it by accident.

A window is a rule that turns an infinite stream into a set of finite groups you are allowed to aggregate — and choosing the rule decides your state size, your latency and what questions you can answer.

Fixed size, contiguous, non-overlapping: every event lands in exactly one. The cheapest window and the only family whose results you are allowed to add together.

Overlapping windows of fixed size, advancing by a smaller step. Every event lands in size ÷ slide of them, which is exactly the factor by which state, output and the risk of double counting all multiply.

Windows whose boundaries the data draws: a session runs until a key goes quiet for longer than a gap. Per-key, data-dependent, mergeable — and the only window family with no upper bound on its own size.

An estimate of how far event time has progressed, derived from the data itself. It decides when a window may be emitted and what counts as late — and it is a claim, not a measurement.

Joining two unbounded inputs means holding both sides in state until a time bound says you may stop holding them. Without that bound it is not a join — it is a memory leak with a schema.

There is no single exactly-once guarantee — there are three separate questions, one about input consumption, one about state update and one about output write, and each is bought by a different, nameable assumption.