Stream Processing
Computation over an input that has no end, where every result is provisional, time becomes a data field, and the job is a long-lived process holding state rather than a script that finishes.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What actually changes about a computation when its input never ends?
Anything that cannot wait for a batch boundary: a fraud rule that must fire before the shipment leaves, an operational dashboard a support team watches during an incident, a feature store an online model reads at request time, an alerting system that compares a rate against a threshold. Each of them needs an answer *now* and is willing to accept that the answer may be revised.
One event — a single immutable record of something that happened, carrying its own timestamp and key. Everything in this module is a question about which set of events an operator is allowed to consider at a given moment, which is the only interesting question once the input is infinite.
Run the batch job more often. The nightly transformation becomes hourly, then every five minutes, and each run reads "everything since the last run" and appends. There is no new technology to learn, the code is the SQL people already understand, and for a long time this genuinely produces fresher numbers.
Each run reads WHERE event_time > :last_run, so any event that happened before :last_run but arrived after it is skipped forever. The run succeeds. The count is quietly low, and it is low by an amount that varies with upstream latency (Incremental Extraction).
- Each run reads
WHERE event_time > :last_run, so any event that happened before:last_runbut arrived after it is skipped forever. The run succeeds. The count is quietly low, and it is low by an amount that varies with upstream latency (Incremental Extraction). - The interval shrinks until fixed per-run overhead dominates: planning, listing files, spinning workers, committing. Below a certain interval most of the wall clock is spent starting and stopping rather than processing (Compute Waste).
- Every run writes a new small file. At five-minute granularity a single day produces hundreds of them per partition and the read side collapses under file listing long before the write side does (File Size and the Small-Files Problem).
- A question arrives that needs memory across runs — "sessions longer than thirty minutes", "the third failed login in a row" — and there is nowhere to keep it. Each run starts empty, so the state has to be re-read from the output table, which makes the job non-idempotent and its result dependent on when it ran (Idempotent Data Pipelines).
- Someone asks for a correction: an event arrived late and yesterday's number should change. The batch answer is to recompute the partition, which is correct and takes as long as a partition takes. There is no mechanism for the *stream* to revise an answer it already emitted (Late-Arriving Data).
What is actually happening
- A stream processor is a long-lived process subscribed to one or more partitions of a log. It reads records in offset order, updates in-memory (and on-disk) state, and emits output continuously. It does not terminate, so there is no natural moment at which a result is final (Kafka as a Log, Not a Queue, Offsets and Commits).
- Because there is no end of input, every aggregate needs an artificial boundary. That boundary is either a window over event time, or a trigger over processing time, or an explicit key expiry. Choosing it is the central design act of a streaming job (Windows).
- Time stops being metadata and becomes a first-class field. An event carries the time it happened; the platform records when it arrived; the processor knows when it ran. Confusing any two of these produces answers that are internally consistent and wrong (Event Time, Ingestion Time, Processing Time).
- Progress is tracked by two independent positions: the offset, which says how far through the log the consumer has read, and the watermark, which says how far event time is believed to have advanced. They move independently, and a job can be caught up on offsets while its watermark is stalled behind an idle partition (Watermarks).
- Fault tolerance is not "retry the job". It is checkpoint and restore: periodically snapshot the state plus the input positions, and on failure restart from that pair. That coupling is what makes a restart produce the same output rather than a doubled one (Checkpointing).
- The output is a stream too. Downstream, "the answer" is a sequence of updates to a key rather than a row that is written once, which is why streaming sinks are usually upserts rather than appends (Upserts and Merges).
A job that never finishes
A batch job has a shape everyone recognises: it starts, it reads a bounded input, it writes a bounded output, it exits, and its exit code means something. A stream processor has none of that. It starts once and then runs until you stop it, which removes the two things batch quietly relied on — a moment at which the input is complete, and a moment at which the answer is final.
What replaces them is state and time. The processor holds accumulated results between records, so it can answer questions that span more than one event; and it carries an explicit notion of how far time has progressed, so it can decide when a group of events is complete enough to emit. Those two additions are the whole subject, and everything else in this module is a consequence of one or the other.
The diagram below is the shape of every stream processing job regardless of engine. Note that state sits beside the processor rather than downstream of it — it is read and written on every record, which is why it must be local, and being local is why it must be checkpointed somewhere durable.
Where the boundary really is: completeness, not latency
The usual way to explain the difference is latency, and it is the wrong way. A batch job scheduled every minute is faster than a streaming job that waits ten minutes for late data, and both statements can be true of the same platform on the same day. The distinction that survives is about completeness.
A batch job is handed an input it treats as complete. Whether it really is complete is somebody else's problem — an extract window, a partition marker, a sensor file that landed. The job's contract is: given this input, produce this output, once.
A stream processor is never handed a complete input, so it has to decide for itself when a group of events is complete enough to act on, knowing it may be wrong. Every distinctive piece of machinery in this module — watermarks, allowed lateness, side outputs for late records, retractions — exists to manage that one admission.
Which means the honest question when someone asks for "streaming" is not "how fast" but "what should happen to an event that arrives after we have already answered". If the answer is "nothing, it doesn't happen", they want a batch job with a shorter schedule. If the answer is "the previous answer should change", they want a stream and should be told what that costs.
Schedule the existing transformation every five minutes. Each run selects records where `event_time` falls in the last interval, aggregates them, and appends the result. Nothing is held between runs.
A long-lived job assigns each event to a window by its own event time, holds partial aggregates in keyed state, and emits a window when the watermark passes its end — with a stated policy for records that arrive after that.
The batch version has no way to represent an event that arrives outside its interval, so it either silently drops it or double-counts it depending on which timestamp the predicate uses. That is not a tuning problem; there is nowhere in the design to put the answer. The streaming version costs a permanent process and a state migration story, and buys a defined, observable behaviour for exactly that case.
What the job has to promise at each step
Read the guarantees column below downwards and the module writes itself. Delivery is at-least-once, so the sink must be idempotent. Ordering is per-partition, so any cross-key ordering assumption is already wrong. The window stage promises nothing about events that have not arrived, which is what makes watermarks necessary rather than optional.
The stage most people skip is the trigger — the decision to emit. It is where correctness and latency are traded against each other, it is configured separately from the window in every engine, and it is the single knob that decides whether your output is early and wrong or late and right.
Note also that no stage promises the output is *final*. In a streaming system, "final" is a property you manufacture by declaring a lateness bound and refusing anything past it. Consumers who believe streaming output is final are relying on a promise nothing in the chain has made.
- 1Consume
Reads records from assigned partitions in offset order and tracks its position.
guarantees Every record is delivered at least once, in order within a partition, and not in any order across partitions.
fails by Committing offsets before the effect is durable, which turns a crash into silent data loss rather than a duplicate.
- 2Extract time
Reads the event-time field from the payload and attaches it to the record.
guarantees Only that the field was present and parsed. Not that the producer's clock was right or that the field means what its name suggests.
fails by Falling back to arrival time when the field is missing, so a subset of records is silently on different time semantics from the rest.
- 3Key and shuffle
Partitions records by key so all events for a key reach the same operator instance.
guarantees All records for one key are handled by one instance at a time, so per-key state is safe without locks.
fails by Skew — one key with most of the traffic makes one instance the whole job's throughput (Data Skew).
- 4Assign window
Maps the record's event time to one or more window boundaries.
guarantees Assignment is deterministic and depends only on event time — replaying gives identical assignment.
fails by Being fed processing time by accident, at which point replay produces different windows than the original run did.
- 5Accumulate
Updates the per-key, per-window aggregate in state.
guarantees Durable across restarts if and only if state is checkpointed with the input positions.
fails by Unbounded growth when the key space is unbounded or windows are never purged.
- 6Trigger
Decides that a window may be emitted, normally when the watermark passes its end.
guarantees That the emitted result includes every event that had arrived when the trigger fired. Nothing about events that had not.
fails by A stalled watermark: no trigger ever fires, output stops, and consumer lag stays at zero the whole time.
- 7Emit
Writes the result to the sink as an update keyed by window and group.
guarantees Whatever the sink gives you — an upsert is idempotent, an append is not.
fails by Appending, so every replay adds the same result again and the metric steps upward.
- 8Purge
Drops window state once no further update is possible.
guarantees That state is bounded by the retained time range times the key cardinality.
fails by Retaining windows to catch stragglers, which makes the mitigation for late data the cause of the memory problem.
Eight stages, and only two of them ("consume", "emit") exist in a batch job. The other six are the price of an input with no end.
How to build it
Most important first.
- Start from the decision the output drives, not from the freshness you could achieve. If the number is read once a morning, a stream buys nothing and costs a permanently running job with state, on-call and a restore procedure (Batch vs Streaming Ingestion).
- Decide the time semantics before anything else. Write down which field is event time, who assigns it, and how far out of order events may be. Everything downstream — windows, joins, lateness — is a consequence of that one decision (Event Time).
- Keep the streaming layer thin and push the modelling downstream. Streaming jobs are expensive to change, because changing one usually means discarding or migrating its state; a SQL model over the stream's output is cheap to change (Model Layering).
- Make the sink idempotent on a business key from the first version. A streaming job will be restarted, re-deployed and replayed, and an append-only sink turns every one of those into duplicate rows (Idempotent Data Pipelines, Deduplication).
- Retain the raw stream in the log and in object storage. The ability to replay is the only recovery mechanism a streaming job has, and it is bounded by retention rather than by disk (Retention and Replay, The Raw Landing Zone).
- Emit an operational signal for the two things that actually go wrong: consumer lag (are we keeping up) and watermark lag (is event time advancing). Neither is visible from a task-status monitor because the task never ends (Pipeline Observability).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Delivery from the log is at-least-once by default. A consumer that crashes after processing and before committing its offset reprocesses that range on restart, which is correct behaviour and produces duplicate effects at the sink unless the sink deduplicates (At-Least-Once Delivery).
- Ordering is per-partition only. Two events for the same key are ordered relative to each other if and only if they share a partition; there is no global order and there never was (Topics and Partitions, Event Keys and Partition Assignment).
- Completeness is provisional. At any instant the answer covers the events that have arrived, and nothing promises that every event for a given period has. Whether that gap ever closes is decided by the watermark and the lateness policy, not by the processor (Late Events).
- Atomicity of the *output* is whatever the sink gives you. A stream processor writing to an object store with no table format has no atomic publish, so consumers can and do read half-written results (Atomic Publish).
- What is explicitly not promised: that a result will not change; that two runs over the same log produce the same output if the job used processing time; that events arrive in event-time order; or that a restart resumes at exactly the record it stopped on.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The load-bearing check is a reconciliation against a batch recomputation of the same metric over a closed period: run the equivalent aggregate over the raw events in storage and compare with what the stream emitted for that period. It catches dropped late events, state lost across a restart, and a window boundary that does not mean what you thought.
- It misses anything inside a period that is still open, anything where the batch job shares the same bug as the stream (a shared UDF, the same wrong timestamp column), and any error that affects both by the same amount.
- Add a counter for records the job dropped as too late and alert on any non-zero value. Silent dropping is the characteristic data loss of this module, and by construction nothing else reports it (Late Events).
- Streaming removes the *schedule* from the latency budget, not the latency. What remains is producer buffering, network transit, broker append, consumer poll, and — for anything windowed — the wait for the watermark to pass the window end. That last term usually dominates and is chosen by you.
- A windowed streaming aggregate is not fresher than its window. A five-minute tumbling window with a lateness allowance publishes a complete answer for a period only after that period has ended and the allowance has elapsed; the "streaming" part means the *next* period starts immediately, not that the current one is instant.
- Consumers can have either a final answer late or a provisional answer early, and the only honest design is to say which one they are getting. A dashboard showing an incomplete current window next to complete past windows will be read as a drop in volume by every person who sees it.
- Changing a streaming job usually means changing its state schema, and state is not schema-evolvable the way a table is. Adding a field to a keyed state value may be supported; changing a key, a window definition or an aggregation type is generally a new job with fresh state (Schema Evolution).
- The standard migration is to run the new job alongside the old one from an earlier offset, write to a separate sink, compare, then cut consumers over. That requires the log to still hold the range you need, which makes retention an evolution decision as well as a recovery one.
- A change in the *meaning* of an event field propagates instantly and invisibly, because there is no daily run to notice it at. Contract enforcement at the producer is worth more here than anywhere else in the platform (Data Contracts, Semantic Changes).
- Recovery is replay: rewind the consumer to an earlier offset and reprocess. The result is correct only if the job is deterministic given the log, which fails the moment it uses
now(), processing-time windows, or a lookup against a table that has since changed (Replay from the Log). - Replay bounded by log retention is the hard limit. Beyond it, recovery means reprocessing from the raw records in object storage with a batch job that must produce the same answer as the stream — which is the real, unglamorous reason so many platforms end up with two implementations (Lambda Architecture).
- Never recover by "restarting from latest". It looks clean, it makes the lag graph green, and it silently drops every record between the failure and the restart. The gap will be discovered weeks later by a reconciliation, if you have one.
What can go wrong
- The job is running, the lag is zero, and the watermark has not moved for an hour because one partition is idle. Windows never close, output stops, and every operational metric looks healthy (Watermarks).
- State grows without bound because a key space is unbounded — session state per anonymous visitor, join buffers with no time bound — and the job dies on memory long after the design decision that caused it (Streaming State).
- A restart replays a range and the sink appends it again, so a metric steps up by exactly the amount that was reprocessed. The job is behaving correctly; the sink is not idempotent (Duplicate Rows).
- A rebalance moves partitions between instances mid-flight, and an operator that assumed it saw every event for a key sees a discontinuity (Consumer Groups and the Parallelism Ceiling).
- The mitigation fails too: a lateness allowance large enough to catch stragglers keeps every window open that long, so state grows and output latency rises by the same amount you added.
- "Streaming is more modern, so it is better." It is a different trade, not a later version of batch. Compare freshness, complexity, cost, failure handling and operational burden per use case; plenty of correct architectures have no streaming layer at all (Batch vs Streaming Ingestion).
- "Real-time means instant." It means unbounded and continuous. A windowed streaming aggregate is often *less* fresh than a well-scheduled batch job, because it waits for a watermark that batch never had to think about.
- "The stream and the batch job compute the same metric, so they agree." They agree only if they use the same time semantics, the same lateness policy and the same deduplication rule. Two implementations of a metric are two definitions of it (Two Dashboards, Two Numbers).
- "Lag is zero, so we are fine." Lag measures how far behind the log you are. It says nothing about whether the events are correct, whether the watermark is moving, or whether the sink is deduplicating.
Operating it
- Consumer lag per partition, not aggregated. One stuck partition is invisible in a sum and is the most common shape of a streaming incident (The Backlog Arithmetic: Four Levers and a Drain Time).
- Watermark lag: current watermark versus current wall clock, per job. This is the signal that distinguishes "we are behind" from "event time has stopped advancing", which have opposite fixes.
- Records dropped as late, records emitted per window, and checkpoint duration and size over time — a checkpoint that is getting slower is state growth reported early enough to act on.
- Output freshness measured at the sink, because that is the only place that reflects the whole chain including the wait for the watermark (Freshness Monitoring).
- At 10x throughput the answer is usually more partitions and more parallel instances, and the ceiling is the partition count — consumer instances beyond it are idle and change nothing (Consumer Groups and the Parallelism Ceiling).
- At 10x *key cardinality* nothing about throughput changes and the job may still die, because state is sized by distinct keys and retained windows rather than by records per second.
- At 100x, skew decides everything: one hot key lands on one partition, one instance does most of the work, and adding instances does not help because the work is not divisible (Data Skew, Hot Keys: When Aggregate Metrics Hide a Saturated Node).
- Consumer count scales independently. Adding a second job reading the same topic costs the broker read bandwidth and costs the first job nothing, which is the property that makes a log a better fan-out point than a database.
- A streaming job costs continuously whether or not data is flowing. The relevant comparison with batch is not per-record cost but occupancy: a job that is idle 90% of the time still holds its slots, its memory and its state.
- State is the second driver and the one that surprises people: it is memory, local disk, checkpoint storage and checkpoint network traffic, all growing with key cardinality rather than with throughput (Streaming State).
- Small output files are the third. A streaming sink writing frequently produces exactly the file-count problem that makes the read side expensive, which is why compaction is a required companion to any streaming write into a lake (File Compaction).
- Streaming buys freshness and continuous state, and costs a permanently running system with its own on-call, a state migration story for every code change, and a recovery procedure that is bounded by retention rather than by storage.
- Every increase in correctness — larger lateness allowance, transactional sink, exactly-once state coupling — is paid for in latency, throughput or operational complexity. Buying all of them for a metric nobody reads before lunch is the most common over-engineering in this module.
- A streaming job is much harder to change than a SQL model, so the cheapest architecture keeps the stream dumb (parse, key, land) and does the volatile business logic downstream in batch, accepting the extra hop (Batch and Streaming Unification).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALUnbounded input, event/processing/ingestion time, windows, watermarks and state are the primitives of every stream processor; they predate all current products and will outlive them. What varies is which knobs are exposed and what they are named.
- ENGINE-SPECIFICFlink processes record-at-a-time with its own event-time machinery; Spark Structured Streaming executes micro-batches, so its latency floor is a batch interval and its watermark advances per batch rather than per record. Kafka Streams is a library embedded in your application with no cluster of its own, which changes deployment and scaling entirely.
- SCALE-SPECIFICBelow the volume at which a five-minute batch job misses its window, streaming is a strictly worse trade: same answers, more failure modes. The advice inverts once state must persist between runs or a decision genuinely cannot wait for a boundary.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the delivery and ordering semantics this whole module inherits — what at-least-once really costs, why a global order across partitions is not available cheaply, and what a consistent snapshot across operators actually requires.
- — DevOps / Production Engineering owns deploying and rolling back a job that cannot be stopped and restarted freely, because stopping it means deciding what happens to its state.