Batch and Streaming Unification
Modern engines let you express both with one API. The authoring surface converged; latency, state and completeness semantics did not.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
If the same SQL runs over a bounded table and an unbounded stream, has the distinction between batch and streaming gone away?
A team maintaining two implementations of the same metric — one in a nightly job, one in a streaming job — that have drifted apart, and an analyst who cannot explain why the real-time dashboard and the daily report disagree by two percent every morning (Two Dashboards, Two Numbers).
The unit is what one row of the result means, and this is exactly where unification stops. In batch, one output row is a final answer over a bounded input. In streaming, one output row is the answer *so far* for a window that may still receive events — a value that can be superseded (Late-Arriving Data).
Write the transformation once, in SQL or in a dataframe API, and run it in both modes: on a schedule to rebuild history, continuously to keep the current period fresh. The engine handles the difference, one definition serves both, and the Lambda-architecture problem of maintaining two codebases disappears (Lambda Architecture).
The streaming job emits a value for the 10:00–10:05 window at 10:05, and an event that happened at 10:04 arrives at 10:07. The batch rerun over the same period includes it; the streaming result did not, or did and emitted a correction. Same code, two answers, both correct under their own completeness rules (Watermarks).
- The streaming job emits a value for the 10:00–10:05 window at 10:05, and an event that happened at 10:04 arrives at 10:07. The batch rerun over the same period includes it; the streaming result did not, or did and emitted a correction. Same code, two answers, both correct under their own completeness rules (Watermarks).
- The batch job re-reads the whole day and produces a final table. The streaming job holds state per key, and that state has a retention policy, so a key that was quiet for longer than the retention window is treated as new rather than continued (Streaming State).
- A join behaves differently: in batch both sides are complete, in streaming a row can arrive before its counterpart and either wait, emit unmatched, or be dropped depending on the join type and the configured horizon (Stream Joins).
- The same code uses
now(), or an ordering that is stable in one mode and not the other, so the "one definition" is not actually one definition (Idempotent Data Pipelines). - The streaming job restarts and reprocesses from a checkpoint. Whatever the output sink does with the replayed rows decides whether the result is corrected, duplicated or silently doubled — and that is a property of the sink, not of the unified API (Exactly-Once: Input Consumption, State Update, Output Write).
- A backfill through the streaming path processes a year of history as fast as it can read it, so windows fill in wall-clock seconds and every timer, watermark and lateness setting behaves unlike production (What Backfills Break).
What is actually happening
- What genuinely unifies is the authoring surface: one SQL dialect or dataframe API, one set of connectors, one type system, one deployment model, and one set of operator semantics for the large class of operations where bounded and unbounded agree — projections, filters, most joins on keys, most aggregations (Query Engines).
- The formal idea underneath is that a bounded dataset is a special case of an unbounded one: a stream that ends. That is genuinely true and genuinely useful, and it is why the same logical plan can describe both (Stream Processing).
- What does not unify is completeness. A batch job knows its input is complete because the input is bounded. A streaming job never knows; it uses a watermark — a heuristic assertion that events older than some point are unlikely to arrive — to decide when to emit (Watermarks).
- Nor does state. Batch materialises intermediate results and discards them; streaming holds keyed state indefinitely, checkpoints it, restores it and must bound it. State size, state retention and state migration on code change are streaming-only concerns and they are where the operational cost lives (Checkpointing).
- Nor does the output contract. A batch job overwrites a partition; a streaming job appends, upserts or emits a changelog whose later rows retract earlier ones. A consumer that treats a changelog as an append-only log double counts (Upserts and Merges).
- Nor does latency, which is not a tuning parameter but the consequence of everything above: an engine that waits for a watermark waits, and one that emits early emits values that will change (Cost vs Freshness). So the honest summary is that unification removes duplicated *code* and does not remove the two sets of *semantics* — a large and real benefit, and not the one the marketing describes (Kappa Architecture).
What actually unified
The convergence is real and it is worth being precise about, because vague enthusiasm here is what produces the misreads. A unified engine genuinely gives you one language, one type system, one connector set, one deployment model and identical semantics for a large class of operators. That eliminates the maintenance problem Lambda architectures were criticised for: two implementations of one metric, drifting apart (Lambda Architecture).
What it does not give you is a single set of semantics for the operations where boundedness matters — which is windows, joins with a temporal dimension, and anything that depends on knowing the input is complete. Those are not gaps to be closed in a later release. A system processing an unbounded input cannot know that nothing further will arrive for a past window; it can only decide when to stop waiting (Watermarks).
Read the table as two lists. The left column is real convergence and is the reason to adopt a unified engine. The right column is what you still have to decide twice, write down twice, and monitor twice — and every disagreement between a real-time number and a daily number lives in the right column.
| Concern | Unified? | What that means in practice |
|---|---|---|
| Language and API | Yes | One SQL dialect or dataframe API expresses both. The business logic is written once and reviewed once (SQL Transformations). |
| Connectors and types | Yes | The same sources and sinks, with the same type mapping, in both modes. |
| Stateless operators | Yes | Projections, filters and simple expressions mean exactly the same thing over a bounded and an unbounded input (Stateless Stream Processing). |
| Aggregate functions | Mostly | The functions agree; which rows are inside the group does not, because that depends on completeness. |
| Completeness | No | Bounded input is complete by definition. Unbounded input uses a watermark, which is a heuristic and can be wrong (Watermarks). |
| State | No | Batch materialises and discards. Streaming holds keyed state, checkpoints it, restores it, and must bound its growth (Streaming State). |
| Output contract | No | Overwrite a partition, versus append, upsert, or emit a changelog whose later rows retract earlier ones (Upserts and Merges). |
| Latency | No | A consequence of completeness policy rather than a setting. Emitting sooner is emitting on less (Cost vs Freshness). |
| Recovery | No | A bounded re-run over an explicit range, versus a checkpoint restore plus a replay from a committed offset (Checkpointing). |
The same SQL, two meanings for one row
The clearest way to see where unification stops is to ask what one output row *is* in each mode. In batch it is a final answer over a complete input: the 10:00–10:05 window contains every event with an event time in that range, because the input ended and everything in it was read.
In streaming it is the answer as of a moment, for a window the engine has decided to close. Whether an event that happened at 10:04 is included depends on whether it arrived before the watermark passed 10:05 plus the allowed lateness. If it arrived after, it is either dropped or emitted as a correction that supersedes the earlier row — and a consumer that appends rather than replaces now has both (Late-Arriving Data).
The SQL below is identical in both modes and that is precisely the trap. Nothing in the text says which completeness rule applies, how late an event may be and still count, or what the sink should do with a second row for the same window. Those three answers are configuration and sink behaviour, and they are what decides the number (Windows).
| Stage | One row is | Breaks if |
|---|---|---|
| Input — batch | One event from a bounded set that has been fully read. | The partition was still being written when the job read it, so "bounded" was a claim rather than a fact (Atomic Publish). |
| Input — streaming | One event from an unbounded log, at whatever position the consumer has reached. | The offset was committed before the state that consumed it was checkpointed, so a restart skips events (Offsets and Commits). |
| Windowed aggregate — batch | The final value for that window over the complete input. | The run happens before all late events have landed, making the window final in form and incomplete in fact (Late-Arriving Data). |
| Windowed aggregate — streaming | The value for that window as of the watermark, possibly to be revised. | It is read as final. A window closed on a heuristic is not the same object as a window closed on exhaustion (Watermarks). |
| Output — batch | One row of a partition that is overwritten wholesale on each run. | A consumer reads the partition mid-write and sees a genuinely partial period. |
| Output — streaming changelog | One version of the answer for a key, superseding previous versions. | The consumer appends rather than upserting, so every revision is counted in addition to the value it replaced (Deduplication). |
| Dashboard tile | One number, with the completeness rule that produced it now invisible. | Two tiles fed by the two paths are placed on the same page and expected to match before lateness has resolved (Two Dashboards, Two Numbers). |
The transformation is shared. The grain of the output is not — a final value and a current-best value are different objects, and the SQL that produces them is identical.
1-- Runs unchanged in both modes on a unified engine.2SELECT3 window_start,4 country,5 sum(revenue) AS revenue6FROM TABLE(7 TUMBLE(TABLE events, DESCRIPTOR(event_time), INTERVAL '5' MINUTES)8)9GROUP BY window_start, country;10 11-- What the text does not say, and what decides the number:12-- 1. How late may an event be and still be counted?13-- Batch: as late as the run. Streaming: allowed lateness.14-- 2. When is a window final?15-- Batch: when the input ended. Streaming: never, strictly —16-- the watermark decides when to stop waiting.17-- 3. What does the sink do with a second row for the same window?18-- Overwrite, upsert, or append. Only the first two are correct19-- for a revisable result.Windowing syntax differs between engines; the shape here follows the SQL standard's table-valued window functions. The teaching is the comment block, not the syntax — those three questions have to be answered per mode, and they are answered outside the query (Windows).
Living with two semantics behind one API
The practical stance that works is neither "they are the same" nor "keep two codebases". It is: share the transformation, declare the completeness policy per mode, designate one path as authoritative for any published number, and reconcile continuously. Which path is authoritative is a decision, not a discovery, and it is nearly always the batch one because bounded input is the only thing that makes a number defensible later (Source of Truth).
That is close to the Lambda pattern and it is worth saying so plainly rather than pretending unification abolished it. What unification removed is the duplicated *logic*; what remains is the duplicated *decision about completeness*, which is much cheaper to maintain and does not drift on its own (Lambda Architecture, Kappa Architecture).
The traps below are the ones that reach production, and the common thread is that the shared code hides an unshared assumption. The failure is never a stack trace; it is two numbers on two dashboards, both defended by the person who built them.
Does the freshness change a decision, and does the number have to be defensible later?
when The number is read on a daily or weekly rhythm, or it must be reproducible and auditable.
cost Staleness between runs. Buys bounded recomputation, simple recovery and no state to operate (Batch vs Streaming Ingestion).
when A person or a system acts on the number within minutes, and an approximate current value is more useful than an exact stale one.
cost State, checkpoints, watermarks and a restart story per pipeline, plus a correction path instead of a re-run (Kappa Architecture).
when Operations need it live and finance needs it defensible — the common case for revenue and volume metrics.
cost Reconciliation, two sets of infrastructure, and an explicit statement of which number is the number (Lambda Architecture).
when Freshness of tens of minutes suffices and the input is append-mostly.
cost Watermark and late-data handling of its own, without the continuous state. Frequently the honest middle and rarely the exciting one (Incremental Processing).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A real-time tile and a daily report for the same metric. | They disagree by a small, consistent amount every morning. | Events arriving after the streaming watermark passed, included by the batch rerun. | Publish the delta as a known property with the late-event count beside it, and designate the batch result as authoritative for anything reported (Reconciliation). |
| A streaming changelog written to an append-only table. | Totals climb steadily and are far too high. | Each revision of a window was appended instead of replacing the value it supersedes. | Upsert on the window key, or read the changelog with an explicit latest-version-per-key model (Upserts and Merges). |
| A backfill run through the streaming job. | Windows look plausible and are wrong; the job completes far faster than expected. | Event time advances at reading speed, so watermarks, timers and lateness never behave as in production. | Backfill in batch over an explicit range and publish that; use the streaming path for the live period only (Planning a Backfill). |
| A logic change deployed to a stateful streaming job. | The job will not start, or starts with empty state and reports a collapse in volume. | The new code cannot restore state serialised by the old code. | Treat state schema as a contract with a compatibility policy, and keep a rebuild-from-history path that you have actually tested (Rolling Back Data). |
| Unbounded key cardinality in a streaming aggregation. | State grows without limit; checkpoints lengthen until they fail. | No retention on keyed state, in a key space that never stops growing. | Set state retention deliberately and accept that a key returning after expiry is treated as new (Streaming State). |
now() or processing time inside shared logic. | A rerun of the batch job produces a different answer than the original run. | The transformation is not a pure function of its input, so it is not the same definition in both modes (Processing Time). | Bind time once at the boundary and pass it in. This is also what makes the job idempotent (Idempotent Data Pipelines). |
How completely a given engine unifies the two modes — one runtime or two, whether retractions are emitted, how state schema evolution is handled — has changed materially in recent releases of every major engine and will change again. Verify against the documentation for the version you run, and verify the numbers with a reconciliation rather than with the documentation.
How to build it
Most important first.
- Write the business logic once and the completeness policy twice, explicitly. The transformation can genuinely be shared; the decisions about lateness, window closing and output contract cannot, and pretending otherwise is what produces two disagreeing numbers (Windows).
- Make event time the only time. Any logic that depends on processing time or on
now()will behave differently in the two modes by construction, and will also behave differently on a replay (Event Time, Processing Time). - Define the output contract before choosing the mode: is this table overwritten, appended, or upserted from a changelog? The consumer needs to know, and it is the difference between correct and double-counted (Atomic Publish).
- Reconcile the two continuously where both run. A daily comparison of the streaming result against a batch recomputation of the same closed period is the check that keeps them honest, and the delta it reports is the lateness you are actually experiencing (Reconciliation).
- Reserve streaming for cases where the freshness changes a decision. Continuous processing carries state, checkpoints, watermarks and a restart story; a scheduled job carries none of them, and "more modern" is not a requirement (Batch vs Streaming Ingestion).
- Test the streaming path with a replay that respects event time rather than one that reads as fast as it can, or the test exercises a system your production never resembles (Replay from the Log).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The shared API guarantees the same *logical* transformation in both modes: the same projections, filters and aggregate functions over the same input rows.
- It does not guarantee the same *input rows*. Which events are inside a window depends on completeness, and completeness is bounded input in one mode and a watermark heuristic in the other (Watermarks).
- Streaming output is a value that may be revised. Where the engine emits corrections, the guarantee is eventual convergence for events that arrive within the allowed lateness — and nothing at all for events that arrive after it (Late-Arriving Data).
- End-to-end "exactly-once" is never a property of the API but of a combination — replayable input with committed offsets, checkpointed state restored atomically with those offsets, and an output sink that is transactional or idempotent. Naming which of the three you have is the only honest way to discuss it (Exactly-Once: Input Consumption, State Update, Output Write).
- Nothing guarantees that a batch rerun and a streaming run over the same period agree. If they must, that is a reconciliation you build and an authority you designate — usually the batch result (Source of Truth).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check is a dual-path reconciliation: recompute a closed period in batch and compare with what streaming published for it. It catches lateness beyond the allowed horizon, dropped events, duplicated emissions and state that expired early.
- It misses everything about the current, still-open period — which is exactly the period the streaming path exists to serve. Reconciling closed windows tells you nothing about the number on the live dashboard right now (Freshness Checks).
- Pair it with a late-event counter and a watermark-lag metric. The reconciliation says the two disagree; those two metrics say why (Volume Anomalies).
- This is the axis that did not converge, and it did not converge because it cannot: emitting sooner means emitting on less complete information. That is a trade, not an implementation detail (Cost vs Freshness).
- A unified engine lets you place the same logic anywhere on that trade — continuous with early emission, continuous with watermark-gated emission, micro-batch, or scheduled — which is genuinely valuable and is a choice per pipeline rather than a platform-wide stance.
- The consumer-visible freshness of a stream is not the processing latency; it is the watermark position, because that is what decides when a window is allowed to close (Freshness Monitoring).
- Schema changes affect the two paths differently. A batch job reads whatever the files now contain; a streaming job holds state serialised under the old schema and must migrate or discard it on restart (Schema Evolution).
- Changing the transformation logic is a redeploy in batch and a state-compatibility question in streaming: whether the new code can restore the old checkpoint decides whether the change is a rolling upgrade or a rebuild from history (Rolling Back Data).
- Changing a window definition or an allowed-lateness setting changes the meaning of previously emitted rows without changing any schema — the archetype of a semantic change that no type system catches (Semantic Changes).
- Batch recovery is a bounded re-run over an explicit range, which is why backfills are a batch idea (Planning a Backfill).
- Streaming recovery is a restore from checkpoint plus a replay from a committed offset, and it is only correct if state and offset are restored together. Restoring state without the matching offset reprocesses events the state already includes (Checkpointing, Offsets and Commits).
- Reprocessing history through the streaming path compresses event time into wall-clock time and therefore exercises different watermark, timer and lateness behaviour than production. Where the correction must be trustworthy, run it in batch and publish that (Reprocessing vs Retrying).
What can go wrong
- Two numbers for one metric, both defensible, differing by whatever arrived late (Two Dashboards, Two Numbers).
- A changelog output consumed as an append-only stream, double counting every revision (Deduplication).
- Streaming state growing without bound because a key space is unbounded and no retention was set (Streaming State).
- A code change that cannot restore the existing checkpoint, turning a routine deploy into a rebuild from history.
- A backfill run through the streaming path producing plausible, wrong window contents at high speed (What Backfills Break).
- The mitigation failing: a dual-path reconciliation that runs daily, alerts on a delta that is normal lateness, and is muted within a month (Alert Fatigue: The Page Nobody Reads).
- "Batch and streaming are the same thing now." The authoring surface converged. Completeness, state, latency and output contracts did not, and every disagreement between two numbers lives in one of those four (Batch vs Streaming Ingestion).
- "One API means one definition, so the numbers must match." They match only for periods where both paths saw the same events, which is periods old enough that all lateness has resolved.
- "Streaming is more modern, so it is the better default." Compare freshness against complexity, cost, failure handling and operational burden for the pipeline in question. Most reports are read once a day (Batch vs Streaming Ingestion).
- "We can run the backfill through the streaming job." You can, and event time will pass at the speed of reading, so windows, timers and lateness will behave unlike production (Reprocessing vs Retrying).
- "The engine gives exactly-once, so the numbers are safe." Ask which of input consumption, state update and output write that applies to, and what assumption buys it. Without a transactional or idempotent sink, the output is at-least-once whatever the engine promises (Exactly-Once: Input Consumption, State Update, Output Write).
Operating it
- Watermark lag — the gap between the watermark and wall-clock time — per streaming job. It is the single most informative streaming metric and it has no batch equivalent (Pipeline Metrics).
- Late-event count and dropped-event count, separately. Late-but-accepted and late-and-dropped are different problems with different fixes (Late-Arriving Data).
- State size and checkpoint duration over time. Both grow silently and both fail loudly, much later (Checkpointing).
- The dual-path delta per closed period, as a chart. It is the honest measurement of what your streaming path is missing (Reconciliation).
- At 10x event volume, batch scales by reading more per run and streaming scales by holding more state and checkpointing more often — different constraints, hit at different times (Stream Processing).
- At 10x key cardinality, streaming state is the binding constraint and batch is barely affected. Cardinality, not volume, is what breaks streaming jobs (Data Skew).
- At 10x pipelines, the operational surface diverges sharply: continuous jobs each carry a restart story, a state migration story and a watermark to monitor, and that per-pipeline overhead is what makes teams selective about which ones stream (Pipeline Reliability).
- A continuously running job holds compute permanently; a scheduled job holds it for its run. That is the largest structural cost difference and it is independent of data volume (Compute Waste).
- State storage and checkpointing are streaming-only costs and they scale with key cardinality rather than with event volume (Partition Cardinality).
- Maintaining one codebase instead of two is the real saving unification delivers, and it is an engineering cost rather than an infrastructure one (Data Platform Engineering).
- Reconciliation between the two paths is an ongoing cost that the unified story tends to omit, and skipping it is how the two numbers drift apart unnoticed.
- One API buys a single definition of the business logic, one set of connectors and one skill set, and costs the illusion that the two runtimes are interchangeable. The illusion is expensive precisely because the code really is shared (Data Contracts).
- Running the streaming path as the only path buys freshness and costs the ability to bound a recomputation — a correction becomes a replay of history rather than a re-run of a range (Kappa Architecture).
- Running both paths buys a check on the streaming result and costs the reconciliation, the second set of infrastructure and the question of which one is authoritative when they differ (Lambda Architecture).
Batch and stream, same aggregation
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
| Window | Batch | Stream | |
|---|---|---|---|
| 10:00–10:05 | 40sim a b c late | 30sim a b c | differ |
| 10:05–10:10 | 20sim d e | 20sim d e | match |
| Batch | Stream | |
|---|---|---|
| Answers | When the period is over. | Continuously, and revises. |
| Late data | Included, because it is already there when the job runs. | Included only within the allowed lateness; after that, dropped. |
| Re-running | Idempotent if the pipeline is. Re-running yesterday gives yesterday. | Requires replaying the log through the same state, which is a different operation with different bugs. |
| Right when | The decision is made after the period, which covers most reporting. | The decision cannot wait — and somebody has agreed that a number may be revised. |
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- ENGINE-SPECIFICHow far unification goes differs substantially: some engines run one runtime that treats bounded input as a stream that ends, while others compile a shared API down to two distinct runtimes with different operator implementations. The second kind can differ in results at the edges even when the code is identical.
- GENERALThe four things that do not unify — completeness, state lifetime, output contract and latency — follow from bounded versus unbounded input rather than from any product. No engine can know that no further event will arrive for a past window; it can only assert a watermark.
- SIMPLIFIEDPresented as two modes. In practice there is a continuum: scheduled batch, micro-batch, continuous with watermark-gated emission, and continuous with early emission plus retractions, each with its own freshness and revision behaviour.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why a watermark is a heuristic rather than a guarantee: without a bound on message delay, no system can know that no further event will arrive for a past window. Every completeness decision in this lesson is an application of that result.
- — DevOps / Production Engineering owns the deployment question a stateful streaming job raises — a rolling upgrade that must restore state written by the previous version is a schema migration wearing a deploy's clothes.