Processing Time
The wall clock of the machine doing the work: always available, always monotonic, never late — and the reason a replay produces a different answer than the original run.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What questions are genuinely about my system rather than about the world, and therefore belong on the processor's own clock?
Operators, not analysts. Processing time answers the questions an on-call engineer asks — how much did we handle in the last minute, are we keeping up, how long has this been stuck — and it answers them without waiting for any watermark, which is exactly what you need at three in the morning.
One record, at the moment one particular machine handled it. The grain carries an implicit dimension nobody writes down: *which run*. The same record processed twice has two processing times, and that is the property that makes it useless for business metrics and perfect for operational ones.
Bucket everything by the processor's clock. There is no field to validate, no null to handle, no clock skew from a producer you do not control, no watermark, no lateness, nothing arrives out of order, and every window closes on schedule.
A replay of last week through the same job produces one bucket dated today. Every historical comparison is destroyed and the placement information needed to fix it was never recorded (Replay from the Log).
- A replay of last week through the same job produces one bucket dated today. Every historical comparison is destroyed and the placement information needed to fix it was never recorded (Replay from the Log).
- An upstream stall of twenty minutes shows as a trough followed by a spike. Both are artefacts of the transport; the underlying activity was flat, and somebody will spend an afternoon explaining the spike (Stale Dashboards).
- A mobile client uploads three hours of buffered events on reconnect and all of them are counted in the current minute, so a per-minute rate alert fires on data that describes the morning (Late Events).
- Two consumers of the same topic compute the same daily total and disagree, because each used its own machine clock and they processed the boundary at different moments (Two Dashboards, Two Numbers).
- An incident review asks what the rate was at 14:32 and the answer is unreproducible, because the only way to get it was to have been running at 14:32.
What is actually happening
- Processing time is
now()on the machine executing the operator. It requires no field in the payload, no coordination with any producer and no estimate of progress — the clock always advances, so windows always close (Event Time). - That independence from the data is exactly what makes processing-time windows non-deterministic. Window membership depends on when the record arrived at the operator, which depends on transport, retries, restarts, parallelism and load. Two runs over identical input produce different windows.
- Because it never waits, processing time has no late data by construction. A record is in whatever window was open when it arrived. Nothing is dropped and nothing is held, which is why processing-time jobs have the simplest state and the lowest latency of the three time domains.
- It is the correct clock for measuring the system: throughput, lag, backlog age, time-since-last-record, operator latency. All of these are statements about the machinery, and the machinery's own clock is the authoritative source for them (The Four Golden Signals).
- It is also the natural clock for triggers even in an event-time job. Emitting a provisional result every thirty seconds while a window remains open is a processing-time decision layered on top of event-time assignment — the two coexist and are configured separately (Windows).
- Machine clocks are not synchronised with each other. Two instances of the same operator can disagree, so a processing-time window is not even consistent across the parallelism of one job — a detail that matters when the boundary is close to the aggregation you care about (Ordering Guarantees: Four Levels, Four Prices).
The only clock that is always available
Processing time has one enormous advantage and it is worth stating plainly before the criticisms: it cannot fail. There is no field to be missing, no producer clock to be wrong, no watermark to stall, no straggler to wait for. A processing-time job always produces output, on schedule, forever.
That property makes it the correct choice for a specific and important class of questions: those about the system itself. How many records did we handle this minute. How long has the oldest unprocessed record been waiting. Is the consumer keeping up. Each of these is a question about the machinery, and using the event's own timestamp to answer them would be actively wrong.
It is also the right clock for triggers, which is the use that survives even in a rigorous event-time pipeline. Windows are assigned by event time so the answer is reproducible, and a processing-time trigger emits a provisional result periodically so consumers do not wait. The two clocks are doing different jobs in the same operator, and both are correct.
1# PySpark Structured Streaming. Note which clock does which job.2(events3 # Event time decides WHICH window a record belongs to. This is what4 # makes the result reproducible on replay.5 .withWatermark("occurred_at", "10 minutes")6 .groupBy(window(col("occurred_at"), "5 minutes"), col("account_id"))7 .agg(sum("amount_minor").alias("amount"))8 .writeStream9 # Processing time decides HOW OFTEN we emit. This is a latency knob10 # and has no effect on which records land in which window.11 .trigger(processingTime="30 seconds")12 .outputMode("update") # results are revised, so the sink must upsert13 .option("checkpointLocation", "s3://.../chk/revenue_5m")14 .start())The distinction the API makes explicit: window(occurred_at, ...) is assignment and trigger(processingTime=...) is emission. Reaching for a processing-time *window* to get lower latency confuses the two and gives up reproducibility to solve a problem the trigger already solved.
What processing time is genuinely the right answer for
The rule that survives contact with real systems is a question about audience: is this number about the pipeline or about the business? Operational questions belong on the processor's clock and become harder and worse if you try to put them on event time — a lag metric derived from event time would stall exactly when the pipeline stalls, which is the moment you need it.
The decision below names the cases. Notice that two of the four are not really about windowing at all: rate limiting and timeouts are processing-time decisions embedded inside jobs that are otherwise entirely event-time, and there is nothing inconsistent about that.
The one row to be careful with is the last. "We do not have an event time" is a real situation — some sources genuinely do not carry one — and processing time is then the only option available. The correct response is to record it as a known limitation of that dataset rather than to let it pass as an ordinary metric (Dataset Documentation).
Is this number a statement about the world, or about the machinery that carries it?
when The question is "is the pipeline keeping up", and the answer must be available during an incident without waiting for anything.
cost None worth mentioning. Event time would be actively wrong here, and would stall exactly when the signal is needed most.
when You want low-latency provisional results without giving up reproducible assignment.
cost The sink must accept updates, and consumers must tolerate a number that changes until the window is complete (Upserts and Merges).
when The decision protects a resource in the present — a downstream API, a connection pool, a memory budget.
cost None. These are inherently about now, and no event timestamp is relevant to them (Rate Limiting).
when Almost never. Only when the source genuinely carries no event time and the alternative is no metric at all.
cost Unreproducible history, backfills that produce spikes, and disagreement between any two consumers. Record it as a documented limitation of the dataset, not as a normal metric.
when The number will be compared across periods, reported, audited, or corrected by a backfill — which is most business metrics.
cost Latency to completeness, state to hold windows open, a watermark that can stall, and an updatable sink (Event Time).
The replay test
There is one test that settles whether a job has a hidden processing-time dependency, and it takes an afternoon: reprocess a closed historical range and compare the output with what the job produced originally. If they differ, something in the job depends on when it ran.
The reason to run it deliberately rather than trust a reading of the code is that processing-time dependence hides in places that do not look like windowing. A now() used to compute an age. A lookup against a dimension table that has since changed. A default applied when a field was missing. An ordering that depends on arrival. Each of these makes replay produce a different answer, and none of them appears in the window definition (Deterministic Replay: Making the Schedule Reproducible).
Run the test with a deliberately different shape — different parallelism, a consumer paused and resumed mid-range, records delivered in a different order — because a quiet replay on identical infrastructure can accidentally agree. The failure you are hunting is one that only appears when conditions differ, which is exactly the condition under which you will actually need to replay.
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Reprocess a closed range and diff against the original output | The job is a deterministic function of its input. | Processing-time windows, now() in a transformation, arrival-order-dependent logic, defaults applied from the clock. | Dependence that happens to produce the same answer under identical conditions — run it with different parallelism and a paused consumer to expose those. |
| Compare the streaming output for a closed period with a batch recomputation from raw events | Two independent implementations of the metric agree. | Late records counted in the wrong period, and any systematic shift caused by transport delay. | A definition error shared by both implementations, and anything about the current open period (Reconciliation). |
Alert on processing_time − ingestion_time exceeding its normal range | The pipeline's own contribution to latency is stable. | A slow operator, a stalled instance, a backlog forming — the operational reasons processing time drifts away from event time. | A steady systematic delay, since the check is against the normal range and a delay that was always there is the normal range (Latency Budgets: Spending 200 Milliseconds on Purpose). |
| Monitor clock offset across the job's instances | Parallel instances agree about what time it is. | Skew and stepped corrections, which produce split windows and occasionally negative durations. | Everything about correctness of the *data*; this is purely an infrastructure check, and a perfectly synchronised cluster can still be counting records in the wrong period. |
The first row is the one that matters. It is cheap, it is almost never run, and it is the only check that directly tests the property processing time gives up.
How to build it
Most important first.
- Use processing time for operational signals and never for business metrics. Written as a rule: if the number would appear in a report about the business, it must be on event time; if it appears on a dashboard about the pipeline, processing time is correct (Pipeline Metrics).
- Use it for triggers, not for assignment. Emitting early and often on a processing-time trigger while windows are assigned on event time gives you low latency and reproducibility at once, at the cost of a sink that can accept updates (Upserts and Merges).
- Use it for rate limiting, throttling, backpressure decisions and timeouts, all of which are genuinely about the machine and none of which can wait for a watermark (Rate Limiting).
- Record it as a field alongside event time and ingestion time so that lateness and processing delay can be measured after the fact rather than only observed live (Ingestion Time).
- When a processing-time answer is published, label it. A chart of "events per minute" derived from processing time is a chart about your pipeline, and consumers will read it as a chart about customers unless told otherwise (Dataset Documentation).
- Never mix the two in one aggregate. A job that assigns some records by event time and falls back to processing time when the field is missing produces a metric that is a blend of two definitions with no way to separate them (Semantic Changes).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Availability: the clock is always there, always advances, never null, never out of order. This is the only time domain with no data-quality dependency at all.
- Liveness: windows always close, so output never stalls waiting for progress. A processing-time job cannot exhibit the stalled-watermark failure that is the most confusing incident in this module (Watermarks).
- What is explicitly not guaranteed: reproducibility, comparability across periods, correctness after a replay, agreement between two consumers of the same stream, or even agreement between two parallel instances of the same operator.
- Also not guaranteed: that the clock is right. Machine clocks drift and are corrected by time daemons, and a correction that steps the clock backwards can produce a window that closes before it opens.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that matters is a replay test: reprocess a closed historical range and compare the output with the original run. If the numbers differ, the job depends on processing time somewhere, and the difference tells you how much (Reconciliation).
- It misses processing-time dependence that happens to produce the same answer on a quiet day, which is most of them — run the test with a deliberately different parallelism or a paused-and-resumed consumer to expose it.
- Also monitor clock skew across the instances of a job. It is cheap, it is rarely done, and a stepped clock produces window behaviour that looks like a code bug (Correlation IDs: Turning Lines Into a Story).
- Processing time gives the lowest possible latency, because nothing ever waits. A window closes when the clock says so, and whatever arrived is the answer.
- That latency is bought with an *undefined completeness*. The result covers whatever happened to arrive, and there is no bound on what is missing — which is a worse position than a stated lateness allowance, because it cannot be reasoned about at all.
- For operational questions this is the right trade and it is not a compromise: "how many records did we handle in the last minute" is complete by definition, because it is a question about handling.
- Nothing about the schema changes here — this is a physical concern, and saying so explicitly is worth doing: a processing-time job has no time field to evolve, which is part of why it feels simpler than it is.
- What does change is the *meaning* of the output whenever the deployment changes. New parallelism, a different batch interval or a restart alters window membership, so a metric can shift because of an infrastructure change with no code change at all ("What Changed?" — Deploy Markers and the Invisible Deploys).
- Migrating a job from processing time to event time is a metric redefinition, not a refactor. The two produce different numbers for the same period, and the cutover must be dated, documented and communicated to every consumer (The Metrics Layer).
- There is no recovery for a processing-time aggregate. Reprocessing produces a different answer, so the original cannot be reconstructed and the corrected one cannot be compared with it.
- The only mitigation is to have stored the raw events with their event times, so that a *different*, event-time computation can be run over history to produce a comparable series. This is why raw retention matters even for pipelines that do not currently use event time (Keeping Raw History: The Recovery Position and the Liability).
- For operational uses this is not a problem worth solving: nobody backfills a throughput chart, and if they did the answer would be meaningless.
What can go wrong
- A clock stepped backwards by a time daemon correction, so a window closes before it opens or a duration comes out negative.
- Clock skew between parallel instances, so records that logically belong together are split across two windows depending on which instance handled them.
- A burst caused by upstream catch-up counted as a real burst, triggering an autoscaler or an alert that is responding to transport rather than to demand (Autoscaling Lag: The Gap Where the Outage Lives).
- A processing-time metric published to a business audience, where it is read as a statement about customer behaviour and used in a decision (Stale Dashboards).
- The mitigation failing: switching to event time to fix all of this introduces the stalled-watermark failure mode, which is harder to diagnose than anything processing time can do (Watermarks).
- "Processing time is a simpler approximation of event time." It is a different measurement of a different thing. It measures your pipeline, and it approximates event time only to the extent that your pipeline is instantaneous, which under load is exactly when it is not.
- "There is no late data with processing time." There is no *concept* of late data. The late records are still there; they have simply been counted in the wrong period, silently, with nothing to detect (Late Events).
- "Our processing-time numbers match the event-time ones, so it does not matter." They match while the pipeline is healthy. They diverge precisely during a stall, a retry storm or a backfill — the moments when someone is looking at the chart to make a decision.
- "Machine clocks are synchronised, so processing time is consistent across instances." Time daemons keep clocks close, not identical, and corrections can step a clock in either direction. Do not build a boundary that depends on two machines agreeing (Correlation IDs: Turning Lines Into a Story).
Operating it
- Records processed per unit of processing time, per operator — this is throughput, and processing time is the right and only clock for it (Throughput: Requests, Packets and Bytes per Second).
- Processing delay:
processing_time − ingestion_timeper record. This isolates the pipeline's own contribution to end-to-end latency from the producer's (Latency Budgets: Spending 200 Milliseconds on Purpose). - Clock offset between instances, sampled periodically. Cheap, almost never collected, and the explanation for a whole class of confusing boundary behaviour.
- At 10x volume nothing changes — processing time has no cardinality term and no retention term.
- At 10x parallelism, clock skew between instances becomes more visible, and boundary effects that were negligible at low parallelism start to matter for short windows.
- At 100x, the divergence between processing time and event time widens because transport queues deepen under load. The processing-time chart increasingly describes your backlog rather than your traffic — exactly when someone is most likely to be reading it during an incident (Depth Is Not an Emergency; Age Is).
- The cheapest of the three time domains: no watermark tracking, no windows held open for stragglers, no state retained beyond the current window. State is bounded by the window size alone (Streaming State).
- This is a genuine and often-overlooked advantage. A processing-time job with a one-minute window holds a minute of state; the event-time equivalent with an hour of allowed lateness holds sixty times that for the same data.
- The hidden cost is elsewhere: any metric that later has to be recomputed on event time requires the raw events to have been retained, and if they were not, the cost is that the question cannot be answered at all (The Raw Landing Zone).
- Processing time buys simplicity, low latency, small state and guaranteed liveness, and costs reproducibility entirely. It is the right trade for operational signals and the wrong one for anything a person will compare against last month.
- Using it as a *trigger* over event-time assignment gets most of the latency benefit with none of the correctness cost, and pays for it with an updatable sink and consumers who must tolerate a number that changes (Windows).
- The simplicity is real but seductive: no field to validate and no watermark to stall means fewer incidents, and the one failure it does have — unreproducible history — surfaces months later during an audit rather than immediately.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALEvery processing system has a local clock and every one of them faces the same trade: the clock is always available and always describes the processor rather than the world. This is as true of a nightly batch job stamping
loaded_atas of a stream processor. - ENGINE-SPECIFICSpark Structured Streaming exposes a processing-time trigger that controls how often a micro-batch runs, which is a scheduling knob rather than a windowing one; Flink offers processing-time windows and timers as first-class alternatives to their event-time counterparts. Confusing the trigger with the window assignment is the common error in the Spark case specifically.
- SIMPLIFIEDTreated here as one clock per job. In reality each parallel instance has its own, and they differ by however far apart the time daemons have let them drift — which matters only for short windows and high parallelism, but matters absolutely when it does.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why two machines cannot simply agree on the time, what clock synchronisation actually promises, and why logical clocks exist as an alternative to trusting a wall clock at all.
- — Observability & Performance owns the operational metrics processing time is the correct clock for — this lesson only argues which clock they belong on, not how to interpret them.