StreamingGENERALSOURCE-SPECIFICENGINE-SPECIFIC

Event Time

The time the thing actually happened, carried in the record itself — the only clock that makes a result reproducible when you process the same data again next year.

What actually happens How to build it Can I trust it?

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

If I reprocess this stream in six months, will I get the same answer — and which timestamp decides that?

Who needs this

Anyone who compares a period against itself. Finance closing a month, an analyst comparing this Tuesday with last Tuesday, an experiment measuring a treatment window, a regulator asking what the number was on a given date. All of them are asking about *when it happened*, and none of them care when your job ran.

What one row is

One event carrying its own timestamp, assigned as close to the real-world action as possible. The grain question here is which action the timestamp names: the click, the request arriving at the server, the row being committed, or the record being serialised — those are four different times and only one of them is the event.

The obvious build

Use the timestamp the pipeline has closest to hand. The record has an ingested_at from the loader, or the processor uses now() when it handles the record, and both are always present, always monotonic and never null.

Why it breaks

A mobile client was offline for three hours and uploads a batch of events on reconnect. By arrival, all of them belong to this afternoon; by what happened, they belong to this morning. Every hourly count is wrong in two directions at once (Late Events).

How it breaks with real data

A mobile client was offline for three hours and uploads a batch of events on reconnect. By arrival, all of them belong to this afternoon; by what happened, they belong to this morning. Every hourly count is wrong in two directions at once (Late Events).
A backfill replays six months of history through the same job. With arrival time, every one of those events is dated today, and half a year of activity appears as one enormous spike (Backfills).
The upstream connector stalls for twenty minutes and catches up. The arrival-time chart shows a trough and then a matching spike; the underlying activity was perfectly flat. Somebody will investigate the spike (Stale Dashboards).
Two consumers of the same topic compute the same daily metric and disagree, because they processed the boundary records at different moments and each one used its own clock (Two Dashboards, Two Numbers).
A reprocess to fix a transformation bug produces different daily totals than the original run, so there is no way to say which one was right — and no way to prove the fix worked (Reprocessing vs Retrying).

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

Event time is a field in the payload, assigned by whoever observed the event. Because it travels with the record, it is immutable and identical on every subsequent read — which is precisely why it makes a computation reproducible (Event vs Snapshot Modeling).
A stream processor configured for event time assigns each record to a window by that field, regardless of when the record arrived. That single rule is what makes an event that happened at 10:00 and arrived at 10:07 land in the 10:00–10:05 window rather than the 10:05–10:10 one (Late Events).
Because arrival order and event order differ, event-time processing needs a notion of progress that is separate from the input position. That is the watermark: an estimate of how far event time has advanced, derived from the event times already seen (Watermarks).
Event time is not monotonic in the stream. Records arrive with timestamps that go backwards, and that is normal rather than exceptional — a system that assumes non-decreasing timestamps is assuming a property no distributed producer provides (Ordering Guarantees: Four Levels, Four Prices).
The clock that assigns event time is usually not yours. A device clock can be wrong by hours, deliberately or by accident; a server clock is better but still not synchronised with your processor's. This is a data quality problem, not a configuration one (The Dimensions of Data Quality).
Where the timestamp is assigned matters as much as which clock assigned it. A timestamp taken at serialisation rather than at the action folds the producer's own buffering delay into "when it happened", which quietly makes event time part-arrival-time.

The timestamp that belongs to the event

There are four plausible timestamps around any event and they are routinely conflated. The action happened at one moment; the producer serialised a record at another; the platform appended it to a log at a third; a processor handled it at a fourth. Event time is the first of those, and it is the only one that does not change when you process the record again.

That immutability is the whole argument. Ingestion time is fixed too, but it describes your platform rather than the world. Processing time changes on every run, which means any aggregate that uses it is a statement about when the job ran rather than about what happened — and such an aggregate cannot be reproduced, compared across periods, or corrected by a backfill.

The SQL below is the same aggregate written twice. Both run without error, both produce a plausible chart, and they disagree by exactly the amount of transport delay in the pipeline. Only one of them gives the same answer when the job is re-run next year.

The same metric, dated two different ways

1-- Arrival-time bucketing: this asks "when did we learn about it?"
2-- The answer changes on every replay, and a 20-minute connector stall
3-- shows up as a trough followed by a spike that never happened.
4select date_trunc('hour', ingested_at) as bucket,
5       count(*)                        as events,
6       sum(amount_minor)               as amount
7from   raw_events
8group  by 1;
9 
10-- Event-time bucketing: this asks "when did it happen?"
11-- Stable across replays, correct after a backfill, comparable
12-- between periods -- and incomplete for the most recent bucket,
13-- which is the honest cost of asking the right question.
14select date_trunc('hour', occurred_at) as bucket,
15       count(*)                        as events,
16       sum(amount_minor)               as amount
17from   raw_events
18where  occurred_at >= timestamp '2026-08-01 00:00:00'
19  and  occurred_at <  timestamp '2026-09-01 00:00:00'
20group  by 1;
21 
22-- The diagnostic that should exist in every pipeline: your own
23-- lateness distribution, which is the input to every windowing
24-- decision you will make later.
25select source,
26       percentile_cont(0.50) within group (order by lateness_s) as p50_s,
27       percentile_cont(0.99) within group (order by lateness_s) as p99_s,
28       max(lateness_s)                                          as worst_s
29from  (select source,
30              extract(epoch from (ingested_at - occurred_at)) as lateness_s
31       from   raw_events
32       where  ingested_at >= now() - interval '7 days')
33group by source;

The third query is the one that is usually missing. Window size, watermark delay and lateness allowance are all choices about that distribution, and without it every one of them is a guess dressed as a configuration value.

Reproducibility is the property you are actually buying

Ask what happens when you run the job again. That is the question that separates event time from every alternative, and it is the question that matters most in a domain whose recovery mechanism is "reprocess it".

A job that assigns records to windows by event time is a pure function of the records. Replay them in any order, on any day, at any speed, and every window contains the same set of events and produces the same aggregate. A backfill can therefore target a specific historical range and overwrite exactly it.

A job that assigns by processing time is a function of the records *and the schedule*. Replaying six months of history through it produces one enormous bucket dated today. There is no configuration that fixes this after the fact, because the information needed to place each record correctly was never used and, in an arrival-time-only pipeline, was often never recorded.

This is also the deeper reason event time and idempotent sinks belong together. Event time makes the recomputation correct; the sink decides whether publishing that recomputation twice is safe. Getting one without the other produces a pipeline that computes the right answer and then stores it twice.

What happens when you reprocess

Windows assigned by when the record was processed

The job buckets records by `now()` at handling time. A replay of six months of history writes one giant bucket dated today; a twenty-minute upstream stall produces a trough and a spike; two consumers of the same topic disagree because they ran at different moments.

Windows assigned by the event's own timestamp

The job buckets by `occurred_at` from the payload. A replay reproduces the original buckets exactly; a stall produces a delayed but correctly-dated result; two consumers of the same topic agree because they used the same field rather than their own clocks.

Assignment by processing time makes the output a function of the schedule, and the schedule is not in the data. Nothing recorded in the pipeline can reconstruct which period a record belonged to, so the error is not merely present but permanently unrecoverable — which is a different and worse category than being wrong.

Where event time comes from, and how it lies

SOURCE-SPECIFICThese failure rates differ enormously by source: a CDC stream carrying a database commit timestamp almost never exhibits rows one to three, while a consumer mobile fleet exhibits all five regularly. Set validation thresholds per source rather than globally, or the strict setting needed for mobile will reject legitimate CDC data.

Event time is data, which means it has a source, a quality profile and failure modes — and unlike most fields, an error in it does not produce a wrong value in one column but a wrong *placement* of the whole record.

The trap that costs the most is the future timestamp. Because a watermark is derived from the highest event time seen, a single record from 2087 advances event time past everything real, closes every open window, and causes every genuine record that follows to be classified as late. One malformed record can discard a partition's worth of data with no error raised anywhere (Watermarks).

The trap that is hardest to detect is the systematically wrong clock. A fleet of devices offset by the same amount produces a tight, plausible lateness distribution and perfectly misattributed data. Nothing internal to the pipeline can find it; only a comparison against a source with an independent clock will.

Ways an event-time field misleads

Trigger	Symptom	Cause	Response
A client clock is wrong (drift, wrong timezone, user-set)	A share of records lands in windows hours away from where the activity really was; totals are right and the daily shape is wrong.	The event time was assigned by a clock you do not control and cannot verify.	Prefer a server-assigned timestamp where the number has commercial consequences, keep the client one as a separate field, and monitor the difference between them (Every Input Surface).
A timestamp in the future	Output stops or collapses; a large share of subsequent records is counted as late and dropped.	The watermark is derived from the maximum event time seen, so one bad record moves it past all real data (Watermarks).	Reject any event time beyond a small tolerance ahead of ingestion time, before it reaches the watermark generator. This is the single highest-value validation in the module.
The event-time field is missing and defaulted to `now()`	Part of the stream behaves like arrival-time processing; backfills produce spikes for that subset only.	A silent default put a subset of data on different semantics from the rest, and nothing records which subset.	Never default it. Reject to a dead-letter path with the raw payload attached, and count rejections per source (A Dead-Letter Queue Is a Workflow, Not a Bin).
Timestamp assigned at serialisation, not at the action	The lateness distribution looks tight and correct, and the numbers are still slightly misdated during producer stalls.	Producer buffering has been folded into "when it happened", so event time is quietly part arrival time.	Specify the assignment point in the event contract and verify it during producer review — this cannot be detected downstream (Data Contracts).
A local timestamp with no zone	Daily totals are correct except around midnight, and shift by an hour twice a year.	The producer wrote local wall-clock time and the consumer read it as UTC.	Require UTC in the contract with the local offset carried as a separate field, and validate at the boundary rather than in the model (Nullability & Defaults).

How to build it

Most important first.

Make event time an explicit, mandatory, documented field of the event contract, named for the action rather than for the transport: occurred_at, not timestamp (Data Contracts).
Assign it as close to the real-world action as possible, and record it in UTC with an offset preserved separately if local time matters to the consumer. A timestamp without a zone is a bug that surfaces twice a year (Nullability & Defaults).
Carry all three times through the pipeline: event time, ingestion time and processing time. They cost a handful of bytes and they are the only way to measure your own lateness distribution, which is the input to every windowing decision you will make (Ingestion Time).
Never default a missing event time to the current time. Route the record to a rejection path or stamp it with a clearly invalid marker — a silent default makes a subset of your data run on different semantics from the rest (Data Tests).
Reject or quarantine timestamps outside a plausible range in both directions. A record from 1970 lands in a window nobody looks at; a record from 2087 advances the watermark past everything real and causes mass dropping (Watermarks).
Measure the distribution of ingestion_time − event_time continuously. That distribution *is* your lateness, and choosing a window or a watermark delay without it is guessing (Percentiles: Which One, and How Many Users Is That?).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

Reproducibility: a computation keyed on event time gives the same answer on every replay of the same records, in any order, at any later date. This is the only guarantee event time provides and it is the whole reason to use it.
Correct attribution: a record is counted in the period it happened in, regardless of transport delays, retries or backfills.
What is explicitly not guaranteed: that the timestamp is *accurate*. Event time is only as trustworthy as the clock that stamped it, and a wrong clock produces confidently misattributed data with no error anywhere (Data Quality).
It also does not guarantee completeness. Knowing which period an event belongs to says nothing about whether all of that period's events have arrived — that question is the watermark's, and it is answered with an estimate rather than a fact.

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

Monitor the distribution of ingestion_time − event_time per source, and alert on shifts in its shape rather than on individual outliers. A source whose lateness distribution changed is a source whose windowing configuration is now wrong (Distribution Tests).
Add a plausibility check in both directions — timestamps far in the past and any timestamp in the future — and count rather than silently accept. Future timestamps are the more dangerous of the two because of what they do to the watermark.
These miss the systematic case: a device fleet whose clocks are all wrong by the same offset produces a perfectly tight, perfectly plausible, perfectly misattributed distribution. Only a comparison against a second source with an independent clock catches that (Reconciliation).

Freshness

Event time does not make data fresher; it makes it *correctly dated*. A result keyed on event time may need to wait longer before it is complete, because completeness now depends on stragglers rather than on the clock.
It changes what a consumer can ask for. With event time, "revenue for 10:00–10:05" is a question with a stable answer that converges; with arrival time it is a question whose answer depends on when you asked, and nobody can tell you when it stopped changing.
The freshness cost is explicit and bounded: it is exactly the lateness allowance you chose. That is a much better position than an unbounded, invisible error, which is what arrival-time processing gives you instead.

When the schema or meaning changes

Changing which field is event time is a change to every historical comparison, and it is invisible in the schema because the field name and type are unchanged. Version it explicitly and record which range of data used which definition (Semantic Changes).
Adding a more accurate event-time field alongside the old one is the safe migration: dual-write both, compare on a window of overlap, then move the job's assignment to the new field at a documented boundary.
A producer that starts stamping at a different point in its own code — at serialisation instead of at the click — has changed the meaning of the field without changing the contract. Only monitoring the lateness distribution will show it (Contract Enforcement).

How to re-run this safely

Event time is what makes recovery possible at all: replaying a range of the log recomputes exactly the same windows, so a corrected job produces a directly comparable result (Replay from the Log).
A backfill through an event-time job writes into the historical windows the data belongs to, so it can overwrite a range precisely rather than appending a spike to today (Planning a Backfill).
The corresponding hazard: a replay re-emits results for old windows, and any sink that appends rather than upserts now holds two answers for the same period. Event time makes the recomputation correct; only an idempotent sink makes the *publish* correct (Upserts and Merges).

What can go wrong

Failure modes

A device clock that is wrong, so events are attributed to a period they did not happen in — with no error, no rejection and no way to tell from the data alone.
A future timestamp advancing the watermark far past real time, after which every genuine record looks late and is dropped. One malformed record can silently discard a partition's worth of data (Watermarks).
A missing event-time field defaulted to now() somewhere in the ingestion path, putting part of the stream on arrival-time semantics invisibly.
Timezone handling that differs between the producer and the consumer, producing a metric that is right except at the boundaries — which is where daily numbers are made.
The mitigation failing: a plausibility filter that rejects a legitimate range, such as an IoT fleet that genuinely buffers for days, so the correctness control becomes the data loss (Late Events).

Misreads

"Event time is just a timestamp column." It is the semantics of the entire job. Choosing it decides window assignment, watermark behaviour, lateness policy, replay determinism and whether a backfill is possible (Processing Time).
"Our events arrive in order, so we do not need event time." They arrive in order today, on one partition, at this volume. Ordering across partitions has never existed, and the day a producer retries is the day it stops being true (Topics and Partitions).
"Use the database's updated_at as event time." That column is assigned when the row was written, not when the thing happened, and in many systems it is assigned before commit — so an ordering by it disagrees with the commit order the log actually has (CDC Ordering and Transaction Boundaries).
"Event time makes the numbers right." It makes them *reproducible and correctly attributed*. If the clock that produced it is wrong, event time faithfully preserves the wrong answer forever.

Operating it

How you see it in production

A histogram of ingestion_time − event_time per source, over time. This single chart drives the choice of window size, watermark delay and lateness allowance, and almost nobody has it (Histograms: A Distribution You Can Afford to Keep Forever).
Count of records with a missing, unparseable, future or implausibly old event time, per source, as four separate counters — they have four different causes.
The gap between the current watermark and wall clock, which converts "event time is stalled" into a signal instead of an absence of output.

What changes at 10x and 100x

At 10x volume, nothing about event time changes — it is a per-record field with no aggregation behaviour of its own.
At 10x source count, the lateness distribution becomes multi-modal: a web source is seconds late, a mobile fleet is minutes to hours late, and a partner feed is a day late. A single watermark strategy across all of them is wrong for at least two (Watermarks).
At 100x, out-of-order arrival becomes the normal case rather than the exception, because more producers means more independent clocks and more independent delays. Systems that were accidentally in-order at small scale stop being so.

What drives cost here

Event time costs almost nothing to carry — a field per record — and costs real money in *state*, because holding windows open long enough to accept late data is what keeps them in memory (Streaming State).
It also costs a second write path: any result that can be revised means the sink must support updates, which is more expensive than an append-only sink and is the usual reason people reach for arrival time instead.
The saving is on the other side: correct attribution means a backfill overwrites a bounded range rather than requiring a full reprocess to untangle misdated records, which is often the largest reprocessing cost a platform has (Reprocessing vs Retrying).

What this approach costs

Event time buys reproducibility and correct attribution, and costs latency to completeness, state to hold open windows, and an updatable sink. Those are real costs and they are the reason arrival time keeps being chosen by accident.
Assigning event time at the client is the most accurate and the least trustworthy; assigning it at the server is less accurate and much harder to falsify. For anything with commercial consequences, the server timestamp is usually the right trade (Every Input Surface).
Carrying three timestamps costs payload on every record and buys you the ability to diagnose your own lateness. It is the cheapest observability investment in this module and it still costs bytes on every hop.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALThe distinction between when something happened and when a system learned about it exists in every event-carrying system, from a broker to a webhook to a nightly file drop. What varies is only whether the platform gives you a mechanism to act on it.
SOURCE-SPECIFICA CDC stream can carry the source transaction's commit time, which is an unusually trustworthy event time; a mobile SDK carries a device clock that can be wrong by hours and is under the user's control; a partner file drop may carry no per-record time at all, only a filename. The same watermark strategy cannot be correct for all three.
ENGINE-SPECIFICFlink attaches a timestamp and a watermark strategy per source and processes record-at-a-time; Spark Structured Streaming derives its watermark per micro-batch from a declared event-time column, so its progression is coarser. The semantics are the same; the granularity at which they advance is not.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Databasewrite-ahead-logging mvcc transactions-and-acid

Backenduntrusted-input-sources dead-letter-queues event-naming commands-vs-events

API Designwebhook-ordering consistency-expectations

Observabilityhistograms percentiles

Concurrencyordering-guarantees happens-before determinism

Distributed Systemsevent-time-vs-processing-time

Domains that do not exist yet

— Distributed Systems owns why there is no shared global clock to appeal to, what logical clocks give you instead, and why "the time it happened" is a claim by one machine rather than a fact about the world.