Point-in-Time Correctness
A training example at time T may only use information that existed at T. The as-of join is how you build that, and the offline store exists to make it cheap.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Our training set joins each event to the customer's "current" features. Which of those features existed when the event happened, and how do I build the join so the question cannot arise?
A lending team built a default model by joining each loan application to the customer table and got a validation number the risk committee called "too good to be true". The customer table holds one row per customer with today's values: current balance, current delinquency count, current account status.
Join applications to customers on customer_id. The customer table has the features; the application table has the timestamp and, via the loan, the label. One query, one training set.
An applicant who defaulted eighteen months ago has delinquencies_12m = 3 and status = closed in today's row. At application time both were zero. The model learns that closed accounts default, which is the label wearing a different name.
- An applicant who defaulted eighteen months ago has
delinquencies_12m = 3andstatus = closedin today's row. At application time both were zero. The model learns that closed accounts default, which is the label wearing a different name. - Validation is drawn from the same joined table, so the leak is present in both folds and the offline number is spectacular. It is measuring how well "already defaulted" predicts "will default".
- In production the application arrives with the applicant's actual current state — no delinquencies yet — and the model, which learned to lean on a feature that is only informative *after* the outcome, scores everyone as safe.
- Nobody wrote a bug. The join is correct SQL. The table was simply the wrong shape for a question about the past.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an applicant defaults within twelve months of origination. The label is a delinquency event that occurs, if at all, months after the application.
- The prediction must exist at application time, so the model may only use what the bank knew at that moment. Anything learned afterwards is the future.
- One example is one application with the applicant's features as of the application date. The warehouse holds an
applicationstable with timestamps and acustomerstable that is overwritten in place — the row shows the state today. - Feature aggregates like
delinquencies_12mare computed nightly by a job that reads the current customer row. There is no history of what the row said on any past date. - Labels are derived from a
loan_statustable that is also overwritten; the training query reads it as of now.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Every feature value has a time at which it became known. A training example is honest only if every feature it carries was known at or before the example's timestamp. This is the definition of point-in-time correctness, and it is a property of the *join*, not of the model.
- The join that enforces it is an as-of join: for each example with entity
eand timestampt, take the feature row forewith the largestfeature_ts <= t. That requires the feature source to keep history — every value with the time it became valid — instead of overwriting a current row. - Overwriting tables collapse that history. Once the row shows today's value, there is no query that recovers yesterday's, and any join to it is a join to the future for every example older than the last update.
- Label timing follows the same rule in reverse: the label must be observed *after* the feature snapshot, at the horizon the decision cares about. A label read "as of now" for a recent example is not yet mature (Ground-Truth Delay).
The join that leaks and the join that does not
The leaky query is the one everybody writes first, because the schema invites it: a current-state table with one row per customer is the shape of the question "what do we know about this customer?" and the wrong shape for "what did we know on this date?"
The correct query needs a history table and one extra predicate. For every application, take the customer feature row with the latest valid_from that is at or before the application time. Everything downstream — validation, promotion, production — is honest or not depending on that predicate.
1-- customer_features_history(customer_id, valid_from, delinquencies_12m, balance, ...)2-- one row per customer per change; valid_from is when the value became READABLE3SELECT a.application_id,4 a.applied_at,5 f.delinquencies_12m,6 f.balance,7 l.defaulted_within_12m -- matured label, observed after applied_at + 12m8FROM applications a9JOIN LATERAL (10 SELECT *11 FROM customer_features_history h12 WHERE h.customer_id = a.customer_id13 AND h.valid_from <= a.applied_at -- the point-in-time predicate14 ORDER BY h.valid_from DESC15 LIMIT 116) f ON true17JOIN loan_outcomes l ON l.application_id = a.application_id18WHERE a.applied_at <= now() - interval '12 months'; -- only matured labelsTwo things carry the correctness: valid_from <= applied_at, and valid_from meaning "readable", not "describes". A nightly job that finishes at 03:00 should stamp its rows 03:00, not midnight, or the training set sees values three hours before serving could have.
Why the offline store exists
The as-of join is conceptually simple and operationally expensive: a lateral subquery per example over a history table that grows with every change. Feature stores exist in large part to make this one join cheap — the offline store is a history table with the timestamp semantics already decided and an API that requires a timestamp per row.
That is also why Feature Stores are optional. If the warehouse already keeps snapshot history and the team can write the join, the store adds convenience, not correctness.
looks like A sensible risk feature with a clear name, joined on customer id from the table every analyst uses.
why it leaks The row is overwritten in place, so for a historical application it holds delinquencies counted *after* the application — including the default that is the label.
fix Join to a history table with valid_from <= applied_at; anchor every window-based aggregate to the example timestamp rather than to the job date.
What must stay true after the join is fixed
Point-in-time correctness is not achieved once; it is an assumption about every future rebuild of the training set. The history table must keep its readable-time semantics, backfills must not rewrite it, and late events must not be allowed to make the past better informed than it was.
This connects directly to Temporal Leakage — the as-of join is the mechanical enforcement of the rule that lesson states — and to Time-Based Split, which decides the holdout but cannot repair a leaky row.
For every example, each feature value in the training row was readable by a serving request at that example's timestamp.
holds when Feature sources are append-only with readable-time stamps; the join predicate is valid_from <= event_ts; backfills add rows with the time the recomputed value would have become readable, or are excluded from training.
breaks when A source table is overwritten in place; a backfill rewrites history with today's code; late events are inserted with their event time; a window aggregate is anchored to the job date.
event_ts - valid_from >= 0 across the training set; a synthetic-entity test on the join; a validation gap between the joined set and a strictly-later window that a temporally honest join should not show.respond Rebuild the training set from history with the corrected join; do not tune the model on the leaky set and do not report its metric.
How to build it
Most important first.
- Keep feature history as an append-only table keyed by
(entity, feature_ts)— a snapshot table, a slowly-changing dimension, or a feature store's offline store. Never train against a table that is overwritten in place. - Build the training set with an explicit as-of join and make the
feature_ts <= event_tspredicate impossible to omit — a shared query template, a view, or a store API that requires a timestamp per row. - Include a positive margin for pipeline latency: if the nightly job that computed a feature ran at 03:00, the value was not *available* to a serving request at 01:00 even though it describes events before midnight. Join on the time the value became readable, not the time it describes.
- Test the join with a synthetic entity whose feature changes at a known time, and assert that examples before the change see the old value (Data & Feature Tests).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The gap between the joined training set's validation metric and the metric on a strictly-later time window. A large gap with a temporally honest join indicates leakage; this is the leakage simulator's "future timestamp" control at
/ml/leakagemade real. - For each feature, the distribution of
event_ts - feature_tsin the training set. A feature whose lag is always zero or negative is joined to the future. - Validation metric on the leaky join is not a measurement of anything about the product; do not report it, even as a baseline.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The feature history table records the time each value became readable, not merely the time it describes, and that history is never rewritten in place.
- The training join takes the latest feature row at or before each example's timestamp, and a test with a synthetic entity proves it.
- Labels in the training set are matured to the decision horizon and were observed after the feature snapshot, not read from a current-status table.
- Offline: compare validation on the as-of-joined set with validation on a strictly later window. The two should be close; a large gap is a leak somewhere in a feature definition.
- Online: for a sample of served predictions, rebuild the feature row from history at the request timestamp and check it equals what was served. A mismatch is either a timing skew or a rewritten history.
- Over time: assert per feature that
min(event_ts - readable_ts) >= 0in every rebuilt training set, and fail the pipeline otherwise.
What can go wrong
- The as-of join is correct and the feature *definition* looks backwards past
t:delinquencies_12mcomputed by a job with access to the whole history reads events aftertbecause the window was anchored to the job date, not to the example date. - A backfill rewrites the snapshot table with values recomputed by today's code, quietly making history point-in-time incorrect for everything before the backfill.
- Late-arriving events are inserted into history with their original timestamps, so a rebuilt training set contains information that was not readable at
t, and the model is a little better offline than it can be online (Late-Arriving Data).
- History tables are larger than current-row tables by the number of changes, and an as-of join over them is slower than an equality join. The offline store's main job is to make that acceptable.
- Joining on "readable time" rather than "event time" makes features look staler in training than they conceptually are, which is honest but reduces the offline number and disappoints people.
- Every feature source that does not keep history has to be rebuilt to do so, or excluded, and the excluded ones are often the ones the business most wants.
- "Our features are computed from events with timestamps, so they are point-in-time correct." The events have timestamps; the aggregate was computed once, over all of them, and stored as a current value. The timestamps were used and then discarded.
- "We can fix it by dropping the leaky column." The leak is a join shape. Drop
statusand the same join leaks throughcurrent_balance,last_payment_dateand everything else in the current row. - "A time-based split solves this." The split decides which examples are held out. It does not stop each example from carrying features from its own future.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a training example may only use information available at its timestamp holds for every supervised task and every model family; the only systems exempt are those with no time dimension at all, which in production are rare.
- DATA-SPECIFICThe failure needs a feature source that is overwritten in place — a "current customer" table, a mutable status column. Event-sourced or append-only sources make the correct join natural and the leaky one hard to write.
- SIMPLIFIEDThe SQL shown assumes one feature table with one timestamp; real systems have several tables with different readable-time semantics, and any illustrative metrics in this lesson are for the shape of the argument, not measurements.
Where the depth lives
This domain teaches the model and hands the rest off by name.