FeaturesGENERALSIMULATEDCONTESTED

Target Encoding

Replace a category with the mean label for that category. Powerful on high-cardinality features, and a leak unless the rate for each row is computed without that row, out of fold, with a smoothed prior.

Target & dataWhat to measureWhat must stay true▶ Leakage Simulator

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

Encoding a category as its label rate is the most effective trick for high-cardinality features. Why does the obvious version leak, and what does the out-of-fold version cost?

The problem

A fraud team has a merchant id with a hundred thousand values. One-hot is hopeless. A colleague suggests replacing each merchant with its historical fraud rate. The model's validation improves dramatically. A later, stricter holdout — a future month — shows most of the improvement was not real.

The obvious approach

Compute the fraud rate per merchant on the training data, replace the id with that rate, and train. A merchant's history is a legitimate feature; this is just a compact way to represent it.

Why it breaks

For a merchant with one transaction, the encoded value *is* that transaction's label: a fraudulent single-transaction merchant is encoded as 1.0, a clean one as 0.0. The model learns that the feature is the label for rare merchants, and rare merchants are most of them.

How it breaks — usually after the offline metric looked fine
  • For a merchant with one transaction, the encoded value *is* that transaction's label: a fraudulent single-transaction merchant is encoded as 1.0, a clean one as 0.0. The model learns that the feature is the label for rare merchants, and rare merchants are most of them.
  • For a merchant with thirty transactions, each row's encoding includes its own label at one-thirtieth weight. The leak is smaller but present in every row, and a flexible model finds it.
  • Validation looks good because the model, fitted on a leaked feature, is evaluated on a validation set whose encoding was computed from training rows only — but the model has learned to trust the feature far more than its honest signal warrants, and on the future month it over-relies on a feature that is now just a noisy prior.
Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target
  • Predict whether a transaction is fraudulent at authorisation. The label is a confirmed fraud outcome arriving weeks later.
  • Merchant identity is genuinely one of the strongest signals, which is exactly why a leaky encoding of it is so convincing.
Data
  • One example is one transaction with a merchant id, among other features. Most merchants have a few dozen transactions; many have one.
  • The target encoding was computed as the fraud rate per merchant over the entire training set, then joined back onto every training row — including the row whose label contributed to the rate.
  • The split was by time, correctly. The leak is inside the training set, not across the split.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

  • Target encoding replaces category c with an estimate of E[y | c]. Computed naively on the training set, the estimate for row i includes y_i, so the feature carries the label. The contamination is 1/n_c per row: total for singletons, small for frequent categories, and always present. It is target leakage (Target Leakage) manufactured by a preprocessing step (Preprocessing Leakage).
  • The out-of-fold fix computes each training row's encoding from the *other* folds: split the training set into k folds, and for rows in fold j use rates computed on the k-1 remaining folds. No row's label contributes to its own feature. For validation, test and serving, use rates computed on the full training set, which is what serving will have.
  • Smoothing fixes the small-count problem the leak was hiding: the encoded value is (n_c * mean_c + m * prior) / (n_c + m), shrinking rare categories toward the global rate. With m tuned, a singleton merchant is encoded near the prior, not at its single label.
  • In serving the rates are a lookup table fitted on training — state that ships with the artifact — and they go stale: a merchant that turned fraudulent last week still has last month's rate until retraining. That is a freshness question, and it is one reason a windowed, as-of merchant fraud rate (Aggregation Features) is often the better feature.

The row's own label is in its feature

The naive encoding computes the fraud rate per merchant over the training set and joins it back. For a merchant seen once, that rate is the label of that one row. The feature does not correlate with the label for rare merchants; it is the label. And the long tail of a hundred thousand merchant ids is mostly rare merchants.

The model learns the obvious thing: trust this feature enormously. On validation — encoded from training rates, so not itself leaked — the feature is still a decent prior, and the model's over-trust is only partly punished. On a future month, where every rate is a stale prior and new merchants are the global mean, the over-trust costs most of the apparent gain.

leakagemerchant_fraud_rate (computed over the full training set)Target encoding, naive

looks like A single numeric column replacing a hundred-thousand-value id, with an intuitive meaning and very high feature importance.

why it leaks Each training row's value was computed from a set of labels that includes its own. For a singleton merchant the feature equals the label; for others it contains the label at weight 1/n. The answer reaches the model through its own feature.

offline
Training fit is near perfect on rare merchants and validation improves substantially, because the model has learned to weight a feature that was partly the label.
production
On new transactions the rate is an honest, stale prior. The model over-relies on it, and the future-period metric gives back most of the validation gain.

fix Out-of-fold encoding for training rows; full-training-set rates for validation, test and serving; smoothing toward the prior; confirm the gain on a later-period holdout.

when this feature is fine A merchant's fraud rate over transactions that *preceded* this one — a point-in-time windowed aggregate — is legitimate and strong: it is known at authorisation, contains no future or own-row labels, and is computed the same way in serving. That is the feature the naive encoding was approximating.

Out of fold, with a prior

The fix has two parts. Out-of-fold computation guarantees that no row's label is in its own feature: the training set is cut into k folds and each fold's rows are encoded with rates from the other k-1. Smoothing guarantees that a merchant with two transactions is not encoded at its raw two-transaction rate but shrunk toward the global rate, with the shrinkage controlled by a strength m that is tuned on validation.

For validation, test and serving the encoding uses rates from the full training set, because that is the table serving will have. The two code paths — out-of-fold for training rows, full-table for everything else — are the operational cost, and they need a consistency test.

Naive versus out-of-fold target encoding
1import numpy as np, pandas as pd
2 
3def smoothed_rates(cat, y, m, prior):
4    g = pd.DataFrame({"c": cat, "y": y}).groupby("c")["y"].agg(["sum", "count"])
5    return (g["sum"] + m * prior) / (g["count"] + m)     # shrink rare cats to prior
6 
7# NAIVE (leaks): every row's own label is inside its rate
8prior = y_train.mean()
9rates = smoothed_rates(cat_train, y_train, m=10, prior=prior)
10te_naive = cat_train.map(rates).fillna(prior)
11 
12# OUT-OF-FOLD (honest): a row's rate comes from folds it is not in
13te_oof = pd.Series(np.nan, index=cat_train.index)
14folds = np.arange(len(cat_train)) % 5
15for k in range(5):
16    fit, apply = folds != k, folds == k
17    r = smoothed_rates(cat_train[fit], y_train[fit], m=10, prior=prior)
18    te_oof[apply] = cat_train[apply].map(r).fillna(prior)
19 
20# validation / test / serving: full-training rates, shipped in the artifact
21artifact["te_rates"], artifact["te_prior"] = rates, prior
22te_val = cat_val.map(rates).fillna(prior)

The two encodings differ most on rare merchants, which is where te_naive is closest to the label. Compare a model trained on each against a later-period holdout; the naive one's validation advantage should mostly vanish there.

The rate table goes stale

Once shipped, the rate table is a snapshot of merchant behaviour as of training. A merchant that turned bad last week keeps last month's clean rate until the next retrain; a merchant that was cleaned up keeps its bad one. New merchants are all the prior. This is a freshness problem, not a leakage one, and it is why a windowed, as-of merchant rate computed identically in both paths is often the better feature: it is fresh, it is point-in-time by construction, and it has no out-of-fold machinery.

The trade is that the windowed version is an aggregation feature, with that family's skew surface. The static table is simpler and staler. Either way the assumption that has to hold is the same: no row's own label in its feature, and serving using the same rates the model was trained against.

must stay trueNo own-label, same rates

Every training row's target encoding excludes its own label, and validation, test and serving all use the same full-training rate table and prior stored in the artifact.

holds when Out-of-fold encoding in training; the rate table and prior serialised with the model; unseen categories mapped to the prior; a consistency test between the training and serving encoders.

breaks when A second target-encoded feature is added naively; the serving path recomputes rates from recent traffic; the rate table is refreshed without retraining the model; smoothing strength is re-tuned on a spent validation set.

how you would know The leave-one-out assertion on training rows; the naive-versus-OOF holdout comparison; the fraction of serving categories hitting the prior; the age of the rate table against merchant churn.

respond Rebuild the encoding out of fold, retrain, re-evaluate on a later period; if staleness is the problem, move to a windowed as-of rate and treat it as an aggregation feature.

How to build it

Most important first.

  • Compute training-row encodings out of fold; compute validation, test and serving encodings from the full training set; never let a row's own label into its feature.
  • Smooth toward the prior with a strength chosen on validation; treat unseen categories in serving as the prior.
  • Ship the rate table with the artifact and monitor how stale it is; where freshness matters, prefer a point-in-time windowed rate computed the same way in both paths.
  • Confirm the fix with the honest holdout: the gain from target encoding should survive a future-period evaluation. If it does not, the naive version was measuring the leak.

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

  • The metric on a later-period holdout with and without the target-encoded feature. The honest gain is the difference; the naive-encoding validation gain is not.
  • The in-fold versus out-of-fold validation gap, which is the size of the leak on this data.
  • The staleness of the served rate table — days since fit — against the rate at which merchant behaviour changes.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions
  • No training row's encoding was computed using its own label, and validation, test and serving encodings come from the full training-set rates stored in the artifact.
  • The smoothing prior and strength are those fitted on training, and unseen categories map to the prior in serving.
  • The served rate table is refreshed often enough that the category-level rates still describe current behaviour, or a windowed as-of rate is used instead.
How to verify — offline, online, and over time
  • Offline: compute both the naive and the out-of-fold encoding, train on each, and compare on a later-period holdout; a large gap between their validation numbers that vanishes on the holdout is the leak.
  • Offline: assert that for every training row the encoding computed with that row removed equals the stored value.
  • Online: monitor the fraction of serving categories that hit the prior and the age of the rate table; compare production performance on new versus known merchants.

What can go wrong

Failure modes in production
  • The out-of-fold encoding is correct, but the same category also appears in a second target-encoded feature computed naively, so the leak returns through the other column.
  • Smoothing strength is tuned on the same validation set many times, and the encoding starts to fit validation (Evaluation Leakage).
  • The rate table ships but is never refreshed; a year later the model prices every merchant by its behaviour from a year ago, and new merchants are all the prior.
What the recommended approach costs
  • Out-of-fold encoding adds k fits of the rate table and a more complex pipeline; the training-time encoding and the serving-time encoding are now computed by different code paths that must be tested for consistency.
  • Smoothing trades signal on rare categories for robustness, and the strength is another hyperparameter.
  • A static rate table is stale by construction; the fresher windowed alternative is an aggregate with all of that family's skew risks.
Misreads
  • "The split was by time, so there is no leakage." The leak is within the training set — each row's feature contains its own label — and a time split does not touch it.
  • "Validation improved, so the encoding works." Validation improved because the model learned to trust a feature that contained the answer on training rows. The later-period holdout is the test.
  • "Target encoding is just a merchant history feature." A merchant history feature computed as-of the transaction from prior transactions is legitimate. The naive target encoding includes the transaction's own outcome and future ones. Same idea, different clock.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

  • GENERALAny encoding that uses the label — target, leave-one-out, weight-of-evidence, count-of-positives — leaks in the naive form and needs the out-of-fold treatment, under any model family.
  • SIMULATEDThe Leakage Simulator at /ml/leakage does not inject target encoding directly, but its target-derived injection is the same mechanism in its purest form; the qualitative claims here about validation rising and the future period not moving carry that simulator's caveat — a linear model on synthetic data, for the shape of the argument.
  • CONTESTEDSome practitioners hold that with strong smoothing and reasonably frequent categories the naive encoding's leak is small enough to ignore and the out-of-fold machinery is not worth its complexity, particularly for tree models with early stopping on a proper holdout. That is sometimes true in effect; the reply is that the leak size depends on the category frequency distribution, which is long-tailed in exactly the high-cardinality cases target encoding is used for, and out-of-fold is a few lines.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Data Engineeringfeature-pipelines