You target-encoded a high-cardinality categorical and the validation score jumped. What happened?
Answer it out loud before you open anything. The value of the flags below is in comparing them to what you actually said — including whether you asked about the data before naming a model.
An ad-tech team has a click model with a site_id feature of 300,000 distinct values. One-hot is impossible, so an engineer replaced it with the mean click rate per site, computed over the full training table, and validation log loss improved sharply (illustrative). The engineer describes this as "the model learned which sites are good".
React to this
Say what you would question, what you would trust, and what you would need to know first.
feature_pipeline.py (excerpt, illustrative)
site_ctr = df.groupby("site_id")["clicked"].mean() # computed on full df
df["site_ctr"] = df["site_id"].map(site_ctr)
train, valid = train_test_split(df, test_size=0.2, random_state=0)
Validation log loss: 0.412 -> 0.288
site_id cardinality: 301,442; sites with < 5 impressions: 61%What it is really testing
Whether the candidate understands the mechanism of target encoding well enough to see where the label leaks into the feature, and can describe an encoding that does not leak — out-of-fold, with smoothing, computed as of the prediction time — and its remaining risks.