Raw Features vs Learned Representations
Either a person decides what the model sees, or the model decides. Each choice hides something, and the learned one ships inside the artifact and must be versioned like weights.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Should we hand-engineer the features or let the model learn its own representation from raw input — and what does each choice commit us to at serving time?
A marketplace team wants to flag listings that will get complaints. They have the listing text, the photos, and a warehouse table of seller history. One engineer wants to build aggregates over the seller table; another wants to fine-tune a pretrained image-and-text model on the raw listing. The product manager wants to know which is "the modern approach".
Pick one paradigm for the whole problem. Either build a feature table in SQL and train a gradient-boosted model on it, because that is what worked last time; or fine-tune a pretrained multimodal model on the raw listing, because learned representations are the modern approach and the model will find whatever matters.
The hand-engineered path works well for sellers with history and is nearly blind for new sellers, who file the majority of the complaints. Every aggregate is null or zero for them and the model learns that "no history" is safe.
- The hand-engineered path works well for sellers with history and is nearly blind for new sellers, who file the majority of the complaints. Every aggregate is
nullor zero for them and the model learns that "no history" is safe. - The learned path is strong offline and then a change in the photo upload pipeline — a new resize, a different JPEG quality — silently shifts every image embedding. The weights did not change; the representation the model receives did (Train / Serve Skew).
- The learned representation was fine-tuned on last year's listings. Six months later the team swaps in a newer pretrained backbone "for better quality" without retraining the head, and the head is now reading vectors from a space it never saw.
- Nobody can say why a listing was held. The hand-engineered model could point at
complaints_per_100_listings_last_year; the learned one points at dimension 412.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a new listing receives a buyer complaint within 30 days of going live. The label is the complaint ticket, which arrives up to a month after the prediction had to be made.
- The decision is whether to hold the listing for review before it is published, so the prediction must exist at listing-creation time from what is known then.
- One example is one listing at creation time: its title and description, up to eight photos, a category, a price, and the seller's account.
- The seller table has years of history — prior listings, prior complaints, response times — from which a person can write aggregates like
complaints_per_100_listings_last_year. - The photos and text carry most of the signal for a first-time seller, and there is no column for "the photo is a stock image" or "the description contradicts the photo". Nobody can write that aggregate by hand.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A hand-engineered feature encodes a human hypothesis: "sellers with recent complaints will get more complaints." The model only chooses how much to weight it. The hypothesis space is small, the data needed is small, and the feature is legible — and if the hypothesis is missing, the model cannot recover it.
- A learned representation is a function from raw input to a vector, fitted so that the vector is useful for the loss. For images, text and audio this is the only practical route, because the useful features — edge, texture, phrase, tone — are compositions of thousands of raw values that no one can write down by hand.
- The representation is itself parameters. Whether it is a frozen pretrained encoder or a fine-tuned one, it is part of what turns a listing into the tensor the head consumes, so it is part of the model, not part of the data. Changing it without changing the head is the same category of mistake as changing a feature definition without retraining (What a Model Artifact Contains).
Two places to put the assumptions
The traditional pipeline is raw data, then hand-engineered features, then a model that weights them. The representation-learning pipeline is raw data, then a model, and the representation is whatever the intermediate layers learned to produce. The difference is who decides what the model sees: a person with domain knowledge, or the loss.
Neither is more "correct". The hand-engineered path puts the assumptions where you can read them — complaints_per_100_listings_last_year is a hypothesis you can argue about. The learned path puts them in the encoder weights and the fine-tuning data, where you cannot read them but where they can express things no one thought to write.
| Option | Quality | Latency | Cost | Interpretability | Data needed | Operational | Note |
|---|---|---|---|---|---|---|---|
| Hand-engineered aggregates only | Strong for sellers with history, blind for new ones and for everything in the photos; every aggregate is a legible column and a serving dependency. | ||||||
| Learned representation only | Covers photos and text that no aggregate can; needs a GPU on the serving path, an encoder version in the artifact, and offers dimension 412 as its explanation. | ||||||
| Aggregates + pretrained encoder | Best coverage of both seller types; two kinds of drift to watch and two things to version, but the tabular half stays explainable. |
caveat Quality scores assume a marketplace with both a seller-history table and rich listing media; on a purely tabular problem the first row wins outright and the other two are not worth their cost. Interpretability here means the ability to name a column, not that the column is causal.
Either aggregates over the seller table for every listing (blind to new sellers, blind to the photos) or a fine-tuned multimodal model on the raw listing (ignores years of legible seller history and costs a GPU per prediction).
Aggregates for the tabular history, a pretrained encoder for photos and text, concatenated into one vector for a single head; the aggregates stay legible, the encoder covers what no one can write down.
The features a person can write and the features a model can learn are different sets, and the problem contains both kinds of input. The choice is per input, not per project.
What each one hides
The hand-engineered path hides everything the engineers did not think of. The model cannot discover that a stock photo predicts complaints, because no column says so. Its ceiling is the feature list, and the ceiling is invisible in the metric — the model simply looks as good as the features allow.
The learned path hides the meaning of what it found. It may have learned that stock photos predict complaints, or that a particular JPEG encoder — used by one seller tool — predicts complaints. Both raise the offline metric; only one survives the next change to the upload pipeline. The Embedding Projection Caveats lesson is about the same blindness from the other side: even looking at the representation does not tell you what it encodes.
Held-out recall at the review-queue budget clearly above the aggregate-only model, on a validation set drawn from the same quarter.
Hold rate on new listings climbs over one week with no change to any model repository; reviewers report the held listings look normal. The head is unchanged.
- 1The image upload service changed its resize and compression, so every photo embedding moved; the head reads vectors from a slightly different space than the one it was trained on.
- 2The shared embedding service upgraded its backbone version for another team; downstream heads were not retrained.
- 3A genuine shift in what sellers post — new categories, new photo styles — which would show as gradual drift rather than a week-long step.
The representation is part of the artifact
Whether the encoder was trained by you, fine-tuned by you, or downloaded, its weights and its preprocessing are half of the function that turns a listing into a prediction. An artifact that contains the head but references the encoder by name — "the embedding service" — is not reproducible and is not safe to serve.
This is the same discipline as Preprocessing Lives in the Artifact, one level up. A normaliser fitted on the training fold must ship with the model; so must the encoder, or at least its exact version and a check that the deployed one matches. The Agentic domain's view of the same objects — embeddings as a retrieval primitive — has the same versioning problem from the index side.
At serving time the representation is computed by the same encoder weights and the same preprocessing that produced the training vectors.
holds when The artifact pins the encoder version and preprocessing, the registry refuses to serve a head against a different encoder, and the upload and tokenisation pipelines are part of the tested serving contract.
breaks when A shared embedding service is upgraded; an image or text preprocessing library changes its defaults; a "drop-in" newer backbone is swapped for quality without retraining the head.
respond Roll the encoder back or retrain the head against the new one. Do not tune the threshold to compensate; the score distribution moved because the inputs did.
1# saved alongside the head at training time2artifact = {3 "head": head_weights,4 "encoder": {"name": "listing-encoder", "version": "2026.03.1", "sha256": ENCODER_SHA},5 "preprocess": {"resize": [224, 224], "interp": "bilinear", "tokenizer": "v3"},6 "probe": {"inputs": probe_listings, "vectors": encoder(probe_listings)},7}8 9def check_encoder(artifact, live_encoder):10 # same weights, same preprocessing -> same vectors on a fixed probe set11 v = live_encoder(artifact["probe"]["inputs"])12 if live_encoder.sha256 != artifact["encoder"]["sha256"]:13 raise RuntimeError("encoder version differs from the one the head was trained on")14 if max_abs_diff(v, artifact["probe"]["vectors"]) > 1e-5:15 raise RuntimeError("encoder output differs on probe set: preprocessing changed")The probe set is what catches a preprocessing change that leaves the weights untouched. A hash of the weights alone would pass while the resize interpolation silently moved every vector.
How to build it
Most important first.
- Let the modality decide, per input. Tabular history with meaningful domain aggregates: hand-engineer and keep the aggregates legible. Images, text, audio: use a learned representation, usually a pretrained encoder (Transfer Learning) rather than one trained from scratch on your data.
- Combine them when the problem has both. A learned embedding of the photos and text concatenated with the seller aggregates is a normal design, and the tabular part gives you the interpretable half.
- Version the encoder with the head. The artifact must record the encoder weights, the preprocessing (resize, tokeniser, normalisation) and the head together, and the registry must refuse to serve a head against a different encoder (Feature and Model Versioning).
- Monitor the representation, not only the raw input: embedding drift is detectable by the distribution of the vector norm and of the nearest-centroid distances, well before the outcome labels arrive (Embedding Drift).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Recall at the review-queue budget, sliced by seller tenure. The aggregate metric hides that one paradigm serves new sellers and the other serves old ones (Evaluation Slices).
- The serving cost per listing — a learned encoder on eight photos is a GPU-scale expense per prediction, and the comparison has to include it (Inference Cost).
- Do not measure "which paradigm is more accurate" on a validation set dominated by established sellers. That measures the population the aggregates were built for.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The encoder that produces the representation at serving time is byte-for-byte the one the head was trained against, with the same preprocessing in front of it.
- The raw inputs — photo resolution and compression, text length and language — remain within the distribution the encoder was fitted or fine-tuned on.
- The hand-engineered aggregates keep their definition and their coverage: the fraction of examples where they are null does not move.
- Offline: for a fixed sample of listings, assert that the serving path and the training path produce identical embedding vectors, as a contract test on the artifact (Serving Contract Tests).
- Online: monitor the embedding norm and the per-dimension mean for a step change on any deploy of the encoder, preprocessing or upload pipeline.
- Over time: compare recall by seller tenure as complaint labels arrive; the learned half and the hand-engineered half decay for different reasons and on different schedules.
What can go wrong
- The encoder is upgraded by another team who own the "embedding service" and see it as infrastructure; every downstream head degrades at once with no code change in any model repository.
- The image preprocessing at serving time differs from training — a different library resizes with a different interpolation — and the skew lives inside the representation where feature monitors do not look.
- A hand-engineered aggregate that was a strong feature becomes an attack surface: sellers learn that a clean first month unlocks lenient treatment, and the feature's meaning changes because the model is deployed (Feedback Loops).
- A learned representation needs more data, more compute, and a GPU on the serving path; it also removes the ability to point at a column when a seller asks why they were held.
- Hand-engineered features are cheap and legible and cap the model at what the engineers thought of; each one is also a serving dependency that must be fresh and identical in both paths.
- Combining the two gives the best of both and doubles the artifact surface: two things to version, two kinds of drift to watch.
- "Neural networks are always better, so use the learned representation." On the seller history table, a boosted tree over ten aggregates will usually beat a network trained on the same rows, at a fraction of the cost. The learned representation wins where the features cannot be written by hand, not everywhere.
- "The embedding service is infrastructure, like the database." It is half of every model that reads from it. Upgrading it is a model change for every downstream head.
- "Learned representations remove feature engineering." They move it: into the choice of encoder, the preprocessing, the fine-tuning data, and the decision of what raw input to show the model at all.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- DATA-SPECIFICOn tabular data with a few hundred thousand rows and meaningful domain aggregates, hand-engineered features plus a tree ensemble are usually the strongest and cheapest option; on images, audio or free text the features to hand-engineer do not exist and a pretrained encoder's learned representation wins by a wide margin.
- SCALE-SPECIFICA learned encoder on the serving path means a GPU-class cost per prediction; at low volume that is a line item, at marketplace volume it is the dominant cost of the system and decides whether the representation is precomputed in batch or computed online.
- CONTESTEDA serious position holds that with enough data even tabular problems are better served by learned representations of the raw event stream — the aggregates a person writes are a lossy summary, and a sequence model over raw events recovers what they threw away. That is sometimes true at very large scale with rich event logs; for most teams the data needed to learn those representations reliably is not there, and the aggregate is the stronger prior.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — the probe-set check is a golden-file test on a component another team owns, and the question of who is allowed to change a shared encoder is an ownership and contract question this domain assumes rather than answers.