Unsupervised Learning
No labels, so no loss against the truth. The model finds structure in whatever the features and the distance say — and nobody checked that those mean anything to the business.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The data has no target column, so you want the model to "find the structure". Structure according to what, and how would you know it found the wrong one?
A retail marketing lead says: "We have millions of customers and no idea who they are. Group them into a handful of segments we can talk to differently."
Standardise the columns, run k-means, pick k where the elbow flattens, plot the clusters in two dimensions, and name each one — "bargain hunters", "loyal premium". The plot looks clean and the names sound right.
The clusters are mostly the scaling decision. Log-transform monetary value and the segments reshuffle; the marketing lead does not know they were looking at an arbitrary choice.
- The clusters are mostly the scaling decision. Log-transform monetary value and the segments reshuffle; the marketing lead does not know they were looking at an arbitrary choice.
- One cluster is "customers who joined in the last month" — a real structure in the data and a useless segment, because it is defined by tenure and dissolves every week.
- The campaign built for "loyal premium" performs no better than the untargeted control. The cluster was visually separated and had no relationship to campaign response.
- Nothing failed offline because there was no offline. Silhouette score went up, the plot was clean, and the metric that mattered — response lift — was never in the objective.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- There is no target. The optimisation objective is a property of the data — within-cluster distance, reconstruction error, likelihood under a mixture — chosen by the engineer, not observed from the world.
- The decision downstream is which campaign each customer receives, so the useful output is a segment assignment that changes campaign response. Nothing in the objective mentions campaign response.
- One example is one customer as a vector: recency, frequency, monetary value, category shares, tenure, app versus web share. Each column was picked because it existed in the warehouse.
- Columns are on wildly different scales — tenure in days, monetary in currency, category shares in [0, 1] — and the choice of scaling decides the geometry the algorithm sees.
- There is no ground truth to hold out. Whatever the algorithm returns is the answer, unless someone goes and checks.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- An unsupervised objective is a function of the inputs alone. k-means minimises the sum of squared distances to cluster centres; a mixture model maximises likelihood; PCA maximises retained variance. The gradient points toward whatever that function rewards, and the function was chosen by you.
- Distance is the hidden label. Every column, scale and transformation changes which points are near, and therefore what structure exists. There is no "the clusters" — there is the clusters under this metric.
- Because nothing external is predicted, there is no held-out truth to fail against. Internal metrics (silhouette, inertia, reconstruction error) measure how well the structure fits the objective, which is circular by construction.
Distance is the label
k-means asks: which partition of the points into k groups minimises the squared distance from each point to its group centre? The answer depends entirely on what "distance" means, and distance depends on which columns are present and how they are scaled.
That makes the scaling decision the most important modelling choice in the whole exercise, and it is usually made by a default. Standardising to unit variance says every column matters equally; leaving monetary value raw says it is the only thing that matters.
1import numpy as np2 3def kmeans_inertia(X, k, seed=0):4 rng = np.random.default_rng(seed)5 C = X[rng.choice(len(X), k, replace=False)]6 for _ in range(50):7 d = ((X[:, None, :] - C[None, :, :]) ** 2).sum(-1) # squared distance8 a = d.argmin(1) # assignment9 C = np.stack([X[a == j].mean(0) for j in range(k)])10 return a, d[np.arange(len(X)), a].sum()11 12# geometry A: raw columns -> monetary (thousands) dominates the distance13a_raw, _ = kmeans_inertia(X_raw, k=5)14# geometry B: log monetary, then standardise -> every column counts once15a_std, _ = kmeans_inertia(standardise(np.log1p(X_raw)), k=5)16# agreement between the two partitions is the honest number to report17print(adjusted_rand(a_raw, a_std))Both runs converge. Both produce five clean groups. The agreement between them is often low, and that number — not either inertia — tells the marketing lead what they are buying.
No held-out truth, so no offline failure
A supervised model that has learned nothing useful fails on the validation set. A segmentation that has learned nothing useful produces five groups and a silhouette score, and the plot looks like every other segmentation plot. The absence of a label removes the absence of a warning.
The only way to fail is against something outside the objective. For a marketing segmentation that is a randomised campaign; for an operations segmentation it is next quarter's support load. If no such check is planned, the segmentation is a diagram.
Five well-separated clusters on the 2D projection; silhouette improved over the previous three-segment run; each cluster given a plausible name.
Segment-targeted creative shows no response lift over the untargeted control; the "new customers" segment has turned over completely since the run.
- 1The clusters separate on recency and frequency, which were the highest-variance columns after scaling; campaign response varies on something the features did not contain.
- 2One segment was a tenure artefact — customers who joined recently — and its members left it within weeks.
- 3The names were assigned by looking at cluster means, which is a story about the centre and says nothing about the members near the boundaries.
Structure the business can act on
The question to settle before running anything is what structure would be useful. Segments that respond differently to price, groups that need different onboarding, accounts that behave alike over time — each implies different features, a different distance and a different validation.
Once that is settled, the assumption the segmentation depends on is clear: the geometry that produced it still describes the customers, and the segments still separate the outcome they were validated against.
The segment assignment still predicts a difference in the external outcome it was validated against, and the feature geometry has not moved customers across boundaries en masse.
holds when Segment shares are stable month over month and a periodic randomised check still shows response differences between segments.
breaks when A pricing or product change shifts the feature distribution; an upstream column changes definition; the campaign itself changes behaviour so the segments no longer describe anyone.
respond Do not re-run with the same features and hope. Re-ask what structure the business needs, re-validate, and treat the new segmentation as a new artifact with its own check.
How to build it
Most important first.
- Decide what "structure" means to the business before choosing an algorithm: segments that respond differently to a campaign, groups that share support needs, customers that behave alike over time. Then design a check for that (Decision Before Model).
- Validate against an external outcome the objective did not see: response rate per segment in a randomised campaign, churn rate per segment next quarter, revenue per segment. A segmentation that does not separate anything external is a picture (Clustering).
- Treat scaling and feature choice as hyperparameters and report how stable the assignment is across reasonable choices. A segment that survives a log transform is more likely to be real.
- Prefer the simplest structure the business can act on. Five segments that a marketer can name and target beat twelve that fit better.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The number that maps to the decision is the difference in campaign response, retention or revenue between segments in a randomised test. That is the only evidence the segments are more than geometry.
- Assignment stability under re-scaling, re-sampling and re-running with a different seed. Unstable assignments cannot support a campaign that runs for a quarter.
- Silhouette, inertia and the elbow are useful for choosing among runs of the same objective and say nothing about business meaning. Do not report them to the marketing lead as evidence.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The distance metric and feature scaling that produced the segments still reflect what the business means by similarity, and have not been changed upstream without re-running the segmentation.
- The external outcome the segments were validated against — campaign response, churn — still differs between segments, which a periodic randomised check can confirm.
- The feature distribution has not shifted enough to move a large share of customers across segment boundaries.
- Offline: re-run the segmentation under several scalings and seeds and measure assignment agreement. Report the agreement, not the prettiest run.
- Online: randomise a campaign within and across segments and measure whether the segment explains response beyond the untargeted control.
- Over time: recompute segment shares monthly; a segment that grows or shrinks sharply without a business explanation means the geometry moved.
What can go wrong
- The segments are stable and meaningful at launch, and six months later a pricing change moves half the customers across a boundary; the campaign keeps running on assignments that no longer describe anyone (Data Drift).
- A dominant column — order count — swamps the distance, and every segment is a frequency band with a story attached.
- The external validation is run once, and the segments are then reused for a purpose they were never validated for.
- Validating against an external outcome requires a randomised campaign, which is slower and more expensive than presenting a plot.
- Reporting stability instead of one clean result gives the business a messier answer that is harder to sell and closer to true.
- Fewer, actionable segments fit the data worse by every internal metric, which will be raised by whoever compares runs on silhouette.
- "The clusters are clearly separated, so they are real." They are clearly separated in the projection under this scaling. Change either and different clusters are clearly separated.
- "Unsupervised learning has no labels, so it has no bias." The feature choice and the distance metric are the label. They were chosen by a person and encode what that person thought mattered.
- "The elbow says k is five." The elbow says five minimises a curvature heuristic on inertia. Whether five segments can be acted on is a marketing question.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat an unsupervised objective is chosen rather than observed, and that distance plays the role of the label, holds for clustering, mixture models, matrix factorisation and autoencoders alike.
- DATA-SPECIFICOn a handful of well-understood tabular columns the scaling problem is visible and fixable; on learned embeddings the distance is inherited from a training objective nobody in the room chose, and stability across scalings is replaced by stability across embedding versions (Embedding Drift).
- CONTESTEDA serious position holds that customer segmentation should be abandoned in favour of individual-level uplift models, because any segment averages over people who respond differently and a model with a label beats one without. The counter is that segments are a communication device for humans who design campaigns, and an uplift model gives them nothing to design with; the two are complements once the segments are validated against uplift.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Product analytics — deciding what a segment is for, and designing the randomised campaign that tests it, is a product question this lesson depends on rather than answers.