Clustering
k-means and hierarchical clustering find groups under a distance you chose. A visually separated cluster is a fact about the geometry, not about the business — until something external says otherwise.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The algorithm returned clean groups. Under what distance, how stable are they, and what outside the algorithm says they mean anything?
A product lead says: "We have thousands of user journeys through the app. Cluster them into a few behaviour types so we can design an onboarding flow for each."
Standardise, run k-means for k from two to ten, pick k by the elbow or silhouette, plot the result in two dimensions, and name each cluster from its centroid: "explorers", "goal-seekers", "bouncers". Design one onboarding flow per name.
The "bouncers" cluster is the near-zero rows — users who left. It is real, it is the largest, and no onboarding flow can reach it because its members are gone before onboarding begins.
- The "bouncers" cluster is the near-zero rows — users who left. It is real, it is the largest, and no onboarding flow can reach it because its members are gone before onboarding begins.
- The three correlated screens dominate the distance, so two of the clusters are "visited that trio" and "did not", which the product lead already knew.
- k-means finds compact spherical groups because that is what its objective rewards. The journeys may have elongated or nested structure, and k-means cuts it into pieces that look clean and mean nothing (k-Nearest Neighbours has the same distance dependence).
- The plot was clean, silhouette was fine, and a month later the onboarding flows designed per cluster show no difference in retention against a single flow.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- There is no target. The objective is chosen: k-means minimises squared distance to k centres; hierarchical clustering merges the closest pairs under a linkage rule; density methods find regions of high point density. Each finds a different kind of group.
- The decision is which onboarding flow a new user receives, so the output must be an assignment rule that works on a new user with little history, and the groups must differ in something onboarding can affect (Unsupervised Learning).
- One example is one user's first-week journey: counts of screens visited, session lengths, feature-use flags, time to first key action. Each is a column chosen from what the event log offers.
- Users who churned in day one have sparse journeys, and many rows are near zero on everything; they will form a cluster whatever the algorithm does.
- The columns are on different scales and some are strongly correlated — three screens that are always visited together contribute three times to the distance.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- k-means alternates assigning each point to its nearest centre and moving each centre to the mean of its points. It converges to a local minimum of within-cluster squared distance and finds roughly equal-sized, convex groups under Euclidean distance, whether or not the data has them.
- Hierarchical clustering builds a tree by merging closest clusters; single linkage chains through dense paths, complete linkage prefers compact groups, Ward minimises variance increase. The tree is always produced; where to cut it is a choice, and different linkages give different trees on the same points.
- Internal validity metrics — silhouette, inertia, Davies–Bouldin — measure how well the partition fits the objective. They are circular: they cannot say whether the groups correspond to anything outside the feature space, and they reward whatever the scaling made compact.
Two algorithms, two ideas of a group
k-means asks for k centres that minimise squared distance. Its groups are Voronoi cells: convex, and split by straight boundaries. Hierarchical clustering asks which points are closest and merges them repeatedly; its groups are whatever the linkage rule builds, and the dendrogram shows every scale at once.
On the same journeys they will disagree, and neither is wrong. Each is answering a different question about the geometry, and the geometry was set by the columns and the scaling.
1import numpy as np2 3def kmeans(X, k, seed):4 rng = np.random.default_rng(seed)5 C = X[rng.choice(len(X), k, replace=False)]6 for _ in range(100):7 a = ((X[:, None] - C[None]) ** 2).sum(-1).argmin(1)8 C_new = np.stack([X[a == j].mean(0) if (a == j).any() else C[j] for j in range(k)])9 if np.allclose(C, C_new): break10 C = C_new11 return a12 13def stability(X, k, runs=20):14 base = kmeans(X, k, seed=0)15 scores = []16 for s in range(1, runs):17 idx = np.random.default_rng(s).choice(len(X), len(X), replace=True) # bootstrap18 a = kmeans(X[idx], k, seed=s)19 scores.append(adjusted_rand(base[idx], a)) # agreement on the resampled points20 return np.mean(scores), np.min(scores)21 22# report this next to the plot: a partition with low agreement is a picture, not a segmentation23print(stability(standardise(X_journeys), k=4))The plot for seed 0 will look clean. The agreement across bootstrap runs is what says whether another analyst would have found the same groups, and that is the minimum bar before a designer builds anything.
A cluster is not a segment
A cluster is a set of points that are close under a distance. A segment is a set of users who respond differently to something the business can do. The first is a property of the feature space; the second is a property of the world, and only an experiment can show it.
The near-zero journeys are the clearest case: a perfectly real, perfectly stable cluster that no onboarding flow can reach. Its existence proves the algorithm worked and says nothing about what to build.
Four well-separated clusters with a good silhouette; each named from its centroid; the plot shown to the design team.
Per-cluster onboarding flows show no retention difference against a single default flow; the largest cluster is users who never returned after day one.
- 1The partition separated activity level — which retention already tracks — and not anything onboarding could change.
- 2The largest cluster was defined by absence and its members were unreachable.
- 3Three correlated screen counts dominated the distance, so two clusters were the same fact stated twice.
What the assignment rule depends on
Once flows ship, every new user is assigned by a rule fitted to a snapshot of journeys under a fixed scaling. That rule keeps running as the app changes, and it has no label to tell it the groups have moved.
So the assumptions are about the geometry and the validation: the distance is frozen, and the external check is repeated.
The feature set and scaling behind the assignment rule are unchanged, and the clusters still differ in the outcome the onboarding flows were built to move.
holds when The journey schema and scaling parameters are versioned with the assignment rule; cluster shares are stable; a periodic randomised onboarding check still shows per-cluster differences.
breaks when New screens are added and standardised into the distance; a redesign changes what a first-week journey looks like; the flows themselves change behaviour so the clusters dissolve.
respond Re-derive the segmentation from the new journeys, re-validate, and treat it as a new artifact — do not re-fit the same k on the new columns and keep the old names.
How to build it
Most important first.
- Define the external criterion first: onboarding flows are meant to move retention, so clusters must differ in retention or in something known to affect it. Plan the randomised test of that before clustering (Decision Before Model).
- Handle the degenerate mass explicitly — remove or separately model the near-zero journeys — so the algorithm is not spent on a group that is defined by absence.
- Decorrelate or select features so the distance reflects what onboarding can change, and report assignment stability across scalings, seeds and bootstrap samples; unstable groups cannot support a flow.
- Prefer a small number of clusters a designer can build for, and make the assignment rule explicit — a few thresholds on interpretable features — so a new user can be assigned on day one.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Retention difference between onboarding flows in a randomised test, per cluster. That is the number the decision depends on; the clusters exist to make it bigger.
- Assignment agreement across re-runs with different seeds, scalings and bootstrap samples. Below a stated agreement, the partition is noise.
- Silhouette and inertia are for choosing among runs of the same objective on the same features. They do not measure whether "explorers" exist.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The distance, scaling and features that produced the partition are frozen with the model, and a change to the event schema does not silently alter the assignment rule.
- The clusters still separate the external outcome they were validated against — retention under different onboarding flows — as a periodic randomised check confirms.
- The journey distribution has not shifted enough that a large share of new users falls near a boundary or outside all clusters.
- Offline: bootstrap the data, re-cluster, and compute assignment agreement with the original; repeat under two or three reasonable scalings. Report the agreement alongside any plot.
- Online: randomise onboarding flows within each cluster for a cohort and measure retention per flow per cluster; a cluster where flows do not differ is not a segment.
- Over time: cluster-share drift per week and the fraction of new users near a boundary; re-run the randomised check when either moves.
What can go wrong
- The clusters are validated once against retention and then reused for a pricing experiment they were never validated for, where they separate nothing.
- A new feature in the app adds columns to the journey, the pipeline standardises them in, and the assignment rule silently changes for everyone.
- The stable clusters are the trivial ones — active versus inactive — and the interesting structure the designer hoped for is in the unstable residue.
- Removing the degenerate cluster and decorrelating features makes the result less impressive — fewer clean groups, a messier plot — and closer to something onboarding can use.
- A randomised onboarding test per cluster takes a cohort and a retention window, which is slower than shipping the flows.
- An explicit assignment rule is cruder than nearest-centroid assignment and far easier to keep stable and explain.
- "The clusters are visually separated, so they are real segments." They are separated in this projection under this scaling. A separated cluster is a geometric fact; a segment is a business fact that needs external evidence.
- "Silhouette is high, so k is right." Silhouette says the partition fits its own objective. It is highest for the trivial active/inactive split on most product data.
- "Name the clusters from the centroids." The centroid describes the centre; most members are near a boundary, and the name will be wrong for them.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a clustering objective is chosen, that distance and scaling decide the result, and that internal validity metrics are circular, holds for k-means, hierarchical, density and mixture methods alike.
- MODEL-SPECIFICk-means assumes compact, roughly spherical, similar-sized groups under Euclidean distance; density-based methods find arbitrary shapes and label sparse points as noise, which is often the honest answer on behavioural data but gives the designer nothing to build for.
- CONTESTEDA serious position holds that exploratory clustering is valuable precisely because it has no external criterion: it shows analysts structure they did not think to look for, and demanding a randomised validation before anyone looks kills the exploration. The counter is not that exploration is wrong but that its output is a hypothesis, and shipping onboarding flows on a hypothesis is the error this lesson is about.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Product design — deciding what an onboarding flow can change, and therefore what a useful segment would separate, is a design question this lesson depends on rather than answers.