Group Split
When the same entity appears in many rows, all of its rows go to one side of the split. Otherwise the model is evaluated on recognising entities it already saw, and production is full of entities it has not.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Does the same user, patient, device or document appear in more than one row, and will production ask about entities the model has never seen?
A hospital wants to flag which admissions are at high risk of readmission within 30 days. A patient can have many admissions; the first model, validated on a random split of admissions, looked excellent and did poorly on the new patients it was mostly used on.
Split admissions at random. Each admission is a row with its own features and label; independent rows can be shuffled.
The frequent patients appear on both sides of the cut. Their combination of age, conditions and insurance is nearly a fingerprint, and the model learns "this patient readmits" from the training rows and is rewarded on the validation rows of the same patient. The score is memorisation (Entity Leakage).
- The frequent patients appear on both sides of the cut. Their combination of age, conditions and insurance is nearly a fingerprint, and the model learns "this patient readmits" from the training rows and is rewarded on the validation rows of the same patient. The score is memorisation (Entity Leakage).
- Production is dominated by first admissions of patients the model has never seen, because that is where the follow-up programme can help. On those, the model has only population-level signal and the ranking is far weaker than validation claimed.
- The importance analysis names the near-fingerprint features as the strongest, and the team builds more of them, which improves validation and does nothing for new patients (Feature Importance).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an admission will be followed by another within 30 days of discharge. The label is the next admission, observed up to 30 days after discharge.
- The decision is whether to enrol the patient in a post-discharge follow-up programme with limited capacity, so the model ranks admissions for a queue.
- One example is one admission with the patient's history up to that admission and the admission's own characteristics. Four hundred thousand admissions from a hundred and fifty thousand patients over five years.
- A minority of patients account for most admissions; those frequent patients have highly consistent readmission behaviour and dominate the positive class.
- Patient-level features — age, chronic conditions, insurance — plus admission features such as diagnosis and length of stay.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Rows that share an entity are correlated: they share the entity's stable features and its stable behaviour. A model can exploit that correlation only if the entity is present in training, so an evaluation on the same entities measures a mix of true generalisation and entity recall.
- A group split assigns every row of an entity to exactly one of train, validation and test, so the validation entities are unseen. The score then measures what production will see for a new entity, which is the harder and more honest task.
- Groups can nest — admissions within patients within hospitals — and the group to split on is the one production will be new on. If the model is deployed to new hospitals, split by hospital; if to new patients within the same hospitals, split by patient.
The same patient on both sides
The leakage is not a bad feature. Every feature here is legitimate at prediction time. The leak is structural: the patient sits on both sides of the cut, and the model learns them from one side and is graded on the other.
The device describes it as a leakage case because that is what it is — information from the evaluation set reaching the model — even though no column is at fault.
looks like Four ordinary, legitimate patient-level features, each available at admission and each individually well-behaved.
why it leaks Together they identify most frequent patients almost uniquely. With the patient's other admissions in training, the model learns that this fingerprint readmits, and the validation rows for the same patient carry the same fingerprint and the same answer.
fix Split by patient id so every admission of a patient is on one side, and evaluate new and returning patients separately.
Group split by hash, with an optional time cut
The mechanism: hash the entity id with a seed, map to the unit interval, and assign by range. Every row of an entity produces the same hash and lands on the same side, no list of ids has to be stored, and the split is reproducible from the seed.
The second function combines this with a time cut. Entities are assigned by hash; within the validation entities, only rows after the cut are used, so the evaluation is on new entities *and* a new period.
1import hashlib2 3def group_of(entity_id, seed="split-v1", val_frac=0.2):4 h = hashlib.sha1(f"{seed}:{entity_id}".encode()).digest()5 u = int.from_bytes(h[:8], "big") / 2**646 return "val" if u < val_frac else "train"7 8def split(rows):9 train = [r for r in rows if group_of(r["patient_id"]) == "train"]10 val = [r for r in rows if group_of(r["patient_id"]) == "val"]11 assert not {r["patient_id"] for r in train} & {r["patient_id"] for r in val}12 return train, val13 14def split_group_and_time(rows, cut):15 train = [r for r in rows16 if group_of(r["patient_id"]) == "train" and r["admitted_at"] < cut]17 val = [r for r in rows18 if group_of(r["patient_id"]) == "val" and r["admitted_at"] >= cut]19 return train, val20# The assert is the test. If it ever fails, an id is not what you think it is.Hashing on a stable external identifier matters more than the hash function. An internal surrogate key that changes on reload will silently reshuffle the split between runs and make two evaluations incomparable.
Which entity, and what production is new on
The right group is the one production will be new on. A model deployed to the same hospitals for new patients is grouped by patient. A model sold to new hospitals is grouped by hospital, and a patient-level split would still flatter it, because hospital-level practice patterns would sit on both sides.
That makes the group choice a statement about deployment, and it needs to stay true after the model ships.
The entities production scores are new to the model at the level the split grouped on, in roughly the proportion the validation set assumed, and entity ids identify real-world entities one-to-one.
holds when The deployment scope matches the grouping — new patients in known hospitals, or new hospitals — and the id used to group is stable across systems and time.
breaks when The model is rolled out to a new site when it was grouped by patient; ids are reissued on insurer change or system migration; a feature combination becomes a de facto identifier.
respond Re-split at the level production is actually new on and re-evaluate; do not retrain on the old split and read the higher number as improvement.
How to build it
Most important first.
- Identify the entity that production will be new on and hash its id to a split, so the assignment is deterministic and reproducible without a stored list.
- Combine with a time cut when the data drifts: group split decides which side an entity lands on; the time cut decides which periods each side covers (Time-Based Split, Choosing a Split Strategy).
- Evaluate separately on new entities and on returning entities, because production contains both and the model's quality differs sharply between them (Evaluation Slices).
- Check that the group assignment does not accidentally stratify — a hash on patient id is fine; a split by hospital puts each hospital's case mix wholly on one side and may need balancing.
- Apply the same grouping to any resampling or cross-validation, so every fold holds unseen entities (Cross-Validation).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The metric on validation rows whose entity never appears in training. This is the production number for new entities; the random-split number is not.
- The gap between the random-split score and the group-split score — the amount of entity recall the random split was crediting the model with.
- The metric on returning entities separately, and the production share of new versus returning, so the blended production number can be reconstructed.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The entity the split groups on is the entity production will be new on, and its identifier is stable and unique per real-world entity.
- The share of new versus returning entities in production is known, so the validation metrics on each can be blended into an expectation for the whole.
- No feature acts as a proxy identifier that reconstructs the entity across the split — a rare combination of static attributes that only one patient has.
- Offline: assert that no group id appears on more than one side; train a model to predict the entity id from features and check it cannot, which catches proxy identifiers.
- Online: compare production quality on first-time entities against the group-split validation score; they should be close, and a large gap means the grouping key does not match production's notion of new.
- Over time: track the new-versus-returning share in production traffic and re-blend the expected metric when it moves.
What can go wrong
- The split is by patient, but a patient who moves insurer gets a new id, and the same person sits on both sides under two ids.
- The split is by patient, and the training set now has fewer positives because the frequent readmitters were assigned wholesale to one side; the positive count in validation collapses and the estimate is noisy (Stratified Split).
- The hash key is not stable — an internal id that is regenerated on data reload — and the split silently changes between runs.
- A group split gives a lower number and, on data dominated by a few large entities, a lumpier one, because whole entities move between sides.
- It can starve validation of positives when the positive class is concentrated in a few entities, forcing a trade with stratification.
- It requires a reliable entity id, which some datasets do not have, and a decision about which level of a hierarchy is the group.
- "Each admission is its own row, so rows are independent." Rows are independent when nothing links them. A patient links admissions, and the link is exactly what the model exploits under a random split.
- "The group-split score is bad, so the model is bad." The group-split score is the model's performance on new patients. It was always this; the random split hid it.
- "Splitting by patient handles time too." It does not. A patient's admissions can span years, and a group split on its own puts future admissions of validation patients alongside past admissions of training patients from the same period.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat rows sharing an entity are correlated and that a shuffle credits the model with entity recall holds for any recurring-entity dataset regardless of model family.
- DOMAIN-SPECIFICSevere in healthcare, customer analytics and device telemetry where a few entities generate many rows; mild in one-row-per-entity settings, and for ranking or recommendation the group is often the query or the user rather than the item, which changes which side each row goes to.
Where the depth lives
This domain teaches the model and hands the rest off by name.