In your dataset, what does one row represent — and does the model know that?

Answer it out loud before you open anything. The value of the flags below is in comparing them to what you actually said — including whether you asked about the data before naming a model.

The production scenario behind the question

A logistics company is building a model to predict whether a delivery will be late. The training table was built by joining shipments to scan events, and it has 40 million rows. Some shipments have one row, some have thirty. The team reports AUC on a random split and is pleased. The model is meant to run once, when the shipment is created.

React to this

Say what you would question, what you would trust, and what you would need to know first.

The report, the feature list, or the dashboard as it stands

Training table: shipment_scan_features (illustrative)

  rows: 40,212,880     distinct shipment_id: 1,184,301
  label: late = delivered_at > promised_at    (null for 3.1% of shipments)

  columns: shipment_id, created_at, origin_hub, dest_zip, weight, service_level,
           scan_ts, scan_hub, scans_so_far, hours_since_last_scan, last_scan_type,
           late

  split: random 80/20 over rows
  AUC:   0.93   (random split)   —   model runs once at created_at

What it is really testing

Whether the candidate asks the most basic and most skipped question in the domain: what is the unit of prediction, and does each training example correspond to it? The scenario has a fan-out join, a mismatch between training grain and serving grain, and a random split over rows that share an entity.

Where the mechanism is taught

Dataset Construction lessons →Problem says X → think Y →Diagnose a failure →Practice challenges →