Debugging a Data Incident

• Upstream traversal works because corruption propagates downstream and only downstream. If a dataset is correct for the affected period, everything upstream of it is irrelevant to this incident, and the search space halves at every hop that passes (Data Lineage).
• It is a binary search over the lineage path, and the question at each node is the same one, which is what makes it a procedure rather than a talent. The termination condition is the first node where the answer is "no" — the boundary between a correct upstream and an incorrect downstream is the fault.
• The walk terminates in one of three places, and only two of them are your incident. The source is wrong: an application bug or a genuine business change. A hop lost, duplicated or delayed data: an ingestion or delivery fault. Or a transformation is wrong: the code did what it was told (Two Dashboards, Two Numbers).
• Which node to start from is decided by the *evidence*, not by the path. Running the full check suite over the affected period gives a failing set, and the failing set narrows the candidate causes before the first hop is examined — which is what turns a six-hop walk into a two-hop confirmation.
• The fingerprint works because the checks measure independent properties. Completeness compares identities against the source; uniqueness counts repeats; validity inspects types; distribution compares shape; freshness compares clocks; reconciliation compares totals. A fault that moves one leaves the others alone unless it genuinely touches what they measure.

Start at ingestion and check each stage in order until something looks wrong. Every stage needs a query and a judgement, and consequences of the real fault appear as anomalies along the way.

Start at the tile, reproduce the number one layer down, and keep asking the same question at each upstream dataset. Stop at the first node where the affected period is complete and correct; the fault is immediately downstream of it.

WALK THIS WAY  ↑ upstream

  ↑  Source system              is the world actually like this?
  ↑  Ingestion / CDC            did every committed change get emitted?
  ↑  Raw landing                is the payload present, and shaped as before?
  ↑  Staging model              did dedup / window / cast keep the period whole?
  ↑  Serving model              does the fact table reconcile for the period?
  ↑  Mart / aggregate           does it match the model it derives from?
  ↑  Metric definition          does the SQL behind the tile mean what we think?
  ↑  BI layer                   does the tool add a filter, join or cached extract?
  ●  Consumer symptom           "revenue is down eighty percent"

  At each rung, one question:
      is the affected period complete and correct HERE?
  First "no" going up  →  the fault is between that rung and the one below it.

• Run the check suite over the affected period first, and read the failing set as a whole. Two minutes of evidence beats twenty minutes of hypothesis, and it tells you which hop to examine first (Data Tests).
• Then walk upstream, one dataset at a time, asking only whether the affected period is complete and correct there. Resist the urge to fix anything you notice on the way — record it and continue (Data Incidents).
• Bound every investigative query to the affected partitions. An unbounded query during an incident is slow when you most need it fast, and expensive in a way that shows up later (Partition Pruning).
• Compare against a known-good period, not against intuition. The same query over last Tuesday is the control, and having a control is the difference between "this looks odd" and "this is different".
• Preserve evidence before repairing: snapshot the affected partitions, or at minimum record the counts and check results, so a re-run cannot erase what you were looking at (Keeping Raw History: The Recovery Position and the Liability).
• Write the cause down as a fingerprint — which checks failed, at which hop, for which periods. That record is what makes the second occurrence a five-minute diagnosis (Reading a Timeline: Observation Order Is Not Causal Order).

• A node that passes every check for the affected period guarantees that this incident did not originate at or above it — for the properties the checks measure. It says nothing about properties nobody checks.
• The fingerprint narrows the cause; it does not prove it. Two distinct faults with overlapping symptoms can produce the same failing set in a real platform, which is why the walk confirms the hypothesis rather than replacing it.
• Finding the hop guarantees you know where, not why. "The transform is wrong" is a location; the cause is a code diff, a schema change or a configuration change, and those are found by a different search (Deploys Are the First Suspect).
• Nothing guarantees a single cause. Incidents with two simultaneous faults are common enough that a diagnosis explaining only part of the discrepancy should be treated as incomplete rather than as done.