We found a transformation bug affecting six months of data. What do you do?
Whether the candidate separates fixing the code from repairing the history, and whether they treat a backfill as a production change with capacity and blast radius rather than as a re-run.
The situation behind the question
Interviewers ask this because it happened to them.
A revenue model has been failing to subtract refunds since a change six months ago. The figures have been used in monthly reporting throughout. The bug is understood and the fix is two lines.
A strong answer
Flags
Green flags
- Builds re-runnable pipelines and knows that the ability to repair depends on retained raw and a deterministic transformation.
- Treats the backfill as a bulk production write with its own capacity planning and blast radius.
- Validates against a control period the bug never touched, not only against the corrected one.
- Handles the communication: a restatement that is not announced generates a second incident when old slides meet new dashboards.
Red flags
- "A backfill is just rerunning the DAG." It is a bulk write over a range nobody has validated, against data consumers are actively reading, competing for the same compute.
- Fixes the code and considers the incident closed, leaving a permanent notch in the history.
- Runs the correction against production tables in place, with no staging location and no way back.
- Does not ask whether the raw inputs for those months still exist.
Follow-ups
Where the conversation goes if the first answer holds up.
- Raw for the first two of those months has aged out of retention. What changes about your answer?
- The backfill will be the largest job the platform runs this month. How do you keep it from becoming its own incident?
- Finance has already filed figures derived from the wrong number. Does that change the technical plan, or only the communication?