← PracticeIntermediateStructured

The Job That Reported Success and Wrote 94% of the Rows

Pull up the evidence one item at a time, commit to a diagnosis, and only then see the schedule that actually ran.

What was reported

Our nightly reconciliation job logs "processed 240,000 records", exits 0, and the scheduler marks it green. Between 3% and 8% of the records are not in the database the next morning. No errors, no warnings, no failed batches — the log is completely clean. The missing records are different every night. Downstream reporting has been quietly wrong for about two months.
1async function run() {
2  const batches = chunk(await loadRecords(), 500)   // 480 batches
3  let processed = 0
4  for (const batch of batches) {
5    void writeBatch(batch)        // not awaited — "it is faster this way"
6    processed += batch.length
7  }
8  logger.info('processed ' + processed + ' records')
9}
10 
11await run()
12process.exit(0)                   // exits with writes still in flight

Evidence

Nothing here is labelled as relevant. Some of it is not.

What is actually happening?