Job Scheduler Reliability
Five questions that decide whether scheduled work survives real infrastructure: can it run twice, can it overlap, what if the machine dies, can it retry, is it idempotent.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Under what guarantees is my scheduled work actually running, and does the job hold up under the ones I actually have?
Every scheduler makes a weaker delivery guarantee than people assume, and the gap between the assumed guarantee and the real one is where duplicate charges, doubled emails and corrupted aggregates live.
Choose a scheduler that guarantees the job runs exactly once, and then the job does not have to worry about any of this.
Exactly-once execution of a side effect is not something a scheduler can provide. It can deliver a trigger at most once or at least once; whether the *effect* happens once is a property of the job.
- Exactly-once execution of a side effect is not something a scheduler can provide. It can deliver a trigger at most once or at least once; whether the *effect* happens once is a property of the job.
- The failure that breaks the assumption is mundane: the job completed its work and the process died before recording that it did. From the scheduler's side that is indistinguishable from never having run.
- A lease-based scheduler must decide what to do when a lease expires and the holder is unreachable. Assume it is dead and you may get two runners; wait for certainty and the job may never run again.
- At-most-once sounds safer until the run that mattered is the one that got dropped — a missed reconciliation window is not obviously better than a duplicated one.
- Retries are usually configured by someone who has not asked whether the job is safe to retry, so the safety mechanism becomes a multiplier (Retry Storms: The Load You Generated Yourself).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The scheduler and the job split the responsibility, and neither can do the other's part. The scheduler decides *when* and *how many times a trigger is delivered*. The job decides *what happens when a trigger arrives twice*.
- At-most-once means a trigger may be dropped and is never duplicated. At-least-once means it is never dropped and may be duplicated. Effectively-once is at-least-once delivery plus an idempotent job, and it is the only one of the three that is achievable end to end (Idempotency).
- Distributed schedulers coordinate with a lease: a runner claims the job for a bounded time and renews while working. The lease is a bet about liveness, and if a runner pauses long enough — a long collection pause, a network partition, a suspended host — its lease expires while it is still alive and still working.
- That produces two live runners for the same job, and no amount of lease tuning removes the possibility. A shorter lease makes it more likely; a longer one delays recovery when a runner genuinely dies. There is no setting that gives both.
- Which is why the durable answer is at the job: fence the work with a token that makes a stale runner's writes fail, or make the operation idempotent so a second execution converges to the same state (Job Idempotency).
The five questions
Ask these before writing the job, and write the answers where the job is. They are cheap to answer at design time and expensive to answer during the incident where the answer turns out to be "yes, and we did not handle it".
- 1Can it run twice?
Determines whether duplicate effects are possible at all.
fails by Assuming the scheduler prevents it, which it can only do for trigger delivery.
evidence Two deliberate concurrent invocations produce one set of effects.
- 2Can it overlap itself?
Asks whether run N+1 can start while run N is still working.
fails by Interval set once against a runtime that grows with data volume.
evidence Run duration is tracked and stays well under the interval, with an alert if it does not.
- 3What if the machine dies mid-run?
Decides whether partial work is recoverable, resumable or must be redone.
fails by No checkpoint and no run record, so nobody can tell what was completed.
evidence A killed run followed by recovery reaches the correct end state.
- 4Can it retry safely?
Establishes whether a failed attempt can simply be repeated.
fails by Retries enabled by default on work with non-idempotent external effects.
evidence A forced failure mid-run, followed by a retry, produces one set of effects.
- 5Is it idempotent?
The question that makes the previous four survivable.
fails by Idempotent for the database and not for the email, the webhook or the payment.
evidence Replaying the same work unit twice, end to end, changes nothing the second time (Job Idempotency).
The fifth question subsumes most of the others. A genuinely idempotent job survives duplicate delivery, partial failure and retries — which is why "make it idempotent" is a better use of effort than tightening the scheduler's guarantees.
The lease expires while the runner is still alive
This is the failure that makes single-execution guarantees impossible in a distributed system, and it is worth being able to picture. Nothing crashed. One runner simply stopped observing time for long enough that its lease expired.
The countermeasures are all at the job or the storage layer, because by the time the second runner starts, the scheduler has already made every decision it can make.
Three guarantees, and what each demands of the job
Choosing a delivery semantic is really choosing which problem you would rather solve. Each row states what the scheduler provides and what remains yours, and the third row is not a stronger guarantee from the scheduler — it is the second row plus work in the job.
- Derive the idempotency key from the work — period, entity, operation — never from the attempt. A key generated per attempt deduplicates nothing.
- Record the key and the effect in one transaction. Recording it separately reintroduces the gap you were closing, at a smaller width.
- External effects that cannot be deduplicated at the far end need the intent recorded first and the effect performed from that record (The Transactional Outbox).
- Test it by replaying the same work unit twice on purpose. An untested idempotency claim is a hypothesis about the code, and this is the class of hypothesis that gets falsified in production by a customer.
| Semantic | Scheduler provides | The job must | Choose when |
|---|---|---|---|
| At-most-once | A trigger is never duplicated; it may be dropped | Tolerate a missed run, or detect the gap and cover it next time | A duplicate is worse than a miss, and the next run naturally covers the gap |
| At-least-once | A trigger is never dropped; it may be duplicated | Be idempotent, or every duplicate becomes a duplicate effect | A miss is worse than a duplicate — which is most reconciliation, billing and sync work |
| Effectively-once | Nothing extra; this is at-least-once | Deduplicate on a key derived from the work, transactionally with the effect (Idempotency Keys: The Mechanism) | Always, in practice. It is the only end-to-end achievable version and the work is in the job |
How to do it properly
Most important first.
- Answer the five questions explicitly for each job, and write the answers next to the job: can it run twice, can it overlap, what happens if the machine dies mid-run, can it retry, is it idempotent.
- Make the job idempotent by design. Derive an idempotency key from the work itself — the period being processed, the entity and the operation — and record it transactionally with the effect (Idempotency Keys: The Mechanism).
- Where an external side effect cannot be made idempotent, record the intent transactionally and perform the effect from that record, so a duplicate execution finds the intent already satisfied (The Transactional Outbox).
- Use fencing tokens where the platform provides them: a monotonically increasing lease number that the storage layer rejects if it is stale.
- Bound retries with a limit and backoff, and send exhausted work somewhere a human will see it rather than dropping it (Dead Letter Queues Are an Operation).
- Record run history — attempt, outcome, duration, what was processed — because the answer to "did this already run" must come from data, not memory.
- Test the failure explicitly: kill a runner mid-execution and confirm the end state after recovery is correct.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Duplicate execution of a job that mutates shared state or calls external systems affects everything the job touches. Contained by idempotency at the job, fencing at the storage layer, and bounded batches so a bad run is identifiable and reversible as a set.
What can go wrong
- Two runners active because a lease expired under a pause rather than a death, both writing.
- A job completed and its completion record lost, so the next scheduling cycle runs it again.
- Retries configured on a job that is not idempotent, turning one transient failure into repeated side effects.
- A job that is idempotent for its database writes and not for the email it sends or the payment it submits — partial idempotency is the common shape and the dangerous one (Webhook Idempotency).
- A lease held by a dead runner with no expiry, so the job silently stops running until someone clears the lock.
- Exhausted retries dropped silently, so failed work disappears with no record.
- Idempotency keys derived from a timestamp at execution time rather than from the work, so a retry generates a new key and the deduplication never triggers.
- "Our scheduler guarantees exactly-once." It guarantees something about trigger delivery. Exactly-once *effects* are produced by the job being idempotent, and no scheduler can supply that.
- "We use leader election, so only one runner can be active." Leader election guarantees at most one runner *believes* it is leader at a time under the algorithm's assumptions. A paused leader that has not yet noticed it lost the lease is still executing.
- "The job is idempotent because re-running it produces the same rows." Check the whole effect surface: the emails, the webhooks, the payments, the messages published. Database idempotency with a non-idempotent external call is the common and expensive shape.
- "Retries make it reliable." Retries make transient failures survivable. On a non-idempotent job they make one failure into several effects.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Killing a runner mid-execution and letting recovery proceed produces the correct end state, verified rather than assumed.
- Two runners started deliberately produce one set of effects.
- Run history shows attempts and outcomes, and a duplicate attempt is visible as a deduplicated no-op rather than as a second effect.
- Exhausted retries appear somewhere a person looks.
- Duplicate side effects on external systems often cannot be rolled back — a sent email stays sent. The design must prevent them rather than plan to reverse them.
- For internal state, a run identifier attached to every row a run touched is what makes reversal of a bad run possible as a set.
- Stopping the schedule must not require a deploy, and clearing a stuck lease must be a documented operation rather than an improvisation (Runbooks).
- Automate lease management, retry with backoff, dead-lettering and run recording — these are exactly the mechanical parts, and they are where hand-rolled schedulers get it wrong.
- Do not automate the reprocessing of dead-lettered work without a human deciding it is safe. Work that failed repeatedly failed for a reason that a retry did not fix (Dead Letter Queues Are an Operation).
- Do not rely on automation to compensate for a job that is not idempotent. Retry logic on a non-idempotent job is a multiplier on the failure, not a mitigation (The Automation Trap).
- Idempotency requires storing keys or markers, which is extra state with its own retention and growth.
- Shorter leases recover faster from a dead runner and produce more false expiries under load; longer leases do the reverse. This is a genuine trade with no correct setting.
- Fencing requires support from the storage layer. Where it does not exist, the fallback is idempotency plus accepting a window of possible duplication.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe five questions apply to any scheduled or queued work anywhere. What differs is which of them the platform answers for you — and every platform that claims to answer the duplicate-execution question answers it about trigger delivery, not about your job's side effects.
- TOOL-SPECIFICSome schedulers expose fencing tokens or generation numbers that storage can reject as stale; some expose only a lease with a timeout, leaving the stale-writer problem entirely to you. Find out which yours does before designing around a guarantee, because the two require different job designs.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.