Timezone and DST Failures
The hour that happens twice, the hour that never happens, the billing cutoff in the wrong zone, and the incident timeline nobody can reconcile.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What actually goes wrong twice a year, and why does it survive every code review?
Daylight saving transitions and multi-zone operation break assumptions that are never written down: that every hour occurs once, that local times sort, and that "midnight" is a single instant.
We schedule the nightly job at 02:00 local time so it runs during the quiet period, and we store the business day in local time because that is what the finance team reports on.
In autumn, 02:00 local occurs twice. A job scheduled at that local time can run twice, and if it is not idempotent it double-processes (Operating Queues and Scheduled Work).
- In autumn, 02:00 local occurs twice. A job scheduled at that local time can run twice, and if it is not idempotent it double-processes (Operating Queues and Scheduled Work).
- In spring, 02:00 local does not occur at all. The same job silently does not run, and a failure alert cannot fire for a run that never started.
- Billing and reporting cutoffs defined in a local zone put transactions on the wrong day for anyone in another zone, and reconciliations then differ by exactly one boundary.
- Incident timelines assembled from services in different zones cannot be ordered, and the reconstruction takes longer than the incident did (Reconstructing What Actually Happened).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A daylight saving transition changes the offset between local time and the instant timeline. Nothing about the physical timeline changes; only the mapping does.
- When the clock moves back, an hour of local time is repeated: the same wall-clock string maps to two distinct instants. When it moves forward, an hour is skipped: some wall-clock strings map to no instant at all.
- A scheduler that matches on local wall time therefore fires twice for the repeated hour and zero times for the skipped one, unless it explicitly handles both — and schedulers differ in whether they do.
- Adding calendar units is not the same as adding elapsed time. "Tomorrow at 09:00" is a calendar operation on local time; "twenty-four hours from now" is an operation on the instant timeline. Across a transition they differ by an hour.
- Zone rules are political and versioned. A country can change its rules with limited notice, which means correct behaviour depends on a data file being current, not only on correct code (Production Time Is UTC).
The hour that happens twice
This is the autumn transition, written on the local clock, which is how the scheduler sees it. Every row is a real wall-clock reading; the instants behind them are an hour apart.
A job matching on local wall time fires at both of the highlighted rows. If it aggregates the previous day, the day is counted twice.
- The defence against the autumn case is a run key on the logical period, so the second trigger is a no-op.
- The defence against the spring case is asserting that a run happened for each expected period.
- 01:30signalOrdinary operation; offset is still the summer one
- 02:00actionNightly job triggers on local wall time — first execution
- 02:30recoveryJob completes; aggregates yesterday; writes results
- 02:59changeClocks move back; the offset changes
- 02:00changeThe same wall time occurs again, one hour later on the instant timeline
- 02:00actionNightly job triggers again — second execution, same logical period
- 02:30signalAggregation applied a second time; totals now doubled
- weeks latersignalFinance reconciliation finds the discrepancy; nobody connects it to the transition
In spring the same schedule produces the opposite failure: the 02:00 row never occurs, the job does not run, and no failure alert fires because there was no run to fail.
Four failures, one root assumption
The shared assumption is that local wall time is a reliable coordinate. Each row is what happens when a different subsystem relies on it.
| Failure | What breaks | Why review misses it | The fix |
|---|---|---|---|
| Duplicate run | Autumn: the trigger time occurs twice | The code is correct on 364 days | Schedule in UTC; idempotent run key per logical period |
| Missed run | Spring: the trigger time does not occur | A run that never started cannot fail | Assert expected versus actual runs and alert on absence |
| Wrong billing boundary | A cutoff in one zone applied to users in another | Correct for whoever wrote it | Define the boundary once with an explicit zone; apply it everywhere |
| Unorderable incident timeline | Events from services in different zones | Each service is internally consistent | UTC with explicit offsets in every log line (Production Time Is UTC) |
Calendar arithmetic is not elapsed time
The second most common source of these bugs, after scheduling, is date arithmetic. The two operations agree for most of the year, which is exactly why the difference is never noticed until it matters.
1Calendar arithmetic "same wall time, next day"2 input 2025-10-25 09:00 Europe/Berlin3 add 1 day (calendar)4 result 2025-10-26 09:00 Europe/Berlin5 elapsed 25 hours, because the offset changed overnight6 7Elapsed arithmetic "24 hours from now"8 input 2025-10-25 09:00 Europe/Berlin (an instant)9 add 24 hours (duration)10 result 2025-10-26 08:00 Europe/Berlin11 elapsed 24 hours, wall time shifted by one12 13Both are correct operations. They answer different questions:14 15 trial ends "in 14 days" -> calendar, in the user's zone16 token valid "for 1 hour" -> elapsed, on the instant timeline17 report covers "yesterday" -> calendar, in the defined business zone18 retry "after 30 seconds" -> elapsed19 reminder "tomorrow at 09:00" -> calendar, stored as local + zone id20 21Choosing the wrong one is invisible for most of the year and then22shifts a user-visible boundary by exactly one hour.Write down which of the two each business rule means, next to the rule. That sentence is what a reviewer needs and almost never has.
How to do it properly
Most important first.
- Schedule everything in UTC. Where a job genuinely must run at a local wall time, make it explicit that it will occur twice or zero times per year and handle both.
- Make every scheduled job idempotent on a run key derived from the intended logical period, not from the wall clock, so a duplicate trigger is a no-op (Job Idempotency in Backend Engineering).
- Alert on missed runs by asserting expected-versus-actual executions — the only way to detect a run that never started (Job Scheduler Reliability).
- Define business boundaries once, with an explicit zone, and use that definition everywhere: billing cutoffs, daily reports, retention windows, trial expiry.
- Use calendar arithmetic for calendar semantics — "the same time tomorrow" — and elapsed-time arithmetic for durations. Never substitute one for the other.
- Test explicitly across both transitions, with a fixed clock at the boundary, rather than waiting for the calendar to test in production.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Idempotent run keys contain the duplicate case entirely, and missed-run alerting contains the skipped case. Without both, the damage is bounded only by what the job touches, which for an aggregation job is everything it aggregates.
What can go wrong
- A nightly aggregation runs twice and doubles a day's figures, discovered in a reconciliation weeks later.
- A nightly aggregation does not run, and the gap is only noticed when a downstream report is empty.
- A trial or subscription boundary computed by adding hours expires at the wrong local time for a cohort of users.
- An incident timeline is assembled from mixed zones and events appear out of order, sending the investigation the wrong way.
- "Our scheduler handles DST." Some do, some do not, and they differ in *how*: skipping, running twice, or firing immediately after the skipped hour. Find out which behaviour yours has rather than assuming it is the one you want.
- "We are in a zone without daylight saving, so this does not apply." Your users, your third-party APIs and your cloud provider may not be, and zone rules can be introduced or removed by legislation.
- "Adding 24 hours is the same as tomorrow." Not across a transition, and not for anything a user will read as a wall time.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Scheduler configuration expressed in UTC, verifiable by reading it.
- A record of runs per logical period showing exactly one execution per period across both transitions.
- Tests that pin the clock to both transition boundaries and assert on job triggering and on boundary arithmetic.
- A duplicate run is recoverable only if the work was idempotent or if you can identify and undo the second application — which is a data recovery operation, and not a cheap one (Partial and Logical Data Recovery).
- A missed run is usually recoverable by triggering it manually for the intended period, provided the job accepts a period parameter rather than deriving it from "now". Design that parameter in.
- Automate: single-flight locks, run-key idempotency, expected-versus-actual run assertions, and clock-pinned tests at both transitions in CI.
- Automate a pre-transition check — a scheduled review of jobs configured in local zones, run before each transition date.
- Keep human: deciding the zone semantics of a business boundary, and deciding whether a duplicated or missed run needs a correction or can be left.
- UTC scheduling means a job's local run time shifts by an hour twice a year, which occasionally conflicts with a genuine local constraint such as a business quiet period.
- Run-key idempotency needs a store of processed periods — small, permanent state in exchange for tolerating duplicate triggers.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICScheduler behaviour at transitions genuinely differs: some cron implementations run a skipped job immediately after the gap, some skip it entirely, and some run a repeated hour twice. Managed schedulers document their own choice. This is one of the few places where you must read your specific tool's behaviour rather than reason from the general model.
- GENERALThe repeated hour and the skipped hour are properties of daylight saving itself, so any system that schedules or reasons in local wall time is exposed regardless of stack.
- ORG-SPECIFICWhich zone defines a business day is an organisational and sometimes a regulatory decision. Engineering owns making the definition explicit and consistent, not choosing it.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — pinning the clock in tests so a once-a-year failure is reproducible on demand rather than in production.