OperationsGENERALDATABASE-SPECIFICTOOL-SPECIFIC

Production Time Is UTC

Machine timelines, logs, storage and schedules in UTC; local time only at the edges where a human reads it. The conversion belongs in one place.

What actually happens How to do it properly Evidence and rollback

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

Why does every experienced operator insist that production runs on UTC?

The problem

Correlating events across services is only possible if their timestamps mean the same thing. Local time makes two records with the same string refer to different instants, and one instant unrepresentable twice a year.

What teams do first

Store and log times in the timezone the business operates in. It is easier to read, and everyone here is in the same place anyway.

How it breaks

Two log lines an hour apart can carry the same local timestamp during a daylight saving transition, and an incident timeline built from them is wrong in a way that looks right.

How it breaks in production

Two log lines an hour apart can carry the same local timestamp during a daylight saving transition, and an incident timeline built from them is wrong in a way that looks right.
The moment a second region, a third-party API or a remote colleague appears, timestamps stop being comparable and every correlation becomes a conversion argument.
Durations computed by subtracting local times are wrong by an hour twice a year — sometimes silently, sometimes as a negative duration.
Sorting by a local timestamp does not sort by time. During the repeated hour, order is genuinely ambiguous.
Business boundaries — a billing cutoff, a daily report, a retention window — computed in local time land on the wrong instant for anyone outside that zone.
A timestamp without an offset is not a time. It is a string that requires context nobody wrote down.

Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

An instant is a point on the physical timeline. A local time is an instant plus a zone rule, and zone rules change: they are political decisions with a version history, distributed as a database that must be updated.
The conversion from instant to local time is lossy in both directions during transitions: one local time can map to two instants (the repeated hour), and one local time can map to none (the skipped hour).
UTC has no transitions, so instants are totally ordered, subtractable and comparable. That is the entire argument — it is not about preference or convention.
Storing an offset is not the same as storing a zone. An offset tells you what the clock read; a zone tells you what the rules were and lets you compute future local times correctly when rules change.
For future events with human meaning — "the meeting at 09:00 in Berlin" — the correct storage is the local time plus the zone identifier, because the rules may change before the event arrives. For past events, the instant is what happened.
Every layer that reformats a timestamp is a place the zone can be lost: a driver, a serialiser, a log shipper, a dashboard. The defence is one conversion point at the display edge and UTC everywhere behind it.

Instant, local time, and the thing in between

GENERALThe distinction holds in every language and database. What differs is which of these your default type actually is — and the default is frequently the last row.

Most time bugs come from conflating three different concepts that all get called "a timestamp". Separating them makes the storage decision obvious.

Concept	What it is	Store it when	Failure if misused
Instant (UTC)	A point on the physical timeline	Recording something that happened	None — this is the safe default for the past
Local time + zone id	Wall time plus the rules to interpret it	A future human appointment or a local business boundary	Storing an instant instead breaks when zone rules change
Local time + offset	Wall time plus what the clock read then	Recording what a human saw, alongside the instant	Treated as a zone, it produces wrong future conversions
Naive local time	A string with no zone information	Never, in production storage	Ambiguous during the repeated hour; incomparable across hosts
Duration	An elapsed length, not a point	Measuring how long something took	Computed by subtracting local times, it is wrong across transitions

One conversion point, at the edge

The architectural rule is that time is UTC everywhere internally and becomes local exactly once, where a human reads it. Every additional conversion point is a place a zone can be lost or applied twice.

Where the conversion lives

Converted wherever convenient

# each layer decides for itself
db:      naive timestamp column, host zone at write time
api:     serialises without an offset
worker:  os.environ.get("TZ") or system default
cron:    schedule in business local time
logs:    host local time, no offset
report:  converts again, "to be safe"

UTC inside, local at the edge

# one boundary, everything behind it is instants
db:      zone-aware timestamp, always UTC
api:     serialises instants with an explicit offset
worker:  container TZ=UTC; zone-aware types only
cron:    schedule expressed in UTC
logs:    UTC with explicit offset in every line
report:  ONE render function converts to the viewer zone,
         using the zone id, at display time

On the left, every hop can lose or reapply a zone, and the errors compound into stored data where they are permanent. On the right there is a single place to review, a single place to test, and a single place where a daylight saving transition can affect anything — and it affects only what a human reads, never what is stored or compared.

Where the zone gets lost

These are the specific hops where a correct UTC value becomes an incorrect local one. Each is mundane, which is why they survive review.

Zone loss by layer

Trigger	Symptom	Cause	Response
Column declared as a naive timestamp type	Rows from different hosts are an hour apart for the same event	Each writer applied its own host zone	Use a zone-aware type; migrate with expand and backfill (Expand, Migrate, Contract)
Base image changed	Times shift with no application change	The host timezone default changed underneath a library that uses it	Pin TZ=UTC in the image and assert it at startup (Validate at Startup, Fail Clearly)
JSON serialisation between services	Receiver interprets a bare timestamp in its own zone	The offset was dropped in transit	Require an explicit offset in the schema and reject values without one
Report or export built by a different team	Totals differ from the dashboard by one day at the boundary	Two different definitions of when the day starts	Define business boundaries once, with an explicit zone, and share the definition
Zone database not updated	Local times wrong after a country changes its rules	Stale zone data in the runtime or image	Treat zone data as a dependency with an update cadence (Dependency Management)
Duration computed from stored local times	Negative or inflated durations twice a year	Subtraction across a transition	Compute durations from instants only, never from wall times

How to do it properly

Most important first.

Store instants in UTC, with an explicit type that carries the zone or offset rather than a naive local timestamp.
Log in UTC, in a format that includes the offset explicitly, so a log line is unambiguous when read out of context.
Run schedulers, cron and job timers in UTC, and translate to local time only for display (Timezone and DST Failures).
Set server and container timezones to UTC so that anything that ignores your application settings still produces UTC.
Convert at the display edge only, using the viewer's zone, and keep exactly one place in the code that does it.
For future human appointments, store the local wall time plus the zone identifier, and compute the instant at use — not at write.
Keep zone data updated as a dependency. It changes with political decisions and an outdated copy produces wrong local times (Dependency Management).
Put UTC on operator dashboards, or show both — an incident timeline that mixes zones costs minutes exactly when they are expensive (Reconstructing What Actually Happened).

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongEveryone

One testEveryone

What contains it

A single conversion point at the display edge contains it. Scattered conversions do not — the damage is spread across stored data and only becomes visible later (Partial and Logical Data Recovery).

What can go wrong

Failure modes, including of the mitigation

A naive timestamp column stores whatever zone the writing process happened to have, so records from different hosts are incomparable.
A library defaults to the host timezone, and a container image change alters behaviour with no code change.
Durations computed across a transition produce a negative or inflated value that propagates into billing or metrics.
Zone data is stale in a base image, so times are correct until a country changes its rules.
Serialisation drops the offset between services, and the receiver applies its own default.

Misreads this invites

"We only operate in one country, so local time is fine." Zone rules still change, the repeated hour still exists, and your third-party APIs and cloud provider are already reporting in UTC.
"Storing an offset is the same as storing a zone." An offset is a snapshot of a rule. Future local times need the rule itself.
"UTC and GMT are interchangeable." They are close enough for most application purposes and are not the same definition; use UTC and let the platform handle leap seconds (Clock Synchronisation).
"The database handles timezones." Some types store an instant and some store a naive value that merely looks like a time. Check which one your column actually is.

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked

Logs from every service carry an explicit offset and agree with each other during a correlation exercise.
Timestamp columns use a zone-aware type, verified by inspecting the schema rather than by convention.
Scheduler configuration is expressed in UTC and reviewed as such.
An incident timeline can be assembled from multiple services with no manual conversion step (The Debugging Timeline).

How you get back

Migrating stored timestamps to UTC is a data migration with the same discipline as any other: expand with a new zone-aware column, dual-write, backfill with an assumed source zone, verify, then contract (Expand, Migrate, Contract).
The backfill is the risky part, because it requires assuming which zone historical naive values were written in — and that assumption may be wrong for some rows.
Reverting a display-layer change is trivial; reverting a storage change after new data has arrived is not. Keep the conversion at the edges so most changes stay reversible.

What to automate, and what stays human

Automate: linting for naive timestamps and for host-timezone-dependent calls, a CI check that container timezones are UTC, and zone data updates in base images.
Automate the display conversion in one shared component, so no individual feature has to make the decision.
Keep human: choosing the zone semantics for a business rule — a billing day, a cutoff, a retention window — because that is a product decision about which humans the boundary belongs to.

What this costs

UTC everywhere is harder to read for humans on the operations side, and every dashboard needs a conversion or a dual display.
Storing local time plus a zone for future events is correct and more work than storing an instant, and it is the only approach that survives a rule change.

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

GENERALUTC for machine timelines is universal practice across stacks and providers. The only genuine exceptions are user-facing display and business rules that are defined in a specific locality.
DATABASE-SPECIFICTimestamp types differ in a way that matters: some store an instant and normalise to UTC, some store a naive value with no zone at all, and the names are misleadingly similar between engines. Read your engine's documentation for the specific type rather than assuming the behaviour transfers.
TOOL-SPECIFICLanguage and library defaults vary — some default to the host zone, some to UTC, some depend on an environment variable. A container image or base image change can therefore alter time behaviour with no application change.

Where the depth lives

This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.

Backendscheduled-jobs

Observability & Performanceincident-timeline

Securityaudit-logs

Architecturelogs-metrics-traces

Domains that do not exist yet

— Distributed Systems — clocks as an ordering mechanism: logical clocks, causality and why timestamps from different machines cannot establish happened-before. This lesson is about human-readable time, not about ordering.