Operating Dependencies

The day-to-day of running databases, queues, caches and scheduled jobs: connection budgets, dead letters, hot keys, and why production time is always UTC.

Operating a Production Database

The standing duties around the one component you cannot restart your way out of: connections, locks, bloat, replication, and change discipline.

Q · What does owning a production database actually require, beyond it being up?
The Connection Budget

A database accepts a finite number of connections. Every instance, worker, job and console session spends from the same pool — so the pool sizes have to add up.

Q · If the database allows N connections and we run M instances, what may each instance's pool be set to?
Operating Queues and Scheduled Work

Depth, oldest message age, consumer throughput, failure rate and dead letters — plus the clock-driven cousin, where duplicate and missed runs live.

Q · What do you watch on a queue, and what does each signal mean when it moves?
Dead Letter Queues Are an Operation

A DLQ needs an alert, an inspection path, a replay strategy and an owner. Without those four it is a place failures go to be forgotten.

Q · What happens to a message after it fails for the last time — and who finds out?
Operating a Cache

Hit rate, memory, evictions, hot keys and latency — plus the planning question that decides your real architecture: can the system survive losing the cache?

Q · If the cache disappeared right now, would the system stay up?
Production Time Is UTC

Machine timelines, logs, storage and schedules in UTC; local time only at the edges where a human reads it. The conversion belongs in one place.

Q · Why does every experienced operator insist that production runs on UTC?
Timezone and DST Failures

The hour that happens twice, the hour that never happens, the billing cutoff in the wrong zone, and the incident timeline nobody can reconcile.

Q · What actually goes wrong twice a year, and why does it survive every code review?
Clock Synchronisation

Machines need reasonably synchronised clocks for logs, certificates, tokens and scheduling — reasonably, because perfect synchronisation is not available.

Q · How closely do production clocks need to agree, and what breaks when they drift?