Validate at Startup, Fail Loudly
Check everything the process needs at boot and refuse to start — rather than discovering the missing value at 3am on the first request that needs it.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What should a process verify before it declares itself ready, and what should it do when a check fails?
A deploy went out on Friday. On Sunday night, the first refund request of the weekend failed because a payment key was never set in production. It should have been impossible to deploy that build at all.
Read config where it is used and handle a missing value gracefully — fall back to a default, log a warning, and keep the service up. Availability first: never crash on startup.
The failure moves from deploy time, when someone is watching and a rollback is one command, to an arbitrary later moment when nobody is.
- The failure moves from deploy time, when someone is watching and a rollback is one command, to an arbitrary later moment when nobody is.
- It is discovered by a customer. The rarer the code path, the longer the delay and the worse the discovery — refunds, exports, password resets and admin actions are exactly the paths with the longest fuse.
- A graceful fallback often does something wrong quietly: an unset payment mode defaults to sandbox and the service takes real orders that are never charged.
- Debugging is much harder later. At boot the cause is a missing variable; at 3am on Sunday it is an unrelated-looking exception deep inside a handler, with the actual cause three layers up.
- A partially-configured instance passes health checks and joins the load balancer, so a fraction of traffic fails and the pattern looks random (Health Checks: Startup, Readiness, Liveness).
What is actually happening
- A process has a set of preconditions that are either true or false at boot and do not change: config present and well-formed, migrations at an expected version, credentials valid, required directories writable, essential dependencies reachable.
- Every one of those can be checked in a few hundred milliseconds at startup, once, instead of being rediscovered per request.
- Failing fast is a deployment-safety mechanism. A process that exits non-zero during a rolling deploy stops the rollout: the orchestrator sees the new version failing, keeps the old version serving, and the blast radius is one instance instead of the fleet (Rolling Deployments).
- Continuing with a default is the opposite: it converts a loud, contained, deploy-time failure into a quiet, distributed, runtime one.
- There is a real distinction between checks that must fail the process and checks that must only delay readiness. Missing config can never become valid, so exit. An unreachable database may recover in seconds, so retry with backoff and stay not-ready (Liveness vs Readiness).
- The exit must be loud in a machine-readable way: non-zero code, one clear message naming every problem at once, flushed before exit. A crash whose log line was lost in an async buffer is a crash nobody can diagnose.
- Reporting *all* failures rather than the first is a small detail with a large effect: an operator fixes four variables in one iteration instead of four deploys.
Move the failure to where someone is watching
The entire argument for startup validation is a change of *when*, not of *whether*. The configuration is wrong either way. The choice is whether it surfaces during a deploy, with an engineer present, a rollback available and one instance affected — or on a Sunday, on a rare code path, to a customer.
The comparison below is the same missing variable in both columns. Everything that differs is a consequence of when it was checked.
| Discovered at first use | Discovered at startup | |
|---|---|---|
| When | Whenever that path first runs — possibly weeks later | Within seconds of the deploy |
| Who notices | A customer, then support, then an engineer | The engineer running the deploy |
| Blast radius | Every instance, since all of them deployed successfully | One instance; the rollout halts (Rolling Deployments) |
| Symptom | An exception deep in a handler, cause three layers up | PAYMENT_KEY: Required and a non-zero exit |
| Recovery | Diagnose, patch, deploy under pressure | Set the variable and redeploy |
| Worst case | A plausible default did something wrong quietly for a week | A deploy is delayed by ten minutes |
Exit, or retry and stay unready
Not every failed precondition deserves an exit. The dividing line is whether the condition can become true on its own. A missing environment variable cannot; an unreachable database very likely will within seconds.
Getting this backwards produces the two classic startup pathologies: a crash loop caused by a transient blip, and an instance that sits unready forever because a permanently invalid value is being retried patiently.
Can this condition become true without human intervention?
when Missing or malformed config, an unparseable key, an incompatible schema version, a credential that authenticates as the wrong identity.
cost The deploy stops. That is the feature — but it needs an alert, because a crash loop at 2am is silent otherwise.
when A dependency is unreachable: the database is restarting, the secret manager is briefly unavailable, DNS has not converged.
cost A slow start looks like a hang unless you log each attempt and bound the total wait.
when A genuinely optional dependency: analytics, a recommendation service, a non-essential cache.
cost Only correct if the service is genuinely useful without it — and someone must graph the degraded state (The Metrics a Backend Must Emit).
when A required dependency that should be up: give it a fixed window, then treat prolonged absence as a deployment problem.
cost Choosing the window. Too short crash-loops during a database failover; too long stalls the deploy with no signal.
One check function, all failures at once
The implementation detail that operators appreciate most is reporting every problem in one pass. Four missing variables discovered one deploy at a time is four cycles; discovered together it is one edit.
Note the ordering below: pure validation first because it is instant and cannot self-correct, then credential verification, then anything transient. Cheap and decisive checks go first so the common failure is reported in under a second.
1async function boot() {2 // 1. Config — instant, cannot self-correct, report ALL problems together.3 const parsed = ConfigSchema.safeParse(process.env)4 if (!parsed.success) {5 const problems = Object.entries(parsed.error.flatten().fieldErrors)6 .map(([k, v]) => ` ${k}: ${v?.join(', ')}`)7 logger.fatal('configuration invalid:\n' + problems.join('\n'))8 await logger.flush() // or the message dies in the buffer9 process.exit(78) // EX_CONFIG — machine-readable10 }11 const config = Object.freeze(parsed.data)12 13 // 2. Secrets — parsing is not verifying. Does the credential authenticate?14 const secrets = await withBoundedRetry(() => fetchSecrets(config), { maxMs: 30_000 })15 if (!(await paymentClient.verifyKey(secrets.paymentKey))) {16 logger.fatal('PAYMENT_KEY did not authenticate against the configured mode')17 await logger.flush(); process.exit(78)18 }19 20 // 3. Transient dependencies — retry, do not exit on the first failure.21 const pool = await withBoundedRetry(() => openPool(config, secrets), { maxMs: 60_000 })22 23 // 4. Schema version — a mismatch this build cannot tolerate is fatal.24 const version = await currentMigrationVersion(pool)25 if (version < MIN_SUPPORTED_MIGRATION) {26 logger.fatal({ version, expected: MIN_SUPPORTED_MIGRATION }, 'schema too old for this build')27 await logger.flush(); process.exit(78)28 }29 30 await warmCaches(pool)31 32 // 5. Only now do we accept traffic.33 logger.info({ version: BUILD_SHA, config: redact(config) }, 'startup complete')34 startupComplete = true // flips /startupz and /readyz [[health-checks]]35 server.listen(config.PORT)36}Opening the listener last is deliberate: an instance that accepts connections before its pool exists will serve errors that look like application bugs. The listener is the last thing to start and the first thing to stop (Graceful Shutdown).
How to build it
Most important first.
- Parse and validate the entire configuration against a schema as the first thing the process does, before opening a listener (Configuration: Separating Code From Environment).
- Report every validation failure together, with the variable name and what was wrong. Exit with a non-zero status.
- Verify the database schema version matches what this build expects, and refuse to start on a mismatch that the code cannot tolerate (Schema Migrations from the Application Side).
- Fetch secrets at boot and verify they work — a credential that parses but does not authenticate has not been validated (Secrets Are Not Configuration).
- Distinguish hard preconditions from transient ones: exit for anything that cannot self-correct, retry with backoff and stay unready for anything that can.
- Warm what must be warm before signalling ready: pool established, caches primed, JIT paths exercised if that matters for your runtime (Startup Time & Cold Start).
- Log the effective, redacted configuration and the build sha on successful startup. This is the line that answers "what was this instance actually running".
- Run the same validation in CI against the target environment's variable set, so a missing production variable is caught before the deploy rather than by it.
- Keep startup fast enough that failing fast is cheap. A ninety-second boot makes every deploy slow and pushes teams toward skipping checks.
What can go wrong
- Validation that checks presence but not semantics:
DATABASE_URLis a valid URL and points at staging. Presence checks are necessary and weak. - Exiting on a transient dependency failure, so a brief database blip during a deploy causes an infinite crash-loop and the deploy never completes.
- The reverse: retrying forever on a permanently invalid value, so the instance sits unready indefinitely and the deploy stalls with no clear cause.
- The exit log line lost because the logger buffered asynchronously and the process exited before flushing (What a Backend Should Actually Log).
- The mitigation failing: a validation module imported after something else has already read a raw environment variable and cached a wrong value.
- Boot-time checks that hit a rate-limited third-party API on every instance start, so a rolling deploy across many instances trips the limit (Rate Limiting).
- A startup probe grace period shorter than the warm-up, so the orchestrator kills a process that was working correctly (Health Checks: Startup, Readiness, Liveness).
- Startup races with traffic: an instance that opens its listener before the pool is ready receives requests it cannot serve. Open the listener last, or gate on the startup probe (Health Checks: Startup, Readiness, Liveness).
- Concurrent instances running migration checks at boot can race on the schema. Migrations should be a separate, serialized deploy step, not something every instance attempts (Schema Migrations from the Application Side).
- Security-relevant configuration must fail closed. If
REQUIRE_AUTH,TLS_VERIFYor an allowed-origins list is missing or unparseable, refuse to start — never default to permissive (Fail Open vs Fail Closed). - Verify that credentials actually authenticate at boot. A service that starts with a wrong key and fails later leaves a window where behaviour is undefined.
- Do not print the failing *values* in startup errors — name the variable and the expected shape.
PAYMENT_KEY must start with "sk_"is helpful; echoing the received value is a leak (Not Leaking Your Internals). - Refusing to start on a schema-version mismatch is a security control as well as a correctness one: code running against an unexpected schema can bypass constraints it assumes exist (Database Constraints).
- Validate that you are pointed at the environment you think you are. A production build connected to a staging database, or worse the reverse, is a data-integrity incident with no error message.
- "Crashing on startup hurts availability." A process that starts misconfigured hurts availability more, silently, and for longer. An orchestrator that keeps the old version running loses nothing.
- "Graceful degradation means never crash." Degradation applies to runtime dependencies. A missing required credential is not a degraded state; it is a broken build.
- "Validation at startup is redundant with CI." CI validates the code. It does not validate the production environment's variables unless you deliberately make it.
- "If it starts, it is configured correctly." It means nothing was checked, or the checks were shallow. Starting is evidence about your checks, not about your config.
- "Retry forever is safer than exiting." For a transient dependency, yes. For a malformed value, it is an instance that will never become ready and never say why.
Operating it
- Alert on process exits during a deploy window with a non-zero code and a startup-validation message. That alert should be near-silent and extremely informative when it fires.
- Track time-from-start-to-ready per instance. A creeping value predicts the day a probe threshold becomes too tight.
- Emit a build-info metric at startup labelled with version and the low-cardinality config values that matter, so a dashboard shows what each instance is running.
- Count crash-loop restarts. A crash loop caused by validation is correct behaviour and still needs an alert, because nobody is watching the deploy at 2am.
- Compare the set of configured variables between instances of the same deployment. A difference means a partial rollout or a stale instance (Configuration: Separating Code From Environment).
- At 10x instances, startup checks that touch a shared dependency become a thundering herd during a rolling deploy — jitter them or make them cheap (Backoff and Jitter).
- At larger fleets, failing fast becomes more valuable, not less: it stops the rollout automatically, which is the mechanism that keeps a bad config from reaching every instance.
- With autoscaling, startup time is directly on the path from "load spike detected" to "capacity available". A slow validating boot makes autoscaling react late (Autoscaling a Backend).
- Serverless inverts the calculation: a cold start pays the boot cost on a request, so expensive validation moves from a deploy-time cost to a per-cold-start latency cost (Serverless Backends).
- Failing fast means a typo can stop a deploy. That is intended, and it is genuinely inconvenient at the wrong moment.
- Boot-time dependency checks add startup latency and a coupling to that dependency's availability at exactly the moment you are trying to deploy.
- Thorough validation is code to write and maintain, and it duplicates knowledge that also lives in the deployment configuration.
- Strictness can be over-applied: refusing to start because an optional analytics endpoint is unreachable trades availability for tidiness.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALFail-fast-at-boot applies to any long-lived process in any language.
- CLOUD-SPECIFICThe value depends on what the platform does with a non-zero exit. Kubernetes will not progress a rolling update past a crash-looping new ReplicaSet, so failing fast is genuinely protective. A plain process manager that restarts forever gives you a crash loop and no rollback, so pair it with an alert. Serverless platforms may retry initialisation per invocation, turning a config error into per-request failures rather than a stopped deploy.
- RUNTIME-SPECIFICFlushing before exit differs: Node needs an explicit await on the logger flush because
process.exit()discards buffered async writes; Python's logging flushes handlers at interpreter shutdown for normal exits but not on os._exit; Go's os.Exit skips deferred functions entirely. The failure — a crash with no log line — looks identical in all three and has three different fixes.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Testing & Reliability Engineering — asserting that the process refuses to start on each class of invalid configuration is a small, high-value test suite that almost nobody writes.