Validate at Startup, Fail Clearly
Check every configuration value when the process starts and refuse to serve if anything is wrong — rather than discovering an invalid value at 3am, on the first request that happens to need it.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
When should a bad configuration value be detected, and what should happen when it is?
Configuration is read lazily by default. A value that is missing, misspelled, out of range or of the wrong type causes no error until the first code path that needs it runs — which may be hours after deploy, on a rare path, under load, at night.
Read values where they are used. process.env.PAYMENTS_TIMEOUT_MS at the call site, with a sensible fallback so nothing crashes.
The failure is separated from the change by hours or days. The deploy looked healthy, the canary was clean, and the error arrives on the first refund request at 3am — with nothing on the timeline to connect it to (Change Correlation).
- The failure is separated from the change by hours or days. The deploy looked healthy, the canary was clean, and the error arrives on the first refund request at 3am — with nothing on the timeline to connect it to (Change Correlation).
- A fallback turns a misconfiguration into a silent behaviour change. A missing timeout becomes the library default, which may be no timeout at all, and the service hangs rather than fails (How Networks Fail in Production).
- Every environment variable is a string. Without coercion and range checks,
PAYMENTS_TIMEOUT_MS=3O(letter O) becomesNaN,0, or a thrown error deep in a payment call, depending on the language. - A misspelled key is invisible: nothing reads
DB_POOL_MAXX, so the intended change never takes effect and the old value keeps working, which looks like success. - Lazy reads mean a rolling deploy can put instances into service that will fail later, so the rollout completes green and the failure appears after the deploy is considered done (A Successful Deploy Is Not Evidence of a Healthy System).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- There are two distinct times a configuration error can surface: startup, when nothing depends on the process yet and the platform can refuse to route traffic to it, and first use, when a user request is already in flight.
- Startup is the cheap moment. The instance has not been added to a load balancer, the previous version is still serving, and a crash on boot is a contained, visible, well-understood event that a rolling deploy already knows how to stop (Rolling: Two Versions, One Database).
- So validation is really about moving the failure earlier, from an unpredictable time under load to a predictable one where an existing safety mechanism catches it.
- Validation has three layers: presence (is the key there), type and coercion (is the string a number, a boolean, a URL), and semantics (is the number in a range, is the URL reachable, is this combination of values legal). Most implementations do the first and stop.
- The message is part of the mechanism. A process that exits with "invalid configuration" teaches nothing at 3am. One that lists every failing key, its value, the constraint it violated and the layer it came from turns an incident into a one-line fix (Validation Errors: Feedback, Not Verdicts).
Two moments a bad value can be found
The technical difference between these two designs is small. The operational difference is the difference between a failed deploy and an incident.
deploy 14:10 rollout green
canary 14:20 clean, no refunds in
the sample
promote 14:30 100% of traffic
03:12 first refund of the night
PAYMENTS_TIMEOUT_MS unset
-> library default: none
-> request hangs
-> pool exhausted
-> checkout down
03:40 on-call is reading a stack
trace with no mention of
configuration at alldeploy 14:10
instance 1 starts
FATAL: invalid configuration
PAYMENTS_TIMEOUT_MS missing
(required, int, 50..30000)
searched: flags, env, file
did you mean PAYMENTS_TIMEOUT_MS?
(found PAYMENT_TIMEOUT_MS)
exit 1, never becomes ready
14:11 rollout halts on failed
health, old version still
serving 100%
14:14 one-line fix, redeployNothing about the misconfiguration changed — only when it was detected. In the left column the failure arrives thirteen hours after the change that caused it, on a rare path, with no timeline correlation and a symptom (pool exhaustion) two hops from the cause. In the right column the existing rollout safety mechanism catches it for free, because refusing to become ready is something every deployment system already knows how to handle (A Successful Deploy Is Not Evidence of a Healthy System).
Three layers of validation, and where each catches something
Most implementations check presence and stop, which catches the least interesting third of the problem. The errors that cause real incidents are usually type coercion and semantics.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Key absent from the environment | Feature silently uses a development default | Presence not checked; a fallback was supplied at the call site | Required-ness in the schema; fail at startup naming the key |
| Key misspelled in the deployment manifest | The change appears to have no effect | Nothing reads the misspelled key and nothing complains about unknown keys | Reject unknown keys in the service's namespace, and suggest near-matches in the error |
| Numeric value with a letter in it | NaN propagates, or a timeout of zero | Every environment variable is a string and coercion was implicit | Explicit typed coercion that fails loudly rather than producing NaN |
ENABLE_X=false in a language with truthy strings | A feature is on when it was meant to be off | A non-empty string is truthy; false is a non-empty string | Boolean coercion with an explicit accepted set, rejecting anything else (Parse, Do Not Validate) |
| Pool size set far too high | The database refuses connections fleet-wide | Valid type, no range constraint, and the limit lives in another system (The Connection Budget) | Range constraints derived from the downstream limit, checked at startup |
| Debug or insecure flag left enabled | Verbose logging, an open debug endpoint, or TLS verification disabled | A value legal in development with nothing forbidding it in production | Environment-aware constraints that refuse the combination outright |
| Secret reference points at a path that does not exist | Crash loop, or a first-request failure hours later | Secret resolution deferred to first use rather than startup | Resolve secret references during startup validation (When Secrets Fail) |
Ordering: what must happen before the port is bound
The sequence matters as much as the checks. A service that validates after it starts accepting connections has done the work and thrown away the benefit, because the platform has already routed traffic to it.
- 1Resolve layers
Merge defaults, file, environment and flags into one effective set, recording each value's source.
fails by Merge semantics that silently drop nested keys, so a partial override wipes a whole object.
evidence The effective dump names the source layer per key.
- 2Coerce and type-check
Turn strings into numbers, booleans, durations, URLs — rejecting anything ambiguous.
fails by Implicit coercion producing NaN or truthy strings, which pass and then behave strangely.
evidence A deliberately malformed value fails the boot with a message naming it.
- 3Check semantics
Ranges, enums, required pairs, forbidden production combinations.
fails by Only presence being checked, so valid-but-absurd values pass.
evidence An out-of-range pool size is refused at startup rather than exhausting the database.
- 4Resolve secret references
Fetch every referenced secret under the workload identity, once.
fails by Deferring to first use, moving the failure to an unpredictable moment (Workload Identity).
evidence A revoked secret produces a boot failure, not a 3am payment error.
- 5Report or exit
Emit every failure together and exit non-zero, or log the redacted effective configuration and continue.
fails by Exiting on the first failure, forcing an operator through several restarts to find them all.
evidence A boot with three bad keys lists three problems.
- 6Bind and become ready
Open the port and let the readiness probe pass — only now.
fails by Binding first, so the platform routes traffic to a process that is about to die (Probes: Readiness, Liveness and Startup).
evidence The instance never appears healthy with an invalid configuration.
Step four is the one most often skipped, and it is the reason so many secret problems present as a request failure hours after a deploy rather than as a failed deploy.
How to do it properly
Most important first.
- Resolve and validate the entire configuration in one place, before binding a port, connecting a pool or registering with service discovery (Validate at Startup, Fail Loudly).
- Report every failure at once, not the first. An operator fixing keys one restart at a time in an incident is a self-inflicted wound.
- Include the key, the offending value (redacted if secret), the constraint and the source layer in the message.
- Validate semantics as well as presence: ranges, enums, mutually-required pairs, and combinations that must not occur in production.
- Resolve secret references at startup too, so a missing or unreadable secret fails on boot rather than on the first request that needs it (When Secrets Fail).
- Fail the process rather than degrading into a default. The exception is a genuinely optional feature, which should be explicitly optional in the schema rather than implicitly optional through a fallback.
- Keep the readiness probe honest: a process with invalid configuration must never report ready (Probes: Readiness, Liveness and Startup).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained well when validation is at startup: a rolling deploy stops, the previous version keeps serving, and the blast radius collapses to a failed rollout. Contained by nothing when validation is lazy — every instance is already serving traffic when the value is finally read (Version Coexistence: N and N+1, in Both Directions).
What can go wrong
- Validation that runs after the port is bound, so the platform routes traffic to a process that is about to exit.
- A crash loop with a message that does not name the key, which is the single most common way this control fails to help (When Secrets Fail).
- Fail-fast applied to a value that is genuinely optional, so an unrelated feature's missing key takes down the whole service.
- Validation of presence only, so a pool size of
-1or a timeout of0passes and produces behaviour nobody can explain. - A validation step that reaches out to a dependency — resolving a URL, opening a connection — turning a transient dependency outage into a fleet-wide failure to start (Validate at Startup, Fail Loudly).
- Secret values printed in the validation error message, putting credentials into logs at the exact moment everyone is reading them.
- "Fail-fast means the service is fragile." It means the failure is loud and early instead of quiet and late. The alternative is not a service that works — it is one that fails on a code path nobody is watching (Alert on Symptoms, Not on Causes).
- "A default is safer than a crash." A default is a decision made by whoever wrote the library, for their context. Silent defaults are how debug endpoints and disabled TLS verification reach production.
- "We validate config in CI, so runtime validation is redundant." CI validates the values CI can see. Production values are set by the platform, by an operator, or by a store, and often are not visible to CI at all.
- "Validation only matters for required values." Range and combination checks catch the errors that presence checks cannot: a valid-looking number that saturates a pool, or a debug flag left enabled (Requests and Limits).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Deliberately setting an invalid value in a lower environment produces a startup failure naming that key, and the instance never becomes ready.
- The last configuration mistake was caught at deploy time rather than at first use, and the log line was sufficient to fix it without investigation.
- A rolling deploy with a bad configuration value halts with the old version still serving (Rolling: Two Versions, One Database).
- No production incident in recent memory required someone to work out which configuration key was wrong.
- The rollback for a rejected configuration is trivial precisely because validation refused to start: the previous version is still running, untouched, and no traffic was affected.
- That is the argument for the whole lesson. Fail-fast converts a potential incident into a failed deploy, and a failed deploy is a category of event with an established, boring response (Rollback: Only Useful If It Is Actually Safe).
- The case with no clean rollback is a value that is valid but wrong — a pool size of 5 instead of 50 passes every check. Range constraints narrow this, and canarying config narrows it further (A Config Change Is a Production Change).
- Automate schema validation in CI against every environment's value set, so a missing production key fails a build rather than a deploy.
- Automate the check that the readiness probe reflects configuration validity, since the two are commonly wired independently and drift apart.
- Do not automate a retry around a configuration failure. Retrying an invalid value is a crash loop with extra steps, and it hides the message an operator needs to see.
- Fail-fast means one bad key stops the deployment entirely, which is correct and is also occasionally maddening when the key is for a feature nobody is using.
- Strict schemas add friction to adding configuration, which pushes people toward overloading existing keys.
- Rich error messages risk including sensitive values, so redaction has to be part of the same code path rather than an afterthought.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe startup-versus-first-use distinction is language- and platform-independent. What differs is how much help you get: statically typed languages catch shape errors at compile time for values that are literals, and nothing catches an environment variable, which is a string everywhere.
- PLATFORM-SPECIFICThe value of failing at startup depends on whether the platform notices. Orchestrators and managed services that watch health will halt a rollout on repeated startup failure; a process started by a supervisor that restarts unconditionally will crash-loop forever without stopping anything, and a serverless platform may report a cold-start failure only as an elevated error rate (Scale to Zero).
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — property-testing a configuration schema so that an illegal combination cannot be expressed rather than merely being rejected.