A Config Change Is a Production Change
Configuration changes reach production faster than code, apply to everything at once, are reviewed less, and frequently have no rollback story — which is why so many outages are config-only.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why do configuration changes cause so many outages when they involve no new code?
Every discipline built around code changes — review, testing, staged rollout, versioning, rollback — is attached to the artifact, and configuration is the other half of what is running.
Configuration is just settings. Changing a timeout or a feature value is low risk, so it can be done directly in the console or the store, immediately, without a pipeline.
Speed is the danger, not the benefit. A configuration change typically reaches every instance in seconds, with no canary, no progressive rollout and no bake time (Progressive Delivery: Exposure as a Dial).
- Speed is the danger, not the benefit. A configuration change typically reaches every instance in seconds, with no canary, no progressive rollout and no bake time (Progressive Delivery: Exposure as a Dial).
- It is reviewed less. A one-line code change gets a pull request; a one-line configuration change often gets a text field and a save button.
- It is frequently unversioned. A console edit overwrites the previous value with no record of what it was, so "put it back" requires someone to remember.
- Configuration changes look identical whether they are trivial or catastrophic. Nothing in the interface distinguishes changing a log level from changing a connection limit.
- They are invisible on the timeline. Incident response starts with "what changed", and if configuration changes are not on the deploy timeline the answer comes back "nothing" (Deploys on the Same Timeline as the Symptom).
- Config often crosses service boundaries. One shared value — a rate limit, a routing rule, a feature default — changes behaviour in several services at once.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A running service is artifact plus configuration. Change either half and you have changed production. The artifact half has an entire industry of safety practice attached; the configuration half usually has a text box.
- The asymmetry is structural, not cultural. Configuration exists precisely so it can be changed without a build, and everything that makes it convenient — immediate, fleet-wide, no pipeline — also removes the mechanisms that bound a code change's blast radius.
- Rollback is often genuinely absent rather than merely unpractised: if the previous value was not recorded, there is nothing to revert to. Where the change triggered an irreversible effect — a cache flushed, a queue drained, a migration flag flipped — reverting the value does not undo the effect (Roll Forward: When Going Back Is the Harder Option).
- Configuration changes also interact with rollback of the *other* half. Reverting an artifact while leaving new configuration in place produces a combination that has run nowhere, which is a common way a rollback deepens an incident (Rollback: Only Useful If It Is Actually Safe).
A config-only outage, minute by minute
The reason this shape is so common is visible in the timeline: the change was small, correct-looking, applied instantly everywhere, and absent from the place everyone looked first.
- 09:14changeAn engineer lowers
DB_POOL_MAXfrom 50 to 5 in the configuration store, intending to reduce database load. One field, saved immediately. - 09:14changeThe store pushes the value to all 27 instances within a second. No rollout, no canary, no bake time.
- 09:15signalRequest latency rises sharply as requests queue for connections. Error rate is still near zero — requests are slow, not failing.
- 09:18signalLatency alert fires. On-call opens the deploy timeline and sees no deploys since yesterday afternoon.
- 09:21signalCheckout timeouts begin as upstreams give up. Error rate crosses the paging threshold.
- 09:24actionOn-call declares an incident. The working hypothesis is a database problem, because the database is where the queueing is visible (The Connection Budget).
- 09:31signalDatabase metrics show low utilisation and few connections, which contradicts the hypothesis and costs several minutes.
- 09:38actionSomeone asks in the channel whether anyone changed anything. The engineer says yes.
- 09:40actionThe previous value is not recorded anywhere. The engineer remembers it was 50; nobody can confirm it.
- 09:42recoveryValue set back to 50. Pools refill over the next two minutes as instances pick up the change.
- 09:47recoveryLatency normal. Total user impact: 32 minutes, of which roughly 24 were spent finding a change that took one second to make.
Three controls would each have cut this substantially: the change on the deploy timeline (visible at 09:18 rather than found at 09:38), a recorded previous value (no guessing at 09:40), and a schema range constraint tied to the pool floor (rejected at 09:14). None of them requires slowing configuration changes down in general (Validate at Startup, Fail Clearly).
Grade the change by blast radius, not by size
Uniform policy fails in both directions: heavy review on a log level pushes people to bypass the pipeline, and no review on a rate limit leaves an outage one keystroke away. Grading by what the value controls is what makes the policy survivable.
| Value class | Propagates | Worst case | Treatment |
|---|---|---|---|
| Log level, sampling rate | Instantly, fleet-wide | Log volume and cost spike (Cost Drivers) | Self-service; keep it fast, it is needed during incidents |
| Feature flag, small audience | Instantly, scoped | A subset of users see a broken feature | Self-service with a kill switch and an owner (Feature Flags: Deploy Is Not Release) |
| Feature flag, default for everyone | Instantly, fleet-wide | Every user gets untested behaviour at once | Progressive rollout, defined abort signal, treated as a release |
| Timeout, retry policy | On restart or instantly | Retry storms, cascading saturation upstream | Reviewed, rolled out progressively, watched |
| Connection or pool limit | On restart or instantly | Saturation or fleet-wide exhaustion of a shared database | Reviewed, range-constrained, rolled out progressively |
| Rate limit, quota | Instantly, fleet-wide | Legitimate traffic rejected, or a downstream overwhelmed (Load Shedding) | Reviewed, progressive, with the downstream owner informed |
| Routing, endpoint, region | Instantly, fleet-wide | Traffic to the wrong place, or to nowhere (Service Discovery in Operation) | Treated exactly as a deployment, with a rollback plan |
| Credential reference or rotation | On restart or on refresh | Authentication failures fleet-wide (Rotation That Applications Survive) | Overlap window first, then progressive, never a hard swap |
Giving configuration the machinery code already has
The goal is not to make configuration slow. It is to give it the four properties a code change already has — a diff, a previous version, a staged rollout and a record — while keeping an audited fast path for incidents.
- 1Propose as a diff
Express the change as a versioned change to declared values, showing old and new.
fails by A console text field, which shows the new value and destroys the old one.
evidence A reviewable diff exists and names both values.
- 2Validate against the schema
Type, range and combination checks before anything is applied (Validate at Startup, Fail Clearly).
fails by Accepting any string, so a pool size of 5 and of 5000 look equally fine.
evidence An out-of-range value is rejected at propose time.
- 3Grade and route
Classify by what the value controls and require review proportional to blast radius.
fails by Uniform policy, so people bypass it for the trivial cases and habituate to bypassing.
evidence The class is visible in the change record.
- 4Apply progressively
One instance, then one zone, then the fleet, with a bake interval.
fails by Instant fleet-wide push, which is the default behaviour of most configuration stores.
evidence Instances report which configuration version they hold, and the counts move in stages.
- 5Verify
Watch the same signals as a canary — latency, errors, saturation, downstream load (Canary Analysis: Compared Against What?).
fails by Assuming a successful write means a successful change.
evidence A comparison against the pre-change baseline, not just an absence of alerts.
- 6Record
Annotate the deploy timeline with who, what, old value, new value, and configuration version.
fails by No annotation, so the next incident starts with "nothing changed" (Deploys on the Same Timeline as the Symptom).
evidence The change is visible on the timeline an incident responder opens first.
The emergency path skips grading and progressive application, and keeps validation and recording. An unaudited emergency path is how the audit trail stops describing production.
How to do it properly
Most important first.
- Put configuration in version control and change it through the same review and pipeline path as code. The point is not ceremony; it is a diff, an approver and a previous value (Change Management).
- Roll configuration out progressively where the mechanism allows: a subset of instances, one zone, or a percentage — and watch the same signals you would watch for a code canary (Canary Analysis: Compared Against What?).
- Record every configuration change on the same timeline as deploys, with who, what, when and the previous value (The Audit Trail).
- Classify changes by blast radius rather than by size. A log level is low risk; a connection limit, a timeout, a rate limit, a routing rule or a feature default is a high-risk change that happens to be one line.
- Version configuration as a unit and reference that version in the release record, so a rollback can restore the artifact/configuration pair that was healthy (The Release Manifest).
- Require the previous value to be captured before the new one is applied, mechanically, so "put it back" is always possible.
- Give high-risk values a bounded range in the schema so the worst case is limited even when the change is wrong (Validate at Startup, Fail Clearly).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Usually nothing. Configuration is typically applied to all instances at once with no canary and no bake time, and where the previous value was not recorded there is not even a rollback. This is the honest answer for most teams, and it is why config-only outages are so common (Reducing Blast Radius).
What can go wrong
- A change applied fleet-wide in seconds, breaking everything before anyone can read a dashboard.
- A revert that restores the value but not the state: connections already dropped, caches already cold, a queue already drained (Operating a Cache).
- A configuration store outage that leaves instances with mismatched values — some old, some new — producing behaviour that matches no tested combination.
- Two people editing the same value at once with last-write-wins and no record of either.
- A shared value changed for one service that silently changes behaviour in three others.
- Configuration gated so heavily that operators route around it during incidents, which puts the emergency path outside the audit trail (Break-Glass Access).
- "It is only config." Configuration decides connection limits, timeouts, rate limits, routing and feature state. "Only config" describes the effort, not the blast radius.
- "Config changes are safer because there is no new code." There is no new code and no new testing either, and the change reaches every instance immediately rather than through a rollout (Change Size: Why Small Changes Are Safer, and When They Are Not).
- "We can always change it back." Only if someone recorded the old value, and only if the effect was reversible. Neither is automatic.
- "Feature flags mean config changes are safe." Flags make *some* changes reversible and instant, which is genuinely valuable — and an instant fleet-wide flip is still an instant fleet-wide change (Feature Flags: Deploy Is Not Release).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Configuration changes appear on the deploy timeline alongside code deploys, and an incident responder can see them without asking anyone.
- For any current value you can produce its previous value and who changed it, without archaeology.
- A recent high-risk configuration change was rolled out progressively and had a defined abort signal.
- The release record names the configuration version alongside the artifact digest (Promotion Between Environments).
- Where configuration is versioned, rollback is restoring the previous version — the same mechanism as an artifact rollback, and it should use the same path.
- Where it is not versioned, there is frequently no rollback at all, because nobody recorded the previous value. Say this plainly: many teams discover it mid-incident.
- Some configuration changes are irreversible in effect even when the value reverts — a flag that triggered a data migration, a rate limit that caused upstreams to drop work, a cache invalidation. Those need roll-forward plans, not rollback plans (Roll Forward: When Going Back Is the Harder Option).
- Reverting configuration and artifact independently produces untested combinations. Roll back the pair, or be explicit about why you are not.
- Automate the pipeline: diff, validate against the schema, apply progressively, verify, record. Configuration deserves the same machinery as code (The Deployment Pipeline).
- Automate the timeline annotation, since a manual one is exactly the step that is skipped during the incident where it matters.
- Automate an abort: if the signal degrades after a configuration rollout, restore the previous version without waiting for a human.
- Do not automate approval of high-blast-radius values. A human should look at a connection limit or a rate limit change, and know they are looking at a production change (Guardrails, Not Gates).
- Pipelining configuration removes its main advantage — speed — which matters genuinely during incidents, so an audited fast path has to exist or people will bypass the system.
- Progressive configuration rollout requires instances to hold different values simultaneously, which means the system must tolerate mixed state, exactly like a rolling deploy (Version Coexistence: N and N+1, in Both Directions).
- Version control for configuration is awkward when values are set by another team or by a managed platform's own console.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe asymmetry between artifact discipline and configuration discipline exists on every platform. What varies is the propagation speed: values baked into a deployment propagate at rollout speed, while a dynamic store can reach every instance in under a second, which removes the accidental bake time a rollout provides.
- ORG-SPECIFICWhether configuration changes require review is a policy choice, and it should be graded rather than uniform. A blanket approval requirement on every value pushes operators to bypass the system under pressure; no requirement at all leaves connection limits one keystroke from an outage. Where the line sits depends on your risk tolerance and your incident history.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — testing configuration combinations, and why a values file is as testable as the code that reads it.