Environment Drift
Environments diverge from each other continuously and silently, and a drifted environment does not stop answering questions — it starts answering them wrongly.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why does a change that passed staging fail in production when nobody changed either environment?
Environments are changed by many people, many pipelines and several automated processes, and almost none of those changes are recorded anywhere that a person reading a test result would see.
They were built the same way from the same templates, so they are the same. If something differs, we would know.
Emergency fixes are applied to production first and to lower environments later, or never. Every incident is an opportunity for divergence.
- Emergency fixes are applied to production first and to lower environments later, or never. Every incident is an opportunity for divergence.
- Managed services upgrade on their own schedule, and lower environments often sit on a different maintenance window or a different tier.
- Instance sizes, replica counts, connection limits and timeouts get tuned in production under real load and are never backported.
- Feature flag state is per-environment, so the code path exercised in staging may not be the code path running in production (Feature Flags: Deploy Is Not Release).
- Data drifts fastest of all: production accumulates edge cases for years, staging holds whatever was seeded eighteen months ago.
- Third-party sandboxes behave differently from their production counterparts by design — different rate limits, different validation strictness, different latency.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Drift is the accumulated difference between the declared state of an environment and its actual state, plus the difference between two environments that are supposed to correspond.
- It has two distinct sources and they need different responses. Unmanaged change — someone edited a console, a script ran once, an emergency fix landed — is fixable by making the declared state authoritative (Drift).
- Structural difference — production has nine replicas across three zones and a 4 TB database, staging has one replica and 40 MB — is not drift to be fixed. It is a deliberate difference that must be known and reasoned about (Parity That Is Worth Paying For).
- The damage comes from the gap being invisible at the moment someone reads a test result. Drift does not make an environment stop working; it makes it produce confident answers about a system that no longer resembles the one you are shipping to.
Six kinds of drift, in the order they bite
Drift is not one phenomenon. Separating it by kind matters because each kind has a different detector and a different fix, and only some of them are covered by the tool most teams assume covers all of them.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Emergency fix applied directly to production | Staging reproduces a bug production no longer has, or vice versa | The change exists in one environment and in nobody's repository | Break-glass access that records the change and opens a follow-up to land it as code (Manual Production Changes) |
| Managed service auto-upgraded in one environment | A query plan, a TLS handshake or a client library behaves differently in production only | Different minor versions and different maintenance windows per environment | Pin versions explicitly per environment and upgrade lower environments first |
| Capacity tuned under real load | Load behaviour cannot be reproduced anywhere; staging saturates at a different point | Instance class, replica count, connection and thread limits diverged (The Connection Budget) | Report the delta as part of the promotion evidence; treat structural gaps as known, not as bugs |
| Feature flag state diverged | The tested code path is not the running code path | Flags are per-environment state and are rarely part of any diff | Include flag state in the environment diff and in the release record (The Release Manifest) |
| Config key added in production only | Service starts in staging, crash-loops in production, or silently uses a default | Config is edited per environment, so it is the least uniformly reviewed input (Configuration Drift) | Validate the full config schema at startup in every environment (Validate at Startup, Fail Clearly) |
| Data aged out | A migration or backfill that is instant in staging runs for hours in production | Volume and shape diverge continuously and fastest of all | Rehearse data-shape-sensitive changes against production-scale volume (Expand, Migrate, Contract) |
Detecting drift instead of discovering it
The mechanism that catches drift is reconciliation: something compares the declared state to the actual state on a schedule and reports the difference. That is the same loop Kubernetes runs continuously and that an IaC plan runs on demand (Reconciliation: The Loop Under Everything).
The part teams get wrong is the report, not the detection. A drift report is only a control if a human reads it at the moment they are deciding to promote a change.
1staging -> production diff (generated 14:02, before promote)2 3 runtime4 app image same digest sha256:9f3e... OK5 node count 3 -> 27 expected6 cpu request per pod 250m -> 1000m expected7 8 managed services9 postgres engine 16.3 -> 16.1 ATTENTION10 postgres max_connections 100 -> 800 expected11 redis eviction policy allkeys-lru -> noeviction ATTENTION12 13 configuration14 keys only in production PAYMENTS_TIMEOUT_MS, REGION_TIER ATTENTION15 keys only in staging DEBUG_TOOLBAR expected16 17 feature flags18 new-checkout-flow on -> off ATTENTIONThe three ATTENTION lines are the whole point: an engine version behind in production, a Redis policy that will refuse writes rather than evict, and two config keys that have never been exercised anywhere but production. "Expected" rows are structural difference, not drift — see Parity That Is Worth Paying For.
The correction, and how it goes wrong
Reconciling drift is a production change with its own blast radius, and it is one of the few changes where "restore the intended state" can itself be destructive — because the intended state may be older than the reason production diverged.
- 1Detect
Compare declared state against actual, per environment, on a schedule.
fails by Reporting into a channel nobody reads, or reporting so much noise that real differences are invisible.
evidence A dated diff that a human opened before the last promotion.
- 2Classify
Split each row into unmanaged change, structural difference, or a deliberate production-only tuning.
fails by Treating structural difference as drift and starting an expensive parity project.
evidence Every row in the diff has a classification, and the "expected" set is short and named.
- 3Decide direction
Decide whether the code moves to match production or production moves to match the code.
fails by Reflexively reverting production to the declared state, deleting the tuning that was keeping it up.
evidence A plan output that was read line by line before apply (The Plan: Desired vs Current).
- 4Apply
Land the change through the normal path, lowest environment first.
fails by Applying in production first because that is where the diff was noticed.
evidence The same change visible in the audit trail for each environment (The Audit Trail).
- 5Verify
Re-run detection and confirm the row is gone rather than assuming.
fails by Assuming apply means converged, when the resource was recreated with a new default.
evidence A clean diff on the next scheduled run.
Step three is the one that causes incidents. The declared state is a claim about what should be true, and during an incident someone may have made production correct in a way the repository does not know about yet.
How to do it properly
Most important first.
- Make the declared state authoritative and reconcile against it, so unmanaged change is detected rather than discovered (Infrastructure as Code, The Plan: Desired vs Current).
- Generate the difference report rather than remembering it: versions, instance classes, replica counts, flag state, dependency endpoints, config keys present in one environment and absent in another.
- Attach the report to the promotion decision. A diff nobody reads at the moment of the decision is documentation, not a control.
- Close the emergency-fix loop explicitly: any change made directly in production gets a follow-up task to land it as code and apply it everywhere (Manual Production Changes).
- Refresh lower-environment data on a schedule, using anonymised or synthetic data rather than a raw copy (Production Data in Lower Environments).
- Rebuild lower environments from their definition periodically. A rebuild that fails is drift you did not know about, found cheaply.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Partly contained by a canary, which surfaces the difference at 1% of traffic instead of 100% — but only for differences that show up as a signal quickly (Canary Analysis: Compared Against What?).
What can go wrong
- Drift detection that runs and reports into a channel nobody reads, which is worse than none because it looks like coverage.
- A drift report so noisy — every autoscaled replica count, every rotated credential version — that real differences are lost in it.
- Reconciliation that automatically reverts a change someone made deliberately during an incident, mid-incident.
- Fixing drift by making production match staging, which is backwards: production is the system with users.
- Treating structural difference as drift and chasing parity on scale, which is expensive and never finishes (Parity That Is Worth Paying For).
- "We use infrastructure as code, so we do not have drift." IaC covers what it manages. Flag state, database parameters changed by a DBA, manually installed agents, third-party console settings and data are all outside most state files (State).
- "Drift means someone did something wrong." Most drift is the residue of correct urgent decisions. The problem is that the record of them is missing, not that they were made.
- "The diff is small, so the risk is small." A single differing timeout or connection limit is a small diff and a complete explanation for an outage.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- You can produce the current difference between staging and production in one command, and it fits on a screen.
- The last three production incidents caused by an environment difference are each traceable to a specific unrecorded change, and each produced a control rather than a reminder.
- A from-scratch rebuild of a lower environment succeeds without manual steps.
- Reverting a reconciliation is easy when the declared state is versioned — revert the commit and re-apply.
- Reverting a *data* refresh is not: once anonymised production data has been loaded into a lower environment, the exposure has already happened even if you drop the table afterwards.
- A drifted production that has been reconciled toward a stale definition can have deleted the exact tuning that was keeping it up. Read the plan before applying it (Destructive Changes: What a Rename Really Does).
- Automate detection and reporting on a schedule, and automate the periodic from-scratch rebuild of lower environments.
- Automate reconciliation in lower environments freely; in production, generate the plan automatically and keep a human on the apply (The Plan: Desired vs Current).
- Do not automate away the follow-up on emergency fixes. The decision about whether a hot fix belongs everywhere is judgement.
- Strict reconciliation removes the escape hatch operators use under pressure. That is the point, and it needs a sanctioned break-glass path or people will route around it (Break-Glass Access).
- Frequent rebuilds cost time and break long-lived test fixtures that people quietly depend on.
- Keeping lower environments genuinely current is ongoing work that produces no visible feature.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALAny two environments maintained over time diverge. What differs is the rate: fully declarative platforms that reconcile continuously drift within a narrow band, while environments with console access and no reconciliation drift without bound.
- CLOUD-SPECIFICManaged-service versions drift on the provider's schedule, not yours, and the providers differ: some auto-upgrade minor versions in a maintenance window unless pinned, others require an explicit upgrade action and will eventually force one at end of support. Check the specific service's policy rather than assuming your other provider's behaviour carries over.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — why a test that passes in a drifted environment is not evidence, and how to express environment assumptions as assertions.