Change Correlation
Four categories of change — deploy, config, infrastructure, dependency — plus traffic, and the discipline of checking all of them rather than only code.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
When you ask "what changed?", what is the complete list of things that could have?
Teams do ask what changed. They then check the deploy history, find nothing, and conclude that nothing changed — while a config value, an infrastructure apply or an upstream dependency moved an hour earlier.
Change history means deploy history. The pipeline records every release, so if the pipeline is quiet, the system is unchanged.
Configuration changes without a deploy. Someone edits a value in a console or applies a config repository change, and it takes effect at the next restart or the next refresh — with no release to point at (A Config Change Is a Production Change).
- Configuration changes without a deploy. Someone edits a value in a console or applies a config repository change, and it takes effect at the next restart or the next refresh — with no release to point at (A Config Change Is a Production Change).
- Infrastructure changes without your involvement. An IaC apply by another team, a managed service upgrade in a maintenance window, an autoscaling event, a node pool replacement.
- Dependencies change upstream. An unpinned transitive dependency resolves differently on a rebuild, so a release you believe is identical is not (Dependency Pinning).
- Traffic changes with nothing in any feed at all: a marketing send, a client shipping a retry loop, a crawler, month-end batch work.
- Each of these produces the same conversation — "nothing changed, so it must be intermittent" — which is how a fifteen-minute incident becomes a two-hour one.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- What runs in production is a function of more inputs than the artifact: artifact + configuration + infrastructure + dependency versions + the traffic it receives. Change any one and the behaviour changes.
- Each input has a different owner, a different change mechanism and a different record, which is exactly why the union is rarely visible in one place. It is an organisational gap that shows up as a diagnostic one.
- Discipline here is completeness, not cleverness: check all categories every time, including the ones your team does not control. The category you skip is the category the incident is in, at a rate far above chance, because it is the one nobody instrumented.
- Correlation output is a ranked suspect list with times, feeding the decision about what to reverse. It is not a cause (Deployment-Centric Debugging).
- Adjacency strength depends on base rate. In a system with one change a day, a change eight minutes before the symptom is compelling. In a system with forty changes a day, it is background noise and you need per-change signals to separate them (Canary Analysis: Compared Against What?).
The five categories
Four change categories plus traffic. Traffic is not something you change, but it changes what the system does, and leaving it off the list is how "nothing changed" gets said during a load-driven incident.
The column that matters most in practice is the last one: where the record lives. Every empty or unreliable cell there is a category that will be missed during an incident.
| Category | Examples | How it reaches production | Where the record usually lives |
|---|---|---|---|
| Code | A release, a hotfix, a revert | Build, artifact, rollout (The Deployment Pipeline) | CI/CD history — nearly always present |
| Config | Timeout, TTL, feature flag, connection limit, environment variable | Config apply, restart, or a live refresh | Config repository if versioned; nowhere if edited in a console |
| Infrastructure | IaC apply, instance type change, node pool replacement, network rule, managed service upgrade | A plan and apply, an autoscaling event, or a provider maintenance window | IaC run history and cloud audit log — often another team's |
| Dependency | Library bump, base image update, upstream API version, certificate rotation | Picked up at the next build, or changed entirely outside your systems | Lockfile and image digest if pinned; unrecoverable if not |
| Traffic | Campaign send, client retry loop, crawler, month-end batch, a large customer onboarding | It just arrives | Request metrics by client and endpoint — rarely treated as a change record |
Checking all five, every time
The discipline is a checklist, and it is a checklist precisely because the category you would skip under pressure is the one you skip every time. Each step below should take well under a minute if the records exist.
- 1Fix the window
Establish symptom onset from the highest-resolution signal available, then widen by roughly thirty minutes on each side.
fails by A coarse metric puts onset in the wrong minute, so the true trigger falls outside the window.
evidence Onset stated as a timestamp with the metric and resolution it came from.
- 2Code
List releases and reverts touching the affected service and everything it calls.
fails by Only the failing service is checked, missing a change in a dependency you own.
evidence Rollout start and completion times, with artifact digests.
- 3Config
List config applies, flag flips and environment changes, including those applied outside the pipeline.
fails by Console edits leave no record and are invisible to the sweep (Configuration Drift).
evidence A versioned config history with actor and time, or an audit log from the console.
- 4Infrastructure
List IaC applies, scaling events, node replacements and provider maintenance in the window.
fails by Another team's applies are not visible to the responder (Infrastructure as Code).
evidence IaC run history plus the cloud provider audit log, filtered to affected resources.
- 5Dependency
Compare dependency versions and base image digests against the last known-good build; check external dependency health from your side.
fails by Unpinned dependencies make "compare against last known good" meaningless.
evidence A lockfile diff or image digest diff, plus per-dependency error and latency panels.
- 6Traffic
Compare request rate, client mix, endpoint mix and payload sizes against the same hour on previous days.
fails by Only total request rate is graphed, hiding a shift in mix at constant volume.
evidence Traffic broken down by client and endpoint, with a day-over-day comparison.
- 7Rank and act
Order the suspects by proximity, plausibility and reversal cost; reverse the cheapest plausible one.
fails by Several things are reversed at once, so the recovery cannot be attributed.
evidence One reversal at a time, each with the metric that was expected to move.
Empty categories are results. Record them — a sweep that found no infrastructure changes is materially different from a sweep that never looked.
When correlation misleads
Change correlation is a strong prior, and strong priors fail in characteristic ways. Each of these has produced a real incident where the team confidently reversed the wrong thing and then had to work out why nothing improved.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A deploy lands minutes before an unrelated provider degradation | Rollback completes and the error rate does not improve | Coincidence at high deploy frequency — something is always nearby | Treat "reversal did not move the metric" as exoneration and move to the next suspect immediately |
| A release is deployed with its new code path behind a flag | Impact begins days after the deploy, with no change in the window | The flag flip is the real change and it is not in the deploy feed | Put flag flips in the change feed as first-class changes (Feature Flags: Deploy Is Not Release) |
| A change is reverted and the symptom stops | Confident attribution; the incident recurs a week later with no deploy | The change exposed a latent fault — different traffic shape, or a pool limit that was always too low | Ask what made it possible, not only what triggered it (Root Cause vs Contributing Factors) |
| Metrics and the deploy system disagree on the clock | The timeline shows the symptom starting before the deploy that caused it | Clock skew inverts the ordering (Clock Synchronisation) | Verify both sources against a known event before trusting the order |
| Config applies lazily as instances restart | Impact appears over twenty minutes although the change is recorded as instantaneous | Record time is apply time, not effective time | Emit an effective-time event per instance, not only the apply event |
| Nothing found in any category | Search stalls; the incident is labelled intermittent | Growth against a limit, a scheduled event, or a genuinely uninstrumented change | Switch to the no-change branches: utilisation curves, schedules, certificate expiry (Headroom) |
How to do it properly
Most important first.
- Enumerate the categories explicitly during an incident rather than asking the open question. "Any deploys? any config applies? any infra applies? any dependency bumps? any traffic shift?" gets five answers; "did anything change?" gets one and it is usually "no".
- Give every category a queryable record with timestamps and an actor, and merge them into one feed (The Audit Trail).
- Instrument the changes you do not make: watch dependency error rates and latency from your side, and record managed service maintenance events where the provider exposes them.
- Treat traffic as a change category with its own signals — request rate by client and endpoint, payload size distribution, and the mix of operations.
- Rank suspects by proximity in time, plausibility of touching the failing path, and reversal cost, then act on the cheapest reversal among the plausible ones.
- Record negative results. "Reverted the TTL change, no effect" is evidence, and without it the next responder will revert it again.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
An incomplete change record does not cause incidents; it extends them, and it does so most on the incidents that are already hardest to diagnose.
What can go wrong
- Console edits and manual restarts that leave no record, making a real change look like no change (Manual Production Changes).
- IaC drift: the applied state differs from the committed state, so reading the repository misrepresents what is running (Drift).
- A dependency change with no local record because the version was never pinned — the same commit builds differently on different days.
- Config that applies lazily, per instance, over minutes, so the change time in the record is not the change time in production.
- Over-collection: a feed containing every CI run, every non-production apply and every scaling tick, in which the one relevant line is unfindable.
- Correlating against the wrong window because the symptom onset was misread from a coarse metric.
- "We have deploy history, so we have change history." Deploys are one of five categories, and typically the best-instrumented one.
- "Nothing changed" as a conclusion rather than a finding about your records. It should read: "no change is recorded in the categories we record".
- "Correlated therefore causal." It is a ranking. The confirmation is that reversing it changes the signal (A Successful Deploy Is Not Evidence of a Healthy System).
- "Freeze changes and incidents stop." Freezes push change into larger, riskier batches and do nothing about infrastructure, dependency and traffic changes, which continue regardless (Change Management).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- One query returns every change across all five categories for an arbitrary time window, with actor and link.
- During the last incident, all categories were checked and the check is visible on the timeline — including the ones that came back empty.
- When someone says "nothing changed", they can name the categories that were checked.
- Correlation is diagnostic and reverses nothing by itself. Its practical value is proportional to how reversible the identified suspects are.
- Where the top suspect is irreversible — a completed destructive migration, a consumed external side effect — correlation still tells you where to look, but the response has to be forward (Roll Forward: When Going Back Is the Harder Option).
- Automate ingestion of every category into one feed, emitted by the acting system rather than written by a person afterwards.
- Automate the incident-window query so a responder gets a link, not a research task.
- Do not automate the ranking into an action beyond well-bounded cases. Automatic rollback driven by canary analysis is sound; automatic rollback driven by "a change happened near an alert" will reverse innocent releases during unrelated dependency outages.
- A complete feed means integrating systems owned by different teams, and the value only shows up during incidents, which makes it perennially hard to prioritise.
- Completeness fights signal-to-noise. Every additional category makes the feed more complete and harder to read, so filtering by subject and environment is mandatory.
- Instrumenting dependency and traffic changes costs ongoing metric volume for something you hope never to need.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe five categories describe the inputs to any running system. Which of them is best instrumented varies: a single-vendor PaaS may record config and infrastructure changes natively, while a self-managed estate records only deploys.
- ORG-SPECIFICWho is allowed to change what — and therefore which categories are invisible to the responding team — is an organisational fact. In a team that owns its own infrastructure, infra changes are in reach; where a platform team owns it, they usually are not without an explicit shared feed.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.