Deploys on the Same Timeline as the Symptom
The highest-signal overlay there is: a vertical line at each release, drawn across the error rate graph.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why is "what changed?" so hard to answer during an incident, when every change was recorded somewhere?
Change history lives in the delivery system and symptoms live in the monitoring system, so correlating them means an operator holding two timestamps in their head under pressure.
The deploy history is in the CI system and the metrics are in the monitoring tool. During an incident we can look at both.
Two tools means two clocks, two time zones and two time ranges, and reconciling them costs minutes at the exact moment minutes are expensive (Production Time Is UTC).
- Two tools means two clocks, two time zones and two time ranges, and reconciling them costs minutes at the exact moment minutes are expensive (Production Time Is UTC).
- It biases toward the deploy you remember. Someone recalls shipping something around 14:00, so that becomes the suspect, and the config change at 13:47 nobody mentioned is never considered.
- Deploys are not the only changes. Feature flags, config updates, infrastructure applies, certificate rotations, dependency releases and scheduled jobs all change production and are usually recorded in yet more places (Change Correlation).
- Delayed-onset failures become invisible. A change that only breaks under peak traffic surfaces hours later, and without the marker on the graph nobody connects them.
- After the incident the reconstruction is done from memory, which is how postmortem timelines acquire confident and incorrect causal claims (Reconstructing What Actually Happened).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Nearly all production incidents follow a change. Not all — capacity crossings, dependency failures, expiries and traffic shifts happen without you doing anything — but enough that "what changed" is the highest-prior-probability question available.
- Putting change events on the same time axis as the symptom converts that question from a search into a glance. The eye does the correlation; no query is written.
- What the overlay proves is temporal ordering, not causation. A vertical line just before the rise means the change is a strong suspect. A vertical line after the rise exonerates it, and that is the more valuable direction, because it eliminates suspects fast.
- The overlay also answers the recovery question. After a rollback, the second marker shows whether the symptom actually resolved at the moment traffic returned to the old version — which is the difference between a mitigation that worked and a coincidence (Rollback: Only Useful If It Is Actually Safe).
- The mechanism generalises to anything that changes production state: flags, config, migrations, scaling events, infrastructure applies. Every one of those is a candidate marker, and each should carry who, what and a link.
The same incident, with and without the overlay
This is a reconstruction of an ordinary incident. Read the change rows as the vertical lines on the graph: without them, the responder sees only the signal rows and has to ask people what happened around 14:00.
Notice that the overlay's most useful contribution here is negative. The 13:52 flag flip looks like the obvious suspect until the timeline shows the error rise began at 14:03, eleven minutes later, alongside a config change nobody had mentioned.
- 13:41changecheckout-api v482 rollout starts (annotated: actor, digest, diff link)
- 13:49changeRollout completes; both versions served traffic for 8 minutes
- 13:52changeFeature flag
new-tax-calcenabled for 10% of traffic - 14:03changePayment provider timeout config applied to the shared gateway
- 14:04signalcheckout 5xx rises from baseline to ~4%; latency p99 climbs
- 14:06signalSymptom alert fires: "Checkout failing for users"
- 14:07actionResponder opens dashboard; sees three markers in the window
- 14:09actionv482 and the flag ruled out: both precede the onset by more than 10 minutes with no change in error rate in between
- 14:11actionGateway timeout config reverted
- 14:13recoveryError rate returns to baseline; revert marker sits exactly at the recovery
- 14:20recoveryImpact confirmed ended via synthetic probe and support queue
The recovery marker is doing as much work as the causal one: without it, "we reverted and it got better" is a claim about memory. With it, the drop and the revert are on the same axis and the mitigation is evidenced (A Successful Deploy Is Not Evidence of a Healthy System).
Everything that changes production is a candidate marker
Teams annotate deploys and stop, which leaves the categories that cause incidents most quietly unannotated. A flag flip changes behaviour with no artifact, no pipeline run and often no review.
The last column is the reason each one belongs on the axis: what an operator concludes when they see the marker.
| Change type | Emitted by | Often missing because | What the marker tells the operator |
|---|---|---|---|
| Code deploy | Delivery pipeline | Rarely missing; this is the one everyone has | Which version, and what the diff was |
| Feature flag | Flag platform | It is not thought of as a deploy (Feature Flags: Deploy Is Not Release) | Behaviour changed with no new artifact |
| Config change | Config system or pipeline | Config is edited outside the delivery path (A Config Change Is a Production Change) | The artifact is unchanged and the inputs are not |
| Infrastructure apply | IaC pipeline | Applied from a laptop (The Plan: Desired vs Current) | Capacity, networking or permissions may have moved |
| Database migration | Migration runner | Runs on a different schedule than the deploy | Schema and locks are in scope (Why Migrations Are the Dangerous Change) |
| Scaling event | Autoscaler or platform | Considered routine rather than a change | Capacity moved on its own (Autoscaling) |
| Manual intervention | Break-glass session | By definition it bypassed the pipeline (Break-Glass Access) | Someone touched production directly |
| Dependency release | Upstream provider | You do not control the emitter | Something you depend on changed underneath you |
What the overlay does and does not prove
The overlay makes correlation instant, and instant correlation is easy to over-trust. The discipline is knowing which inference each arrangement of marker and symptom actually supports.
The strongest inference available from a timeline alone is exclusion. Causation needs a mechanism you can state.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Marker immediately before onset | Error rate rises within a minute of a deploy | Strong temporal association | Treat as leading suspect and roll back if impact is growing; confirm the mechanism afterwards (Stop the Harm Before You Understand It) |
| Marker after onset | Deploy landed while errors were already rising | The change cannot have caused what preceded it | Exonerate and move on — the most valuable reading the overlay gives you |
| Marker hours before onset | Symptom appears at peak traffic | Load-dependent defect shipped earlier and dormant | Widen the time range before excluding; onset is not always deploy time (Canary Analysis: Compared Against What?) |
| No marker anywhere near onset | Nothing you own changed | Dependency, data, traffic shift, expiry or a capacity crossing | Move to dependency and saturation signals rather than searching harder for a change (Headroom) |
| Several markers in one window | Three changes within ten minutes | Changes are batched or unrelated teams ship simultaneously | Revert the most recent first if impact is growing; separating them is postmortem work (Change Size: Why Small Changes Are Safer, and When They Are Not) |
| Revert marker with no recovery | Rolled back and the symptom persisted | Wrong suspect, or the change left persistent state behind | Stop assuming the rollback was the mitigation; check for state the revert did not undo (Destructive Migrations) |
How to do it properly
Most important first.
- Emit an annotation from the deployment pipeline itself, not from a person. It should carry the service, the version or digest, the actor and a link to the change (The Deployment Pipeline).
- Annotate every kind of production change, not just code deploys — flag flips, config changes, IaC applies, migrations, manual interventions (Manual Production Changes).
- Mark start and finish of a rollout separately. A rolling deploy is a window, not an instant, and the difference matters when both versions were serving (Rolling: Two Versions, One Database).
- Put the annotations on the symptom graphs at the top of the dashboard, not on a separate change panel — the point is the shared axis (Dashboards an Operator Can Act On).
- Record everything in UTC and render in the viewer's zone, so the timeline is unambiguous when responders are in different places (Timezone and DST Failures).
- Keep the same annotations available after the incident so the postmortem timeline is reconstructed from records rather than recollection (Postmortems).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The absence of annotations contains nothing; it lengthens every incident by the time it takes to answer a question that should have been free.
What can go wrong
- Annotations that only cover code deploys, so a config or flag change becomes an invisible suspect and the innocent deploy takes the blame.
- Missing annotations from anything not deployed through the standard pipeline — the emergency manual fix is the one least likely to be recorded and most likely to be relevant (Break-Glass Access).
- Clock skew between the deploying system and the metrics pipeline, placing the marker on the wrong side of the symptom (Clock Synchronisation).
- Annotation density so high that the graph is a solid wall of lines, which is common on a busy platform and defeats the purpose.
- Confident false correlation: the operator sees a marker before the rise, rolls back, and the symptom continues because the cause was elsewhere.
- Annotations that record only "deployed" without a version, leaving the operator unable to say what to roll back to.
- "The marker before the spike is the cause." It is the leading suspect. The exonerating direction — markers after the onset — is the more reliable inference.
- "We deploy from CI so the history is already recorded." Recorded somewhere is not on the same axis, and the axis is the entire mechanism.
- "Only deploys need annotating." Flag flips and config changes cause incidents at a similar rate and are far less visible (Feature Flags: Deploy Is Not Release).
- "Rolling back fixed it, so the deploy caused it." Sometimes the rollback also restarted every process, which is a second intervention hiding inside the first.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- During the last incident, the question "what changed" was answered from a graph rather than from a person's memory.
- A change that turned out to be innocent was ruled out by its marker falling after the symptom began.
- The postmortem timeline was assembled from recorded events, and the recorded times matched what people remembered — or corrected them.
- The annotation feed itself is low-risk: if it becomes too noisy, filter by change type or service rather than disabling it, because losing the overlay costs you the fastest question in an incident.
- If markers are wrong — skewed clocks, mislabelled services — treat that as a defect to fix rather than a reason to stop trusting the overlay, because an operator who half-trusts it will go back to reconstructing from memory.
- Fully automate emission from the pipeline. An annotation a human has to remember to add is missing exactly when the change was unusual, which is when it matters.
- Automate the reverse link too: from a marker, one click to the diff, the pipeline run and the artifact digest (Tags Versus Digests).
- Automate collection of the same events into the incident channel, so the timeline is being built while the incident is happening (Telling People What Is Happening).
- Do not automate the inference. "Symptom followed change, therefore roll back" is often right and sometimes expensively wrong; a human decides (Stop the Harm Before You Understand It).
- On a platform with frequent deploys the overlay gets crowded, and filtering it well is ongoing work.
- The overlay makes correlation so easy that it encourages stopping at correlation. That is an acceptable cost only if the team knows the difference.
- Annotating every change type means integrating several systems into one feed, which is a small but real piece of platform engineering.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALEvery delivery system can emit a change event and every monitoring system can render one. The mechanism does not depend on either being a particular product.
- TOOL-SPECIFICSome monitoring platforms have first-class annotations with links and filters; others only support a marker series you construct yourself. Where the platform has neither, the workable substitute is a change feed pinned next to the dashboard on the same time range — worse, because the eye has to move, but it still answers the question.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — why change correlation is evidence for a hypothesis rather than a substitute for one.