ReleaseGENERALORG-SPECIFICCLOUD-SPECIFIC
The Audit Trail
Who changed what, when, why — and what the previous state was. The last field is the one that turns a log into something you can act on.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The production question
During an incident, how do you find out what changed and what it was before?
The problem
Production changes arrive through many paths — deploys, config, flags, infrastructure, database, access grants, third-party consoles — and most of them leave a record somewhere unhelpful, or nowhere at all.
What teams do first
Everything goes through the pipeline, so the pipeline history is the audit trail. Anything else is visible in git.
How it breaks
Flag changes do not go through the pipeline, and they change production behaviour immediately. This is why so many incidents begin with "nothing was deployed" (Deployment Is Not Release).
How it breaks in production
- Flag changes do not go through the pipeline, and they change production behaviour immediately. This is why so many incidents begin with "nothing was deployed" (Deployment Is Not Release).
- Console changes — a scaling limit raised, a security-group rule added, a managed-database parameter adjusted — are made in a provider UI and recorded, if at all, in a provider log nobody has opened (Manual Production Changes).
- Emergency changes bypass the normal path by design, so the changes made under the most pressure are the ones with the least record (Break-Glass Access).
- Most records say what changed and not what it was before, which is precisely the field you need in order to put it back.
- Records scattered across five systems with different clocks, retention policies and identity models cannot be read as one timeline, which is the only way an incident actually needs to read them (Reconstructing What Actually Happened).
- Automation acts on behalf of people, so an entry attributed to a service account tells you nothing unless it also carries who or what triggered it.
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- An audit entry answers five things: who made the change, what changed, when, why, and what the previous state was. The fifth is the one most often missing and the one that makes the entry actionable rather than merely informative.
- "Who" must resolve to a human even when the actor is automation. An entry attributed to
deploy-botis only useful if it also carries the run, the commit and the person who triggered it (Human vs Machine Identities). - "Why" is usually a reference rather than prose — a ticket, an incident id, a pull request. Free-text reasons decay into "fix" and stop carrying information.
- The trail must be append-only and stored outside the system it describes. A record you can edit is evidence of nothing, and a record inside the failed system is unavailable when you need it (Audit Logs for Privileged Actions).
- Coverage is the hard part, not format. The trail is only as good as its least-covered path, and the least-covered path is always the one used during incidents.
- This is where release engineering and security overlap exactly: the operational need ("what changed?") and the security need ("who did that, and were they allowed to?") are answered by the same record, and building it twice is a common and avoidable waste.
Five fields, and the one everyone omits
The first four are what people build. The fifth is what makes the record usable at 3am, and it is nearly always missing because capturing it requires reading the old value before writing the new one.
One audit entry, from a flag change
1{2 "at": "2026-08-26T10:02:41Z",3 "actor": {4 "human": "alice@example.com",5 "via": "flag-console",6 "session": "sess-7c2f",7 "elevated": false8 },9 "action": "flag.cohort.update",10 "target": { "service": "checkout-api", "flag": "checkout_v2_reads" },11 "before": { "state": "on", "cohort": "10%" },12 "after": { "state": "on", "cohort": "50%" },13 "why": { "ref": "PROJ-4471", "note": "ramp step 2 of 4, canary clean" },14 "request_id": "01J9M2K...",15 "source_ip": "10.4.2.19"16}before is the field that makes this actionable: without it the entry says the cohort changed and not what to set it back to. actor.human alongside actor.via is the other one — automation acts on behalf of people, and an entry naming only the tool answers "what" while leaving "who decided" open.
Every path that changes production
CLOUD-SPECIFICProvider activity logs cover the console and API rows well, with a delay and a default retention that is usually shorter than an investigation window — extending it is a deliberate configuration, not a default. They do not cover application-level changes such as flags or in-app admin actions at all.
Coverage is the whole game, and the gaps are predictable: the paths that bypass the pipeline are exactly the paths that are quick, and quick is why people use them under pressure.
| Change path | Usually recorded? | Where the gap is |
|---|---|---|
| Deploy via pipeline | Yes | Emergency deploys that bypass it entirely (Break-Glass Access) |
| Config change | Sometimes | Config applied out of band, with no previous value captured (A Config Change Is a Production Change) |
| Feature flag change | Rarely with previous state | The highest-frequency behaviour change in most systems (Deployment Is Not Release) |
| Infrastructure via IaC | Yes, in version control | Manual console edits that diverge from it silently (Drift) |
| Provider console action | In a provider log | Nobody queries it; retention is short; it is not in the incident timeline |
| Database schema change | In a migration table | Ad-hoc SQL run by a person, which the migration tool never sees (Operating a Production Database) |
| Access grant | In an identity system | Temporary elevation that expires, leaving no trace of what it was used for (Least Privilege in Production) |
| Third-party or SaaS setting | In their system | Outside your trail entirely; changes there cause incidents in yours |
| Scheduled job enabled or disabled | Rarely | A disabled job is a silent change with delayed, confusing symptoms (Cron Jobs in Production) |
How trails fail in practice
Each of these produces a trail that looks healthy and is useless at the specific moment it is needed, which is why they are only ever discovered during an incident.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Incident begins, "nothing was deployed" | Behaviour changed with no corresponding deploy record | Flags and config are not in the trail | Instrument every behaviour-changing path, starting with flags (Deployment Is Not Release) |
| Change identified, cannot be reverted | You know what changed but not what it was | No previous-state field | Read the old value before writing the new one; make it a required field at every source |
Entry attributed to ci-deploy | Cannot determine who authorised the change | Machine identity with no human attribution | Carry the triggering human, run and commit on every automated entry (Human vs Machine Identities) |
| Merged timeline is out of order | A change appears to follow the symptom it caused | Clock skew or mixed timezones between sources | UTC everywhere, and monitor clock offset as a first-class signal (Clock Synchronisation) |
| Investigation reaches back six weeks | Relevant entries have aged out | Retention set for cost rather than for investigation length | Retain change events far longer than application logs — they are low volume and high value |
| The audited system is down | The trail is unavailable during the incident | The trail lives in the system it describes | Store it in a separate account or system, append-only (Audit Logs for Privileged Actions) |
| Too much to search | Finding the relevant change is itself the investigation | Automated changes dominate volume with no filter for the affected service | Index by target service and separate human from automated actors |
How to do it properly
Most important first.
- Enumerate every path that can change production behaviour, and check each one produces an entry: deploy, config, flags, infrastructure, schema, access grants, provider console, third-party integrations.
- Capture the previous value on every entry. Without it, an entry tells you something happened and not how to undo it.
- Resolve automation to a human. Every automated actor should carry the trigger — the run id, the commit, the requester.
- Make "why" a structured reference to a ticket, PR or incident, so the reason survives the person.
- Normalise timestamps to UTC and use a consistent identity across sources, so entries from different systems merge into one timeline (Production Time Is UTC).
- Send everything to one queryable place with a retention longer than your longest investigation, and store it where the described systems cannot modify it.
- Annotate observability with change events so a metric graph shows the changes on it, which is where the trail is actually consumed (Deploys on the Same Timeline as the Symptom).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Blast radius if this is wrongEveryone
One testEveryone
What contains it
Nothing — a missing trail does not cause an incident, it extends every incident. Its absence is measured in time to diagnosis, which is usually the largest controllable component of impact.
What can go wrong
Failure modes, including of the mitigation
- A trail with excellent coverage of the pipeline and none of the flag system, so the highest-frequency behaviour changes are invisible.
- Entries with no previous state, so an investigation identifies the change and still cannot say what to restore.
- Everything attributed to a service account, making every entry equally uninformative about who decided.
- Retention shorter than the investigation window — a change made six weeks ago, aged out three weeks ago, when the resulting problem surfaces today.
- Clock skew or mixed timezones across sources, so the merged timeline has events in the wrong order and someone reasons from a false causal chain (Clock Synchronisation).
- The mitigation failing: an audit system in the same account or cluster as the systems it audits, so an incident that takes those down takes the evidence with them.
- So much volume that finding the relevant change is itself the investigation, with no way to filter to changes that touched the affected service (The Log Bill and What It Is Buying).
Misreads this invites
- "We have logs, so we have an audit trail." Application logs record what the system did. An audit trail records what people and automation changed about it. Different questions, usually different systems (Structured Logging: Fields a Program Can Read).
- "It is a compliance requirement." It is also the fastest path from symptom to cause during an incident, and treating it as a compliance artefact is how it ends up with perfect coverage of the paths auditors ask about and none of the ones engineers use.
- "Git is the audit trail." Git covers code and, if you practise it, infrastructure. It does not cover flags, console actions, manual database changes or access grants (Drift).
- "We can reconstruct it from the deployment system." That covers one path. Every other path is exactly the one that produces the incident where "nothing was deployed".
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
How you know it worked
- Pick a recent incident and reconstruct every production change in the preceding hour, from one query, including flags and console actions.
- Every entry in that window has a previous-state value.
- Every automated entry resolves to a human or to an explicit unattended policy.
- A deliberate out-of-band change — someone edits something in a console — appears in the trail within minutes, unprompted.
- Retention exceeds the longest investigation you have actually run, not the shortest one you can imagine.
How you get back
- The trail is append-only, so it has no rollback and should not have one. Corrections are new entries.
- Its value to rollback is the previous-state field: it is what converts "this changed" into "restore it to this".
- Where a change has no automated rollback — a console edit, a manual grant — the trail is the only reconstruction available, which raises the importance of capturing the prior value at the moment of change rather than inferring it later.
What to automate, and what stays human
- Automate capture at every source. A trail that depends on people recording their own actions is complete only for the actions people remember.
- Automate normalisation and aggregation into one timeline, because the merge is where the value is and doing it by hand during an incident is the cost you are trying to remove.
- Automate the annotation of dashboards with change events (Deploys on the Same Timeline as the Symptom).
- Keep human: reading it, and deciding what a change means. The trail is evidence, not a diagnosis (Stop the Harm Before You Understand It).
What this costs
- Capturing previous state means reading it before every change, which is extra work at every source and occasionally an extra API call in a latency-sensitive path.
- Long retention and durable independent storage cost money, and the trail contains sensitive material — who accessed what — so it has its own access-control problem (Security-Safe Logging).
- High-volume automated changes can dominate the trail, making human changes harder to find unless the two are separable.
- Comprehensive coverage means instrumenting paths that are rarely used and easy to forget, and the ones you forget are the ones used in emergencies.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe five fields are universal. So is the failure mode: the gap is always in the path that bypasses the normal one, on every platform.
- ORG-SPECIFICRetention periods, what counts as an auditable event and who may read the trail are governed by policy and regulation. What is not a policy question is whether previous state is captured — that is an engineering decision, and it is the one that determines whether the record is operationally useful.
- CLOUD-SPECIFICProviders offer an API-level activity log that covers console and API actions with a delay and a default retention that is usually shorter than you want. It covers infrastructure changes well and application-level ones — flags, feature config, in-app admin actions — not at all, so it is a component of the trail rather than the trail.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.