The question this answers
Who changed this infrastructure, what did they change, when, and from where?
At 09:40 the production database became reachable from the internet. Nobody on the team says they did it. Before anything can be fixed permanently, someone has to establish whether this was a human with a laptop, a pipeline applying a merged change, or a compromised credential — and the answer determines whether this is a code review problem or an incident response.
An append-only record of every control-plane action: the identity that made the call, the exact API action and parameters, the resource it touched, the timestamp, the source address, and whether it succeeded or was denied.
The anatomy of one audit event
An audit log is not an application log with more fields. It is a record of *control-plane* calls — the API operations that create, modify and delete infrastructure — captured by the provider on the far side of your account boundary so that it records the call even when the caller would rather it did not. That property is what makes it evidence rather than telemetry.
Read one event and the four questions answer themselves. The identity block answers *who*. The action and request parameters answer *what*. The timestamp answers *when*. The source address, user agent and session context answer *from where* — and that last field is often the one that resolves the incident, because "from the CI runner's address range" and "from a residential address in another country at 03:00" are different investigations.
The denied events matter as much as the successful ones, and teams routinely filter them out as noise. A burst of AccessDenied from one identity across many unrelated resources is what enumeration looks like: something is probing what it can reach. A successful action is a fact; a pattern of denials is a warning.
{
"eventTime": "2026-08-24T09:40:17Z", <- WHEN
"identity": { <- WHO
"type": "AssumedRole",
"principal": "role/platform-deployer",
"sessionIssuer": "user/j.novak",
"mfaAuthenticated": false
},
"action": "network:AuthorizeIngress", <- WHAT
"resource": "security-group/sg-prod-db",
"parameters": { "cidr": "0.0.0.0/0", "port": 5432, "protocol": "tcp" },
"sourceAddress": "203.0.113.44", <- FROM WHERE
"userAgent": "terraform/1.9 (+darwin)",
"result": "Success",
"requestId": "b1f0e7c2-..."
}
nearby, same identity, previous 90 seconds:
09:38:51 iam:ListRoles result: AccessDenied
09:39:04 storage:ListBuckets result: AccessDenied
09:39:22 secrets:ListSecrets result: AccessDeniedThe identity in the log is only as good as your identity model
Here is the failure that makes audit logs useless without anyone noticing. The log faithfully records deploy-key-1 as the actor. Six services and four engineers share deploy-key-1. The log answered *who* with a string that identifies nobody. You have a perfect record of an action and no way to attribute it, which in an incident is nearly the same as having no record at all.
Attribution is therefore a property of your identity design, not of your logging configuration. Every workload gets its own identity; every human authenticates as themselves and assumes a role rather than sharing a credential; every pipeline runs as a distinct pipeline identity. Only then does the actor field mean something — see Human vs Workload Identity and Roles vs Static Keys. This is the single strongest practical argument for short-lived role credentials over long-lived shared keys, stronger than the rotation argument people usually make.
The panel below shows the shape that preserves attribution: a human identity assumes a deployment role, so the audit event carries both the role that acted and the human who assumed it. Compare that to a static key stored in a shared secret manager entry, where the trail ends at the key. Note also the deliberate denial: the deployment role cannot modify the audit configuration itself. An identity that can turn off logging can erase its own history, which is the first thing a competent attacker does.
- compute:*, network:*, storage:* within the production scope
- iam:PassRole limited to the roles this pipeline is allowed to attach
- Exactly the resource types this pipeline provisions, scoped to production, with session credentials that expire in under an hour
- audit:StopLogging, audit:DeleteTrail, audit:PutConfiguration
- logs:Delete* on the audit log destination
- iam:CreateUser, iam:CreateAccessKey — no minting of non-attributable credentials
Blast radius: Broad by design — this identity can rebuild production. What keeps it survivable is that it cannot mint new credentials, cannot silence the audit trail, and produces sessions traceable to the human or pipeline that assumed it. Take away any one of those and the blast radius becomes unbounded and invisible.
A log nobody reads is a storage bill
This is the uncomfortable part, and it is the reason most audit programmes fail: enabling the trail is roughly 5% of the work. The other 95% is routing it somewhere durable and outside the account it describes, retaining it long enough that an intrusion discovered months later is still investigable, alerting on the handful of events that should never happen quietly, and actually reviewing it on a schedule. An organization that enabled audit logging in 2023 and has never opened it has bought storage, not accountability.
The alerting layer is where the value concentrates, because the useful events are rare and specific. A security group opened to 0.0.0.0/0 on a database port. A change to the audit configuration itself. A root or break-glass credential used at all. A new identity created outside the pipeline. An access key minted. A first-ever API call from an unfamiliar region. Each of those is a small number of events per year and each is worth waking someone for. Everything else is material for review, not for the pager.
The lifecycle below is the honest checklist. Most teams stop after retain and believe they are done — and it is exactly the gap between retain and alert where the 09:40 incident sits, discovered by a customer report rather than by the log that recorded it perfectly at the time.
- 1Emitimmediate
The control plane records every API call — successful and denied — including calls made by the provider console, the CLI, the SDK and your IaC tooling.
Coverage gaps: a region, an account or a service where the trail was never enabled. Enable organization-wide, not per project.
- 2Route out of the blast radiusseconds to minutes
Deliver to a store in a separate account or project, write-only from the source, with object-lock or equivalent immutability.
Logs stored inside the account they describe can be deleted by whoever compromises that account.
- 3Retainmonths to years
Keep long enough to investigate an intrusion discovered late — commonly a year or more, driven by regulation as much as by forensics.
Short retention silently converts a solvable investigation into a shrug. Median intrusion dwell time exceeds most default retention windows.
- 4Alert on the small set that should never happen quietlyminutes
Public exposure of a data port, audit configuration changes, break-glass credential use, new identities or access keys, calls from unfamiliar regions.
This is the step most programmes skip. Without it, the trail explains an incident after a customer found it.
- 5Review on a schedulemonthly or quarterly
A recurring human pass over privileged actions, denied-access patterns and identities that have stopped being used.
Slow-burn problems — a role that quietly accumulated permissions, a key still active for a departed contractor — are invisible to alerting and only surface here.
- 6Actdays
Every finding closes as a change: a narrowed policy, a revoked credential, a guardrail that makes the action impossible rather than merely recorded.
Findings that produce no change train the organization that the review is theatre.
Key points
- An audit log answers four questions — who, what, when, from where — about control-plane calls, recorded by the provider outside your workload's reach.
- Attribution is an identity-design property: a shared key makes the actor field a string that identifies nobody.
- Denied events are signal, not noise. A spread of AccessDenied across unrelated resources is enumeration.
- Store the trail outside the account it describes, immutably, or an account compromise deletes its own history.
- Enabling the log is a small fraction of the work. Routing, retention, alerting on the rare never-events, and scheduled review are the rest — and skipping alerting is how a perfect record still fails to prevent anything.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Every control-plane API call passes through the provider's authorization layer, which records the request before and independently of executing it.
- • The event captures the caller identity (including the session issuer when a role was assumed), the action, the resource, the request parameters, the source address, the user agent and the result.
- • Denied calls are recorded with the same fidelity as successful ones, which is what makes enumeration detectable.
- • Events are delivered to a configured destination — object storage, a log store or an event stream — typically within minutes.
- • Immutability is enforced at the destination through object lock or retention policy, not by the emitting service, which is why the destination choice is the security decision.
- • Enable across every account, project and region — including the ones nobody uses, because that is where unexpected activity shows up first.
- • Deliver into a separate security account with write-only access from source accounts and no delete permission for anyone in the day-to-day path.
- • Maintain the never-event alert list and test it: deliberately open a non-production security group and confirm the alert fires. An untested detection is a hope.
- • Keep a break-glass identity that is alarmed on use rather than removed. You will need it one day; you should hear about it every time it is used.
- • Schedule the review and record its outcome. A review with no written findings did not happen.
- • Enabled in one region or one account and assumed to be global; the activity you care about happens in the unmonitored one.
- • Delivery to a destination inside the same account, deleted alongside everything else during the compromise it was meant to document.
- • Retention shorter than time-to-discovery: the intrusion is found in month seven and the evidence expired in month three.
- • Attribution collapse: the actor is a shared static key, so the log establishes that something happened and nothing about who did it.
- • Alert fatigue from logging every successful call, followed by muting, followed by the one real event arriving into a muted channel.
- • Out-of-band changes that bypass the control plane entirely — someone editing a config file on a VM by hand leaves no audit event at all, only Drift: When the File and Reality Disagree.
- • Event volume grows with automation, not with traffic: an IaC pipeline running every merge generates far more control-plane calls than human operators ever did.
- • Cross-account aggregation multiplies volume by account count, and the storage tail grows monotonically because retention is measured in years.
- • Query cost becomes the practical limit long before storage does — searching a year of events across dozens of accounts needs partitioning by time and account or it simply times out.
- • The dimension that runs out first is human attention, which is why the never-event alert list must stay short enough to be believed.
- • The audit trail is the evidence base for every infrastructure investigation. Its integrity is a higher-order concern than its completeness.
- • No production identity may modify audit configuration or delete audit data. This denial belongs in policy, not in a runbook.
- • Audit events contain identity names, source addresses and resource identifiers — a map of your organization and its infrastructure. Restrict read access to the people who investigate.
- • Cross-link the Security domain's treatment of audit logging, detection engineering and incident response; this lesson covers only the infrastructure control plane.
- • The trail records control-plane actions, not data-plane access. Who read a row in the database is a different log with different retention — see Infrastructure Logs.
- • Management-plane events are commonly included at no charge; data-plane events (every object read, every secret fetch) are billed per event and can dwarf everything else.
- • The dominant meter is long retention of a stream that grows with automation and is queried rarely — a permanent, slowly rising baseline.
- • Lifecycle the destination: recent events in a queryable tier, older ones in archive. Mind the retrieval cost before you need them — see Storage Lifecycle: Hot, Warm, Archive, Delete.
- • Detection tooling on top of the trail is usually priced per event ingested, so the alerting layer costs more than the logging layer.
- • Trail health itself: delivery failures, configuration changes and any gap in event continuity — a silent trail looks identical to a quiet account.
- • Denied-action rate per identity, which surfaces both misconfiguration and enumeration.
- • Privileged and break-glass credential use, alarmed rather than reported.
- • Identities with no activity for 90 days, which are credentials waiting to be abused.
- • The signal that lies: an empty audit view. It usually means delivery broke, a region was never enabled, or the change was made outside the control plane — not that nothing happened.
- • For a very small system, provider audit logging enabled with delivery to object storage and one alert on public-exposure changes covers most of the value for almost none of the effort. Do that before buying a detection platform.
- • Enforce the change path instead of watching it: if production is only reachable through a reviewed pipeline, the audit trail becomes confirmation rather than the primary control. Prevention beats detection where it is available — see Reading a Plan Before You Apply It.
- • Version control history plus pull-request review already answers who-changed-what for everything managed as code. The audit trail is then most valuable for the changes that bypassed it.
- • If you need to know who read data rather than who changed infrastructure, data-plane audit logging in the database is the right tool and this is the wrong one.
- • Buys attribution and a forensic timeline; costs a permanent retention bill, a detection pipeline priced per event, and the discipline to actually read it.
- • Immutable, cross-account storage is what makes the trail trustworthy and what makes it awkward to manage — you deliberately gave up the ability to clean it up.
- • Alerting on rare events means most alerts are false starts on a quiet system, and the temptation to widen thresholds is exactly how the list stops being believed.
- • Data-plane audit events give far deeper visibility at a cost that can exceed the workload they observe.
What people believe, and what is true
We enabled audit logging, so we have accountability.
Accountability needs attribution (per-identity credentials), integrity (storage outside the blast radius) and attention (alerts plus review). Enabling the trail supplies none of those on its own.
Denied events are noise.
A spread of denials from one identity across unrelated services is the clearest available signature of enumeration, and it is free to detect.
The audit log shows everything that happened.
It shows control-plane API calls. Someone editing a file on a VM, or a change made through a path that bypasses the API, leaves no event — only drift.