The question this answers
When the application log says nothing useful, which infrastructure log stream holds the answer?
A customer reports that uploads failed for eleven minutes yesterday afternoon. Nothing appeared in the application error log, because the requests never reached the application. Something between the client and the process rejected them, and you have to find out what.
A record produced by the infrastructure itself rather than by your code: who called, what the edge decided, which target it chose, what the platform did to your workloads, and which deploy landed at 14:07.
Five streams, five different questions
Infrastructure logs are not one thing. They are produced by different components, at different layers, with different retention defaults and different access-control needs, and confusing them is how teams end up with a hundred gigabytes a day and still no answer. The useful mental model is to name the exact question each stream is the *only* source for — if two streams answer the same question, one of them is a cost line item you can delete.
The eleven-minute upload failure resolves in the load-balancer log, not the application log, because the requests were rejected at the edge: request body too large, or a target group with zero healthy members returning 503 before any of your code ran. That is the whole point of the stream. The application cannot log a request it never received, and no amount of application instrumentation will change that.
Access logs deserve one specific warning. They contain client addresses, user agents, request paths and often query strings, which makes them simultaneously the most operationally useful stream and the one most likely to accumulate personal data you did not intend to retain. Path parameters leak identifiers; query strings leak tokens if anyone was careless. Retention and access control on this stream are a data-protection decision, not a storage decision.
| Stream | Produced by | The question only it answers | Watch out for |
|---|---|---|---|
| Audit log | The control plane / IAM layer | Who changed infrastructure, what changed, when, and from where — see Audit Trails | Off or short-retention by default in more places than you would expect |
| Access log | Edge, CDN, reverse proxy, API gateway | Which requests arrived, from whom, and what status they got before your code ran | Personal data and tokens in paths and query strings |
| Load-balancer log | The load balancer itself | Which target was chosen, whether it was healthy, and whether the balancer or the target produced the error | Distinguishing a balancer-generated 502 from a target-generated one is the whole diagnostic value |
| Control-plane events | Scheduler / orchestrator / scaling group | Why a workload was moved, restarted, evicted or refused scheduling | Frequently retained for only an hour by default — the incident outlives the evidence |
| Deployment events | CI/CD system and the deploy target | Which artifact went where, at what time, under which identity | Correlating this against the others is how "it started at 14:07" becomes "the 14:06 deploy" — see The Pipeline as Infrastructure |
Reading them together is the technique
Individually each stream is a partial account. Together, ordered by timestamp, they reconstruct a causal chain that no single one contains. The dump below is that reconstruction for the upload incident: a deploy lands, the new pods fail their readiness gate because a configuration key is missing, the balancer drains the old targets faster than the new ones become ready, and for eleven minutes the target group is empty and every request is rejected at the edge with a 503 that your application never sees.
Note what makes this readable: consistent timestamps in a single timezone, and a request identifier that appears in both the edge log and the application log. Without the shared identifier, correlating a specific user complaint to a specific edge decision is manual and often impossible. Generating one at the edge and propagating it inward is a small piece of infrastructure work that pays for itself in the first incident.
The corresponding discipline is retention. Control-plane events are the stream most likely to be gone by the time anyone investigates — many platforms keep them for an hour. If those events are not exported somewhere durable, the only record that explains *why* the platform did what it did expires before your post-incident review starts.
14:06:12 deploy artifact sha256:9f31... -> checkout-api actor: ci-deployer result: accepted 14:06:40 control Scheduled pod checkout-api-7d9 node-b 14:06:44 control Unhealthy pod checkout-api-7d9 readiness probe failed: 500 14:06:51 control Killing pod checkout-api-4a2 (old revision, drained) 14:07:02 lb target group checkout-tg healthy: 0 unhealthy: 4 14:07:02 access POST /uploads status 503 target: - rid: 01HX3... bytes 0 14:07:02 lb 503 generated by load balancer (no healthy targets) 14:18:35 control Ready pod checkout-api-7d9 after config key UPLOAD_BUCKET restored 14:18:41 access POST /uploads status 201 target: 10.0.2.31 rid: 01HX9... bytes 4194304 application error log for 14:07-14:18: (empty)
Logs are a metered product
Infrastructure logging is one of the few systems that bills you more precisely when things are going worse. A retry storm multiplies access-log lines. A crash loop multiplies control-plane events. A debug flag left on in production multiplies everything. The volume that arrives during your worst hour is the volume you are least able to reason about and most likely to be charged for at an on-demand rate.
Three meters run: ingestion per gigabyte, retention per gigabyte-month, and query or scan cost. Ingestion usually dominates, and the standard mitigations are unglamorous — sample high-volume success paths, keep failures unsampled, drop health-check lines at the source rather than paying to store and then filter them, and set per-stream retention rather than one global policy. Shipping logs to an endpoint outside the provider network adds an egress charge on top, which is why the log pipeline is a recurring guest star in Egress: Moving Data Costs Money, Not Just Storing It.
The honest framing: decide the budget before the volume decides it for you. "Everything, forever, at full fidelity" is a choice with a price, and it is almost never the choice a team would make if someone had written the number down first.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Key points
- Five distinct streams — audit, access, load balancer, control-plane, deployment — each the only source for one specific question.
- Requests that fail before reaching your code exist only in the access and load-balancer logs; the application log is silent by construction.
- Control-plane events are usually retained for an hour or less by default, so the record explaining a platform decision often expires before the review.
- A request identifier generated at the edge and propagated inward is what turns four separate streams into one causal chain.
- Logging bills hardest exactly when the system is worst: incidents and retry storms are the volume spikes.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Edge components write one structured line per request at completion, containing client address, path, status, chosen target, bytes and elapsed time.
- • The load balancer distinguishes errors it generated itself (no healthy target, timeout waiting on target, request too large) from errors a target returned — this field is the diagnostic core of the stream.
- • The orchestrator or scaling group emits events as objects with a reason and a message, held in a short-lived buffer until exported.
- • The CI/CD system records artifact identity, target environment, acting identity and result at each promotion step.
- • A collection agent or native export path forwards each stream into a store with its own retention, index and access policy.
- • Export control-plane events to durable storage immediately; the default buffer will not survive to your post-incident review.
- • Set retention per stream. Audit logs are a compliance obligation measured in years; health-check access lines are worth hours.
- • Own the personal-data question on access logs: which fields are captured, who can query them, how long they live, and whether query strings are redacted at ingestion.
- • Keep timestamps in a single timezone across every stream. Cross-stream correlation dies on mixed local times faster than on anything else.
- • Budget the volume explicitly, and review the top talkers monthly — one chatty component usually accounts for most of the bill.
- • The stream was never enabled: load-balancer logging is off by default in several places, and you find out during the incident it would have explained.
- • Retention expired: the events that explain a platform decision are gone an hour later, leaving a timeline with a hole in exactly the interesting place.
- • Logs stored inside the failure domain they describe — a cluster-hosted log store is unavailable during the cluster outage you need it for.
- • Volume-triggered throttling: the pipeline starts dropping lines under the load spike that caused the incident, silently, so the record is incomplete precisely where it matters.
- • Secrets and personal data captured in query strings, discovered months later during an access review, now replicated across every backup of the log store.
- • Volume grows linearly with request rate and super-linearly during failure, because retries multiply lines while errors add them.
- • Index cost grows faster than storage cost; full-text indexing every field is what turns a large log bill into an enormous one.
- • Cross-region aggregation adds egress per gigabyte on top of ingestion, so a global fleet shipping to one region pays twice.
- • The dimension that runs out first is usually query performance, not storage: an unbounded search over months of access logs times out long before the disk fills.
- • Access logs are personal data in most jurisdictions: client addresses, user agents and paths identify people. Classify them and restrict them accordingly.
- • Log stores must be write-restricted and separately owned. An attacker who can delete logs erases the record of what they did — see Audit Trails.
- • Never log credentials, tokens or full request bodies from authenticated endpoints. Redaction at ingestion is the only redaction that works; redacting on read leaves the original at rest.
- • The collection agent has a broad read scope across every node. Treat its identity as a privileged workload identity, not as plumbing.
- • Ingestion, retention and query are three separate meters, and ingestion normally dominates.
- • Verbosity is the single biggest lever you actually control: debug logging left on in production has ended more log budgets than traffic growth has.
- • Sampling success paths while keeping every failure preserves nearly all diagnostic value at a fraction of the volume.
- • External logging destinations are billed twice — the vendor's ingestion and your provider's egress.
- • Ingestion volume per stream per day, with a per-component breakdown so a new top talker is obvious within a day.
- • Pipeline drop and throttle counters — a silently truncated log stream is worse than no log stream, because it looks complete.
- • Retention configuration as a checked property, not a memory. Streams get created without policies.
- • The signal that lies: a healthy-looking log dashboard during an outage. If the store is inside the failing blast radius, "no errors logged" means "no logs arrived".
- • For a single VM, the systemd journal plus the provider's built-in access log is enough. A centralized pipeline is not the starting point.
- • Structured application logs with a shared request id often answer more questions per gigabyte than raw access logs — start there and add streams as specific questions go unanswered.
- • Metrics instead of logs for anything you count: a counter of 503s by target costs a rounding error compared to storing every line and aggregating on read.
- • Sampling, not more storage, when volume is the problem. Full fidelity on failures and 1-in-N on successes keeps the answers and drops the bill.
- • Buys the ability to explain failures that happen outside your code; costs a metered pipeline that spikes exactly when the system is worst.
- • Longer retention buys post-incident and compliance answers, and creates a permanent, growing store of data that is mostly personal and read almost never.
- • Centralizing every stream in one place makes correlation easy and makes that place a single, highly attractive target with a broad read scope.
What people believe, and what is true
If it is not in the application log, it did not happen.
Requests rejected at the edge — oversized bodies, no healthy targets, TLS failures, rate limits — never reach your code. They exist only in the access and load-balancer logs.
Log everything; storage is cheap.
Ingestion and indexing are the meters, not storage, and both peak during incidents. Full-fidelity-forever is a budget decision people make by accident.
Control-plane events will be there when I need them.
Default retention is frequently an hour. If they are not exported, the explanation for a platform decision expires before the post-incident review begins.