Infrastructure Observability

Infrastructure Logs

Five log streams your application never writes — audit, access, load balancer, control-plane and deployment — and the specific question each one is the only source for. Plus the reason they are a top-five line item on the bill.

The question this answers

Infrastructure question

When the application log says nothing useful, which infrastructure log stream holds the answer?

Application requirement

A customer reports that uploads failed for eleven minutes yesterday afternoon. Nothing appeared in the application error log, because the requests never reached the application. Something between the client and the process rejected them, and you have to find out what.

What it provides

A record produced by the infrastructure itself rather than by your code: who called, what the edge decided, which target it chose, what the platform did to your workloads, and which deploy landed at 14:07.

Application RequirementInfrastructure RequirementComputeNetworkStorageIdentityDeploymentScalingReliabilityObservabilitySecurityCostTrade-offs

Five streams, five different questions

Infrastructure logs are not one thing. They are produced by different components, at different layers, with different retention defaults and different access-control needs, and confusing them is how teams end up with a hundred gigabytes a day and still no answer. The useful mental model is to name the exact question each stream is the *only* source for — if two streams answer the same question, one of them is a cost line item you can delete.

The eleven-minute upload failure resolves in the load-balancer log, not the application log, because the requests were rejected at the edge: request body too large, or a target group with zero healthy members returning 503 before any of your code ran. That is the whole point of the stream. The application cannot log a request it never received, and no amount of application instrumentation will change that.

Access logs deserve one specific warning. They contain client addresses, user agents, request paths and often query strings, which makes them simultaneously the most operationally useful stream and the one most likely to accumulate personal data you did not intend to retain. Path parameters leak identifiers; query strings leak tokens if anyone was careless. Retention and access control on this stream are a data-protection decision, not a storage decision.

StreamProduced byThe question only it answersWatch out for
Audit logThe control plane / IAM layerWho changed infrastructure, what changed, when, and from where — see Audit TrailsOff or short-retention by default in more places than you would expect
Access logEdge, CDN, reverse proxy, API gatewayWhich requests arrived, from whom, and what status they got before your code ranPersonal data and tokens in paths and query strings
Load-balancer logThe load balancer itselfWhich target was chosen, whether it was healthy, and whether the balancer or the target produced the errorDistinguishing a balancer-generated 502 from a target-generated one is the whole diagnostic value
Control-plane eventsScheduler / orchestrator / scaling groupWhy a workload was moved, restarted, evicted or refused schedulingFrequently retained for only an hour by default — the incident outlives the evidence
Deployment eventsCI/CD system and the deploy targetWhich artifact went where, at what time, under which identityCorrelating this against the others is how "it started at 14:07" becomes "the 14:06 deploy" — see The Pipeline as Infrastructure
What each infrastructure log stream is the only source for

Reading them together is the technique

Individually each stream is a partial account. Together, ordered by timestamp, they reconstruct a causal chain that no single one contains. The dump below is that reconstruction for the upload incident: a deploy lands, the new pods fail their readiness gate because a configuration key is missing, the balancer drains the old targets faster than the new ones become ready, and for eleven minutes the target group is empty and every request is rejected at the edge with a 503 that your application never sees.

Note what makes this readable: consistent timestamps in a single timezone, and a request identifier that appears in both the edge log and the application log. Without the shared identifier, correlating a specific user complaint to a specific edge decision is manual and often impossible. Generating one at the edge and propagating it inward is a small piece of infrastructure work that pays for itself in the first incident.

The corresponding discipline is retention. Control-plane events are the stream most likely to be gone by the time anyone investigates — many platforms keep them for an hour. If those events are not exported somewhere durable, the only record that explains *why* the platform did what it did expires before your post-incident review starts.

14:06:12  deploy    artifact sha256:9f31... -> checkout-api  actor: ci-deployer  result: accepted
14:06:40  control   Scheduled       pod checkout-api-7d9  node-b
14:06:44  control   Unhealthy       pod checkout-api-7d9  readiness probe failed: 500
14:06:51  control   Killing         pod checkout-api-4a2  (old revision, drained)
14:07:02  lb        target group checkout-tg  healthy: 0  unhealthy: 4
14:07:02  access    POST /uploads  status 503  target: -           rid: 01HX3...  bytes 0
14:07:02  lb        503 generated by load balancer (no healthy targets)
14:18:35  control   Ready           pod checkout-api-7d9  after config key UPLOAD_BUCKET restored
14:18:41  access    POST /uploads  status 201  target: 10.0.2.31   rid: 01HX9...  bytes 4194304

application error log for 14:07-14:18:  (empty)
ILLUSTRATIVE — one incident, four streams, ordered by time

Logs are a metered product

Infrastructure logging is one of the few systems that bills you more precisely when things are going worse. A retry storm multiplies access-log lines. A crash loop multiplies control-plane events. A debug flag left on in production multiplies everything. The volume that arrives during your worst hour is the volume you are least able to reason about and most likely to be charged for at an on-demand rate.

Three meters run: ingestion per gigabyte, retention per gigabyte-month, and query or scan cost. Ingestion usually dominates, and the standard mitigations are unglamorous — sample high-volume success paths, keep failures unsampled, drop health-check lines at the source rather than paying to store and then filter them, and set per-stream retention rather than one global policy. Shipping logs to an endpoint outside the provider network adds an egress charge on top, which is why the log pipeline is a recurring guest star in Egress: Moving Data Costs Money, Not Just Storing It.

The honest framing: decide the budget before the volume decides it for you. "Everything, forever, at full fidelity" is a choice with a price, and it is almost never the choice a team would make if someone had written the number down first.

Where the logging bill comes from. Relative weights, not currency.COST-VARIES
Ingestion · surprisespiky
driven by GB accepted per day × request rate × verbosity · Peaks precisely during incidents and retry storms, when volume is highest and value per line is lowest.
Retention fixed
driven by GB stored × months kept · A flat, permanent tail that grows with every day you never revisit. Per-stream policies beat one global setting.
Query / scan usage
driven by GB scanned per search × investigations · Cheap until someone runs an unbounded search across a quarter of access logs.
Egress to an external destination · surpriseusage
driven by GB shipped out of the provider network · A third-party logging vendor is billed twice: their ingestion, and your provider's egress.
Health-check noise · surprisefixed
driven by probe interval × targets × 86400 · Pure overhead. Drop it at the source; storing it and filtering it later still costs ingestion.

Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.

Key points

  • Five distinct streams — audit, access, load balancer, control-plane, deployment — each the only source for one specific question.
  • Requests that fail before reaching your code exist only in the access and load-balancer logs; the application log is silent by construction.
  • Control-plane events are usually retained for an hour or less by default, so the record explaining a platform decision often expires before the review.
  • A request identifier generated at the edge and propagated inward is what turns four separate streams into one causal chain.
  • Logging bills hardest exactly when the system is worst: incidents and retry storms are the volume spikes.

The loop, answered

Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.

How it works
  • Edge components write one structured line per request at completion, containing client address, path, status, chosen target, bytes and elapsed time.
  • The load balancer distinguishes errors it generated itself (no healthy target, timeout waiting on target, request too large) from errors a target returned — this field is the diagnostic core of the stream.
  • The orchestrator or scaling group emits events as objects with a reason and a message, held in a short-lived buffer until exported.
  • The CI/CD system records artifact identity, target environment, acting identity and result at each promotion step.
  • A collection agent or native export path forwards each stream into a store with its own retention, index and access policy.
What you still own
  • Export control-plane events to durable storage immediately; the default buffer will not survive to your post-incident review.
  • Set retention per stream. Audit logs are a compliance obligation measured in years; health-check access lines are worth hours.
  • Own the personal-data question on access logs: which fields are captured, who can query them, how long they live, and whether query strings are redacted at ingestion.
  • Keep timestamps in a single timezone across every stream. Cross-stream correlation dies on mixed local times faster than on anything else.
  • Budget the volume explicitly, and review the top talkers monthly — one chatty component usually accounts for most of the bill.
How it fails
  • The stream was never enabled: load-balancer logging is off by default in several places, and you find out during the incident it would have explained.
  • Retention expired: the events that explain a platform decision are gone an hour later, leaving a timeline with a hole in exactly the interesting place.
  • Logs stored inside the failure domain they describe — a cluster-hosted log store is unavailable during the cluster outage you need it for.
  • Volume-triggered throttling: the pipeline starts dropping lines under the load spike that caused the incident, silently, so the record is incomplete precisely where it matters.
  • Secrets and personal data captured in query strings, discovered months later during an access review, now replicated across every backup of the log store.
How it scales
  • Volume grows linearly with request rate and super-linearly during failure, because retries multiply lines while errors add them.
  • Index cost grows faster than storage cost; full-text indexing every field is what turns a large log bill into an enormous one.
  • Cross-region aggregation adds egress per gigabyte on top of ingestion, so a global fleet shipping to one region pays twice.
  • The dimension that runs out first is usually query performance, not storage: an unbounded search over months of access logs times out long before the disk fills.
Security
  • Access logs are personal data in most jurisdictions: client addresses, user agents and paths identify people. Classify them and restrict them accordingly.
  • Log stores must be write-restricted and separately owned. An attacker who can delete logs erases the record of what they did — see Audit Trails.
  • Never log credentials, tokens or full request bodies from authenticated endpoints. Redaction at ingestion is the only redaction that works; redacting on read leaves the original at rest.
  • The collection agent has a broad read scope across every node. Treat its identity as a privileged workload identity, not as plumbing.
Cost shape
  • Ingestion, retention and query are three separate meters, and ingestion normally dominates.
  • Verbosity is the single biggest lever you actually control: debug logging left on in production has ended more log budgets than traffic growth has.
  • Sampling success paths while keeping every failure preserves nearly all diagnostic value at a fraction of the volume.
  • External logging destinations are billed twice — the vendor's ingestion and your provider's egress.
What to watch
  • Ingestion volume per stream per day, with a per-component breakdown so a new top talker is obvious within a day.
  • Pipeline drop and throttle counters — a silently truncated log stream is worse than no log stream, because it looks complete.
  • Retention configuration as a checked property, not a memory. Streams get created without policies.
  • The signal that lies: a healthy-looking log dashboard during an outage. If the store is inside the failing blast radius, "no errors logged" means "no logs arrived".
Simpler alternatives
  • For a single VM, the systemd journal plus the provider's built-in access log is enough. A centralized pipeline is not the starting point.
  • Structured application logs with a shared request id often answer more questions per gigabyte than raw access logs — start there and add streams as specific questions go unanswered.
  • Metrics instead of logs for anything you count: a counter of 503s by target costs a rounding error compared to storing every line and aggregating on read.
  • Sampling, not more storage, when volume is the problem. Full fidelity on failures and 1-in-N on successes keeps the answers and drops the bill.
What adopting this costs
  • Buys the ability to explain failures that happen outside your code; costs a metered pipeline that spikes exactly when the system is worst.
  • Longer retention buys post-incident and compliance answers, and creates a permanent, growing store of data that is mostly personal and read almost never.
  • Centralizing every stream in one place makes correlation easy and makes that place a single, highly attractive target with a broad read scope.

What people believe, and what is true

Claim

If it is not in the application log, it did not happen.

Reality

Requests rejected at the edge — oversized bodies, no healthy targets, TLS failures, rate limits — never reach your code. They exist only in the access and load-balancer logs.

Claim

Log everything; storage is cheap.

Reality

Ingestion and indexing are the meters, not storage, and both peak during incidents. Full-fidelity-forever is a budget decision people make by accident.

Claim

Control-plane events will be there when I need them.

Reality

Default retention is frequently an hour. If they are not exported, the explanation for a platform decision expires before the post-incident review begins.

Apply it