Operationsloggingstructured logsredactionsecretspii

API Logging Without Leaking

One structured line per request: operation, status, duration, request id, principal, safe context. The hard part is the discipline of absence — no tokens, no passwords, no full bodies — because logs are the widest-read, longest-retained copy of your traffic.

Follow the failure

Frame the contract

API design starts with a consumer, a design question and a guarantee — never with a URL.

Design question
Does every request leave exactly one useful, queryable record — and is it impossible for that record to contain a credential?
Consumers
The on-call engineer pivoting from a metric spike to specific failing requests; support resolving a ticket from a request id; the security team, for whom every log line is both an audit record and a potential leak.
The promise
Every request produces one structured, request-id-keyed line that supports debugging and audit — and the logging pipeline structurally cannot capture credentials, secrets or sensitive payloads, rather than merely promising not to.
Requirement→Consumers→Resource Model→Style→Contract→Validation→Authorization→Errors→Idempotency→Pagination→Versioning→Observability→Evolution→Trade-offs

One line per request, structured, keyed

The workhorse of API observability is the access log: exactly one structured record per request, emitted at completion, carrying the fields every investigation needs. Structured means machine-parseable key-value (JSON or equivalent), not prose — "payment failed for user" cannot be filtered, aggregated or joined; {"route": "/payments", "status": 502, "duration_ms": 3012} can. The non-negotiable key is the Request IDs: The Contract's Correlation Clause value, because the access log is the table every other log line joins against.

The useful field set is boringly stable: timestamp, request id, route *template* plus the specific resource ids (unlike API Metrics: Rate, Errors, Duration, Sizes labels, logs are events — high-cardinality values like project_id are welcome here; this is exactly the detail metrics must refuse), method, status, duration, principal (user/key/service identity — the *identifier*, never the credential), response bytes, error code from your The Error Model: Structure Over Apology when present, and caller metadata (IP, user agent, API version). One line that answers who, what, which one, how long, and how it ended.

One request, one record — everything an investigation joins on
{
  "ts": "2026-08-25T14:47:03.219Z",
  "request_id": "req_01HQX4T9GJ8Z",
  "method": "POST",
  "route": "/projects/{id}/tasks",
  "project_id": "prj_42",
  "status": 409,
  "error_code": "version_conflict",
  "duration_ms": 38,
  "principal": "key_live_7f3a",        // identifier, never the secret
  "api_version": "2026-06-01",
  "resp_bytes": 412,
  "ip": "203.0.113.7"
}

The discipline of absence

Logs are the most dangerous data store you run, precisely because nobody thinks of them as one. They are copied to aggregators, indexed for search, readable by half of engineering, retained for months, and shipped to third-party vendors — every property you would never grant your production database. A bearer token in a log line is a credential with a 90-day lifetime and a full-text index; industry postmortems are littered with exactly this (major providers have had to mass-revoke credentials after finding passwords in plaintext logs). The blast radius of a logged secret is the blast radius of the log system, and a token that leaks via logs can be replayed by anyone who can read them — the same theft Session Hijacking describes, self-inflicted.

The forbidden list: credentials in any form (Authorization headers, API keys, session cookies, signatures), password fields even on failed logins (failed attempts are often *almost*-correct passwords), full request/response bodies by default (bodies are where PII and secrets live), tokens inside URLs (query strings leak through logs on every hop — a reason Authentication in the Contract favors headers over query parameters), and regulated PII beyond what the record's purpose requires. When body context is genuinely needed for debugging, log named, allowlisted fields — never the raw body.

The load-bearing word is *structurally*. A policy that says "please don't log tokens" fails at the first 02:00 console.log(req.headers) committed during an incident. The controls that work are mechanical: serializers that redact denylisted keys and header names by construction, an *allowlist* of loggable fields at the logging library layer (allowlists fail closed when a new sensitive field appears; denylists fail open), redaction in the shared middleware everyone inherits, and scanners over the log stream that page when something token-shaped appears anyway.

One incident-night debug line, retained for 90 days
1log.info("payment request", {
2  headers: req.headers,        // Authorization: Bearer eyJhbGc…
3  body: req.body,              // card number, name, address
4  user: req.user               // full record incl. email, phone
5})
6// → indexed, searchable, readable by all of eng,
7//   shipped to a third-party log vendor, kept 90 days
Allowlisted serializer: the leak is unrepresentable
1log.request(req, res, {
2  // serializer emits ONLY schema fields:
3  // request_id, route, status, duration_ms,
4  // principal_id, amount, currency, error_code
5})
6// headers/body are not reachable from the log API;
7// "Authorization" and "password" keys redact by
8// construction if they ever appear in context

The left line was reasonable at 02:00 and catastrophic for a quarter. The fix is not vigilance — it is a logging API where the dangerous fields are not expressible, so the 02:00 commit is safe by construction.

Volume, retention and the division of labor

Logs are the expensive tier of telemetry — every event, stored and indexed — so decide deliberately what runs at 100% and what samples. The access log earns 100%: it is the audit trail and the join target, and a sampled audit trail is not one. Debug and diagnostic logging inside handlers should be level-gated and sampled, with an escape hatch: errors always log fully (with redaction), and a per-request debug flag or dynamic level lets you turn one consumer's traffic verbose during an investigation without drowning in everyone else's.

Retention is a contract with your legal and security teams as much as an ops setting: access records might keep 30–90 days hot for debugging and longer cold for audit, while anything carrying user-identifiable data inherits deletion obligations — a user-erasure request extends to your logs, which is one more argument for never logging more of the user than the record's purpose requires. And keep the division of labor sharp: metrics aggregate (API Metrics: Rate, Errors, Duration, Sizes), logs record events, traces decompose latency (Request IDs: The Contract's Correlation Clause is the key that joins all three). The log line that tries to be a metrics system produces dashboards built on grep, and the metric that tries to be a log produces the cardinality bomb.

  • Access log at 100% — it is the audit trail; sampling it makes both debugging and audit probabilistic.
  • Debug logs sampled and level-gated — with always-on-error and per-request verbose switches for investigations.
  • Retention by data class — operational records vs anything PII-bearing; user erasure reaches logs too.
  • Errors log richly, never rawly — full error context passes through the same redacting serializer as everything else.
  • Log volume is a metric — a 10× log-rate spike is either an incident or a runaway logger; both page.

Key points

  • One structured record per request — operation, status, duration, request id, principal identifier, safe context — is the join table of all API debugging.
  • Logs welcome the high-cardinality detail (resource ids, specific requests) that metrics must refuse; that is the division of labor.
  • Never log credentials, passwords (especially failed ones), full bodies, or tokens in URLs — logs are indexed, widely readable, long-retained, and often shipped to third parties.
  • Enforce absence structurally: allowlist serializers and middleware redaction by construction, not code-review vigilance; allowlists fail closed, denylists fail open.
  • Access logs run at 100%; diagnostic logging is sampled and level-gated with an always-on-error path.
  • Retention is a legal surface: PII in logs inherits deletion obligations, which is one more reason to log identifiers, not records.

Follow the failure

How the contract fails or gets misused, hop by hop — and what it costs when it completes.

  1. 1
    Team → services: each service logs ad hoc — prose messages, occasional full request dumps, no shared serializer.
  2. 2
    Engineer → incident: at 02:00, adds log.info(req.headers, req.body) to debug a payment failure; the fix works, the line ships.
  3. 3
    Logs → aggregator: three months of Authorization headers and card fields are now indexed, searchable, and replicated to the log vendor.
  4. 4
    Security review → team: a routine audit (or worse, an attacker with log access) finds live bearer tokens; every credential that passed through the endpoint must be rotated.
  5. 5
    Org → consumers: mass token revocation forces every consumer to re-authenticate; the disclosure email explains that the API's own logs were the leak.
What breaks
  • A logged credential converts log-read access into API access — the log system's (usually generous) ACL becomes the API's effective auth boundary.
  • Incident response degrades without the join: no request id, no principal, no error code means every investigation starts from grep and hope.
  • Unbounded logging costs real money and drowns signal: the team that logs everything reads nothing, and the aggregator bill becomes its own incident.

Design, observe, evolve

A contract decision is incomplete until you know how you would notice it failing and how it changes later.

Design the contract
  • • Ship one shared logging middleware that emits the standard access record with redaction built in; services inherit it rather than reimplementing it.
  • • Allowlist loggable fields at the serializer layer and redact denylisted keys (`authorization`, `password`, `token`, `cookie`, `set-cookie`) by construction as defense in depth.
  • • Run secret-pattern scanners over the live log stream and page on hits — the control that catches what construction missed.
  • • Classify log fields by sensitivity at schema time and let retention and access policy follow the classification automatically.
Observe in production
  • • Scanner hit rate for token/secret patterns in logs — the target is zero, and any hit is a rotate-now event.
  • • Log volume and bytes per service per release: a step change means a new logger in a hot path, a new failure mode, or a new leak.
  • • Coverage joins: percentage of 5xx responses whose request id resolves to a complete access record — gaps here are your blind spots during the next incident.
Evolve without breaking
  • • Adding fields to a structured schema is additive; renaming or retyping breaks saved queries, dashboards and alerts — version the log schema like the internal API it is.
  • • Tightening redaction is safe and should ship immediately; loosening it (logging a new body field) goes through the same review a new PII store would get.
  • • As the fleet grows, promote the access-record schema into the platform (sidecar, gateway, or library) so consistency survives team turnover — one schema, joined on one id, across every service.
What it costs
  • • Allowlist logging occasionally omits the one field that would have explained a novel bug — the escape hatch is per-request debug verbosity, not weakening the default.
  • • Redaction, scanning and schema discipline are real engineering overhead versus `console.log` — paid continuously, repaid at the first prevented leak and every fast incident.
  • • 100% access logging at high traffic is a real bill; the alternatives (sampling the audit trail, shortening retention) trade money for investigative and compliance blind spots.

Misconceptions

Claim
“Our logs are internal — only engineers can read them, so sensitive data is fine.”
Reality
"Internal" logs are indexed, retained for months, replicated to vendors, and readable far more broadly than production data stores. A token in a log is a credential with the log system's ACL, which is to say: barely one.
Claim
“Log everything now, filter later — storage is cheap and you never know what you'll need.”
Reality
Storage is cheap; indexed retention, third-party shipping and liability are not. "Everything" includes credentials and PII, and once logged they exist in every downstream copy. The record you need is the structured access line — deliberately designed, not accidentally captured.
Claim
“Redacting on failed login attempts is unnecessary — the password was wrong anyway.”
Reality
Failed passwords are near-misses: typos of real credentials, or a user's valid password for a different account. A log of failed attempts is a cracking wordlist sorted by likelihood.
← API Metrics: Rate, Errors, Duration, SizesTesting the Contract, Not Just the Code →