Logsstructured loggingjsonfieldsqueryableschema

Structured Logging: Fields a Program Can Read

A log line is either a sentence a human greps or a record a program queries. The difference decides whether "how many payment timeouts hit provider X in the last hour" takes ten seconds or an afternoon of regex archaeology.

Follow the diagnosis

Frame the diagnosis

Performance work starts from a symptom and a signal — never from a resource dashboard.

Diagnostic question
Can a program answer questions about my logs, or only a human with grep and patience?
Symptom
During an incident, answering "how many checkouts failed and why" means a chain of greps, cuts and sorts against free-form text — and the answer is only as good as the regex someone wrote under pressure.
Signal
Whether the log store can aggregate by field. The misleading comfort is that the logs "contain the information" — containing it and being able to query it are different properties.
Symptom→Signal→Measurement→Hypothesis→Evidence→Root Cause→Change→Validation→Regression Check

The same event, two ways

A prose log line is written for one reader at one moment: an engineer tailing a file. It reads well and aggregates terribly. The moment the question becomes "how many, grouped by provider, over the last hour", prose forces you to reverse-engineer a parser out of a format nobody specified, on a format that changes whenever someone edits the string.

A structured log line is a record with named fields. It reads slightly worse in a terminal and answers questions in one query. The trade is real but heavily one-sided, because the situations where log ergonomics matter most — incidents, audits, capacity questions — are exactly the situations that involve aggregation rather than reading.

The property that matters is stable field names with stable meanings. duration_ms must always be a number of milliseconds in every service that emits it; if one service emits seconds, every aggregate over both is wrong and nothing will tell you. This is the same inter-service schema problem as Label Sets That Survive a Year, with the same fix: shared helpers rather than shared discipline.

Prose: readable once, unqueryable forever
1log.error("Payment failed for order 8412 after 4.2s: provider timeout")
2log.error("payment error - order=8419, took 3.9 seconds (timeout from provider)")
3log.error("Could not process payment (order 8431) - upstream timed out")
4 
5# Three engineers, three formats, one concept.
6# "How many payment timeouts in the last hour, by provider?"
7#   -> write a regex per format
8#   -> hope no fourth format exists
9#   -> the answer is a guess with a confidence interval nobody states
Structured: one query, and it stays correct
1log.error("payment_failed", {
2  event:       "payment_failed",
3  order_id:    "8412",
4  provider:    "acme_pay",
5  error_code:  "provider_timeout",
6  duration_ms: 4200,
7  attempt:     2,
8  request_id:  "a91f3c",
9  trace_id:    "4bf92f3577b34da6",
10})
11 
12# count by provider where event="payment_failed"
13#   and error_code="provider_timeout" in the last hour
14# -> one query, correct by construction, still correct next month

Both lines contain the same facts. Only the structured one lets a program group, count and correlate them without a parser that has to be maintained against prose written by three different people.

The field set that makes logs useful

A small, consistent core carries most of the value: a timestamp, a level, the service and version, a stable event name, the correlation ids, an outcome, a duration, and the domain identifiers relevant to the operation. Everything else is situational.

The event field is the one teams most often skip and most often regret. A stable event name — payment_failed, not a message that changes when someone rewords it — is what makes a log line countable. Message text can then be as human as you like, because nothing aggregates on it.

Correlation ids deserve their own field, not embedding in the message. request_id and trace_id are what turn a pile of independent lines into one request's story across services, and what let you jump from a log line to the trace that contains it (see Correlation IDs: Turning Lines Into a Story and Where the Request Actually Went).

ILLUSTRATIVE — a core field set worth standardizing across services
timestamp      2026-08-25T14:03:11.482Z   RFC 3339, UTC, milliseconds
level          error                       see log-levels
service        checkout-api
version        7f3a91c                     enables per-deploy comparison
event          payment_failed              STABLE name -- this is what you count
message        "provider timed out"        human text -- never aggregated on
request_id     a91f3c...                   see correlation-ids
trace_id       4bf92f3577b34da6...         links this line to the trace
outcome        failure                     success | failure | degraded
error_code     provider_timeout            closed enum, same values as the metric
duration_ms    4200                        number, always milliseconds
order_id       8412                        domain identifier
provider       acme_pay

not present, deliberately:
  card number, auth token, session cookie, full request body
  -- see logs-and-secrets

Schema drift, and why logs are not free-form

Structured logging fails in a specific way: the fields drift. One service emits duration_ms, another latency_ms, a third elapsed. One emits user_id as a string, another as an integer — and some log stores will then refuse to index the field at all, or index only the first type they saw and silently drop the rest.

The failure is quiet. Queries return fewer results than they should, and there is no error to notice; a dashboard counting payment_failed events undercounts by exactly the services that spell it differently. This is why the field set is a schema even though the format is JSON, and why it belongs in a shared logging helper rather than in a style guide nobody reads.

Cost is the other constraint that shapes the schema. Every field is stored and often indexed, so wide records with dozens of situational fields multiply the bill without improving debuggability. Keep the core small and stable, add situational fields deliberately, and see The Log Bill and What It Is Buying for what to do when the volume outgrows the budget.

Symptoms of schema drift in a log storeILLUSTRATIVE
SignalValueWhat it tells youVerdict
Distinct field names for duration`duration_ms`, `latency_ms`, `elapsed`, `took`Four spellings of one concept — no aggregate covers all servicessmoking gun
Type conflicts on `user_id`string in 3 services, int in 2The store may index one type and silently drop the othersmoking gun
Events matching `event="payment_failed"`1,204/hourPlausible — but only counts services using that exact event namesuspect
Lines with no `event` field at all38% of error-level linesMore than a third of errors are uncountable without regexsmoking gun
Average fields per record47Wide records inflate storage and index cost; most fields are never queriedsuspect

Key points

  • Prose logs contain information; structured logs make it queryable — the difference only matters when you need an aggregate, which is during every incident.
  • A stable event name is what makes a log line countable; message text should stay human and never be aggregated on.
  • Correlation ids belong in dedicated fields, not embedded in the message, so logs can join to traces.
  • The field set is a schema: drifting names and types cause silent undercounting with no error anywhere.
  • Enforce the schema in a shared logging helper — style guides do not survive contact with five teams.

Follow the diagnosis

The causal chain, hop by hop — and the readings that invite the wrong conclusion.

  1. 1
    Engineer → log call: each service writes its own message string with its own wording and its own field names.
  2. 2
    Log call → store: records arrive with inconsistent field names and, worse, inconsistent types for the same field.
  3. 3
    Store → index: the store indexes the first type it sees and drops or ignores conflicting records for that field.
  4. 4
    Index → query: an aggregate over duration_ms silently covers only the services that spelled it that way.
  5. 5
    Query → responder: the responder reports a number that is confidently wrong, and nothing in the tooling flagged the omission.
What this evidence makes people conclude — wrongly
  • • "The information is in the logs" — being present in text and being queryable are different properties, and only the second helps at 3am.
  • • "JSON output means we have structured logging" — JSON with inconsistent field names is prose with extra punctuation.
  • • "The query returned results, so it worked" — silent undercounting from drift returns results too; check the service breakdown.
  • • "More fields is better" — every field costs storage and index, and fields nobody queries are pure cost.

Measure, fix, validate

An optimization is not finished until the metric that motivated it has moved.

How to measure it
  • • Count error-level lines with no `event` field; that fraction is the share of failures you cannot aggregate.
  • • List distinct field names across services and look for synonyms and type conflicts on the same concept.
  • • Time a real incident question ("how many X by Y in the last hour") — if it needs a regex, the logs are not structured enough.
  • • Track average fields per record and index size to know what the schema actually costs.
What actually fixes it
  • • Emit records with a stable `event` name plus a small, standard core field set from a shared logging helper.
  • • Normalize units and types in the helper (`duration_ms` always integer milliseconds) so drift cannot originate in service code.
  • • Include `request_id` and `trace_id` on every line so logs join to traces (see [[correlation-ids]]).
  • • Deprecate synonym fields with a dual-emit period, the same way a metric rename is handled in [[metric-labels]].
How you know it worked
  • • Re-run the incident question and confirm one query answers it, with a per-service breakdown proving no service is missing.
  • • Check that the count of error lines lacking an `event` field has dropped to near zero.
  • • Confirm a log line can be pivoted to its trace by `trace_id` in one step.
What it costs
  • • Structured records are harder to read raw in a terminal; local development usually needs a pretty-printer.
  • • Every indexed field costs storage and ingest CPU, so wide records are a real and recurring bill.
  • • A shared logging helper is another cross-service dependency to version and roll out.
Stop it coming back
  • • CI check that fails when a log call passes an unknown field name or a known field with the wrong type.
  • • Alert on the rate of records missing required core fields, so drift is caught as it is introduced.
  • • Keep a short documented field dictionary and require additions to go through it.

Accuracy

Performance numbers are conditional. These are the conditions.

What these numbers depend on
  • ILLUSTRATIVEThe field set, event names and drift percentages are teaching examples. Real field dictionaries are larger and shaped by the log store's indexing model.
  • ENVIRONMENT-SPECIFICHow a log store handles type conflicts — reject the record, drop the field, index the first type seen — varies by product and determines how silent the drift failure is.

Misconceptions

Claim
“Structured logging is just logging in JSON.”
Reality
The format is the easy half. The value comes from stable field names with stable types and meanings across services; JSON with drifting fields is no more queryable than prose.
Claim
“Humans read logs, so optimize for human reading.”
Reality
Humans read logs one line at a time during development and aggregate them during incidents. Keep a human message field for the first case and structured fields for the second — you do not have to choose.
Claim
“We can add structure later by parsing.”
Reality
Parsing recovers structure from formats that were never specified and change without notice. It works until someone rewords a message, at which point the parser silently stops matching and the count silently drops.
Log Levels Are a Convention, Not a Standard →