Metrics, Logs, Traces, Profiles
Four signals, four different questions. Metrics tell you something changed; traces tell you where the time went; logs tell you what exactly happened; profiles tell you what the CPU was doing. No single one explains an incident, and knowing which to reach for first is most of the speed.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Four instruments, four questions
The signals are not competing products; they are instruments with different resolutions. Metrics are pre-aggregated numbers over time: cheap, always-on, and excellent for "did something change and when". They are blind to individual requests, and blind along any dimension you did not declare up front.
Traces follow one request across process boundaries, breaking it into spans with timings and attributes. They answer "where did the time go" — the single most valuable question in a distributed system — and they carry high-cardinality context cheaply because they are sampled. What they cannot do is tell you what happened at 14:03:20 in a request that was not sampled.
Logs are discrete events with arbitrary context: the instrument of last resort and first detail. They are the only signal that reliably captures "this specific unusual thing happened", and they are the most expensive per byte of insight. Profiles are the odd one out: they measure cost *inside* one process — CPU samples, allocations, lock waits — and are the only way to attribute time to a function rather than a component (see When the Trace Runs Out of Answers).
| Signal | Best at | Blind to | Cost driver | Typical retention |
|---|---|---|---|---|
| Metrics | Trends, rates, alerting, "did it change and when" | Individual requests; any dimension not pre-declared | Label cardinality (series count) | Months to years |
| Traces | "Where did this request spend time" across services | Unsampled requests; anything outside the instrumented path | Span volume × sampling rate | Days to weeks |
| Logs | Exact detail of one event; forensic reconstruction | Aggregates; anything not explicitly logged | Bytes ingested | Days to weeks |
| Profiles | CPU/allocation cost per function inside a process | Anything across processes; time spent waiting (unless wall-clock profiling) | Sample rate × process count | Days to a month |
| Events | Deploys, config flips, scaling actions, feature flags | Continuous behavior | Negligible | Months |
One incident, four views
The clearest way to internalize the split is to watch one incident through each instrument. Checkout latency jumps at 14:03. Each signal contributes something the others cannot, and the investigation is fastest when they are used in the order that narrows fastest: metrics to confirm and bound, traces to localize, logs or profiles to explain the mechanism.
Notice what happens when a signal is missing. Without traces, you know checkout is slow but not which of six dependencies is responsible — you are reduced to correlating dashboards, which is guesswork with charts. Without metrics, you have no baseline and no alert, so you learn about the incident from a customer. Without logs, you know the payment span is slow but not that the provider returned a specific pending status your retry logic mishandled.
The deploy event is the cheapest signal here and frequently the most valuable: it converts "latency rose around 14:00" into "latency rose three minutes after v2.4 shipped" (see "What Changed?" — Deploy Markers and the Invisible Deploys). It is also the one most often missing from dashboards.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Metric · checkout p99 | 240ms → 1,820ms at 14:03 | Confirms the symptom, bounds the start time, shows it is the tail not the mean | suspect |
| Metric · request rate | 1,180 rps → 1,205 rps | Traffic is flat, so this is not a load-driven slowdown | normal |
| Event · deploy marker | v2.4 at 14:00:12 | A change three minutes before the symptom — a strong lead, not yet a cause | suspect |
| Trace · span breakdown | payment span 90ms → 1,610ms; db span 38ms | Localizes the time to one dependency and exonerates the database | smoking gun |
| Log · payment client | 3.1x outbound calls, many `status=pending` retries | Explains the mechanism: the new retry path re-sends on pending | smoking gun |
| Profile · CPU | flat, 22% utilization, no new hot frames | Rules out an in-process CPU regression entirely | normal |
They are strongest when correlated
The multiplier is not having four signals; it is being able to move between them for the same request. A latency spike on a metric should lead to exemplar traces from that bucket. A slow span should link to the logs emitted inside it. A log line should carry the trace id that produced it. Without those links, each instrument is a separate investigation and the engineer is the join key.
That linkage is what a correlation id buys (see Correlation IDs: Turning Lines Into a Story), and it is why context propagation is worth the effort even in modest systems (see Carrying the Trace Across the Gap). It also reframes the cost question: rather than logging everything at INFO, log less but attach the trace id, so the cheap signal points at the expensive one only when needed.
A pragmatic default for a service that has none of this: start with metrics for the golden signals, add trace context propagation, make logs structured with the trace id, and turn on low-rate continuous profiling. That is a small amount of work that makes each subsequent investigation dramatically shorter — and it is a starting posture to revise, not a mandate.
1// One id, emitted into every signal, so any of them can lead to the others.2async function handleCheckout(req: Request) {3 const span = tracer.startSpan('checkout')4 const ctx = { traceId: span.traceId, route: 'POST /checkout', region: REGION }5 6 try {7 // metric: low-cardinality labels only — route and status, never user id8 return await withTimer(checkoutDuration, { route: ctx.route }, async () => {9 const result = await charge(req.body, ctx) // ctx propagates into the span tree10 // log: structured, carries the trace id so the log can be found FROM the trace11 log.info({ event: 'checkout.ok', ...ctx, amount: req.body.amount })12 return result13 })14 } catch (err) {15 span.setStatus('error')16 // the error log and the failed span are now the same investigation17 log.error({ event: 'checkout.failed', ...ctx, error_code: codeOf(err) })18 throw err19 } finally {20 span.end()21 }22}Key points
- Metrics answer "did it change and when", traces answer "where did the time go", logs answer "what exactly happened", profiles answer "which function cost what".
- Each signal is blind in a specific way; an investigation with only one instrument is guesswork with charts.
- Cost drivers differ: metrics scale with label cardinality, logs with bytes, traces with span volume × sampling rate.
- The multiplier is correlation — a trace id in every log line and exemplars linking metrics to traces.
- Deploy and config events are the cheapest signal and frequently the fastest route to a lead.
Progressive depth
Overview
Four instruments: metrics for trends, traces for request paths, logs for exact events, profiles for in-process cost. Ask which question you have, then pick.
Practical
Confirm and bound with metrics, localize with traces, explain with logs or profiles. Keep a trace id in every log line so you can move between them for the same request.
Advanced
Cost shapes usage: metric cardinality explodes multiplicatively, so high-dimensional context belongs on spans. Head sampling is cheap but drops the tail; tail-based sampling keeps slow and errored traces at higher cost and complexity (see Sampling Without Throwing Away the Evidence).
Internals
All four are the same pipeline with different aggregation points: in-process buffer, batching exporter, collector, storage. Metrics aggregate at emission (cheap, lossy along undeclared dimensions); traces and logs aggregate at query time (expensive, flexible). That single difference explains nearly every cost and capability trade-off between them.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Question → instrument: an aggregate question ("which dependency is slow") is asked of logs, which cannot aggregate cheaply.
- 2Investigator → volume: grepping large log volumes returns matches without proportion, so nothing is ranked by contribution.
- 3Investigator → wrong layer: with no span breakdown, dashboards from several components are compared by eye and the busiest-looking one is blamed.
- 4Change → no effect: the blamed component was not the constraint, and the real one keeps consuming the latency budget.
- 5Team → tooling: the conclusion is "we need better logging", when the missing instrument was tracing.
- • "We have logs, so we do not need tracing." Logs record events; reconstructing a cross-service timing breakdown from them is possible in principle and painful enough in practice that it does not happen during incidents.
- • "The profiler shows the hot function, so that is the bottleneck." A CPU profile is silent about time spent waiting — the dominant cost in most request paths (see Computing or Waiting?).
- • "Metrics are flat, so nothing is wrong." Metrics are flat along the dimensions you declared; a failure isolated to one tenant or client version can be invisible in the aggregate.
- • "Sampled traces mean we might miss the problem." For latency investigation, sampled traces plus tail-based rules capture the shape well; the risk is real but far smaller than having no traces.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Classify the question before choosing the instrument: aggregate, path, event, or in-process cost.
- • For "where is the time", read a trace waterfall split by span before reading any log.
- • For "did something change", read the metric with a deploy-marker overlay over a window that includes the last known-good period.
- • Check that a log line and a span for the same request share an id — if they do not, correlation is manual and slow.
- • Propagate one trace/correlation id across every hop and stamp it into logs, so any signal leads to the others.
- • Emit exemplars from latency histograms to trace ids, so a spike on a chart is one click from the slow request.
- • Move high-cardinality context off metric labels and onto span attributes and log fields (see [[cardinality]]).
- • Add deploy, config-change and scaling events as first-class annotations on latency dashboards.
- • Set sampling deliberately per signal rather than defaulting to "log everything, trace nothing".
- • Pick a random slow request from a latency chart and try to reach its trace and its logs in under a minute; if you cannot, the correlation is not real.
- • Measure time-to-localize in the next incident: how long from symptom confirmed to the component named?
- • Confirm telemetry cost per signal against the budget after the change — correlation should not have increased log volume.
- • Four signals mean four pipelines to run, pay for and keep healthy — the operational surface is real.
- • Correlation requires propagation code in every service and every async boundary, including queues and jobs.
- • Rich span attributes and log fields increase the chance of capturing sensitive data, which needs review (see [[logs-and-secrets]]).
- • Add correlation-id propagation to the service launch checklist and to code review for any new outbound call.
- • Alert on telemetry pipeline health: dropped spans, collector queue depth, ingestion lag.
- • Review signal coverage after each incident: which instrument was missing, and what did its absence cost in minutes?
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe incident readings are a constructed teaching example. The point is the relative contribution of each instrument, not the specific latencies.
- ENVIRONMENT-SPECIFICRetention and cost figures depend heavily on vendor pricing model, self-hosted versus managed, and traffic volume.