ObservabilityGENERALFRAMEWORK-SPECIFICRUNTIME-SPECIFICSIMPLIFIED

Tracing From the Backend's Side

What a service must emit and propagate so one request's path across processes becomes a single readable timeline.

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

A request takes three seconds and touches five services. Which one spent the time, and what do I have to emit for that question to be answerable?

The requirement

Checkout is slow for some customers. Each service reports healthy latency for its own work, the sum does not match what the user sees, and nobody can point at a hop.

The obvious build

Log a timestamp when each service starts and finishes handling the request. Line them up afterwards and the gap will be obvious.

Why it breaks

Clocks differ between hosts by enough to invert causality, so a child appears to start before its parent and the timeline is nonsense.

How it breaks in production

Clocks differ between hosts by enough to invert causality, so a child appears to start before its parent and the timeline is nonsense.
Log lines have no parent-child structure. Five services doing three calls each gives fifteen intervals with no way to know which nested inside which (Parents, Children and Links).
Concurrent calls cannot be distinguished from sequential ones, which is precisely what you need to know — three 200 ms calls in parallel and in series are the same log lines and very different systems.
The time that is *not* in any service — queueing at a proxy, waiting for a connection, sitting in a queue — is invisible, and it is frequently where the three seconds are.
Assembling the timeline is manual, so it happens once per incident and never as a routine question.

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

A span is one timed operation with a start, a duration, a name and attributes. A trace is a tree of spans sharing a trace id, linked by parent-span ids (Trace, Span, Attribute, Status).
Context propagation is what makes the tree: the current span id travels to the next hop, where it becomes the parent of a new span. Over HTTP this is the W3C traceparent header; over a queue it is a field on the message.
The backend's job in tracing is narrower than the tooling suggests. It must create spans at boundaries, propagate context outward, and attach attributes only it knows — tenant, route, query shape, cache outcome, retry attempt.
Auto-instrumentation covers the framework, HTTP client and database driver. It cannot know your domain, so the spans are structurally right and semantically thin until you add attributes.
Sampling decides which traces are stored. Head sampling decides at the first span, which is cheap and randomly discards interesting traces; tail sampling decides after the trace completes, which can keep all errors and slow traces but requires buffering (Sampling Without Throwing Away the Evidence).
A gap in a trace waterfall is information: time inside the parent span not covered by any child is either your own code or waiting for a resource that is not instrumented (Reading the Waterfall).
Traces and logs meet through ids. A log line carrying traceId and spanId lets you jump from a slow span to exactly what that code logged.

What the backend actually owns

Most tracing documentation is about the tool. The part that belongs to a backend engineer is small and non-negotiable, and it is the part auto-instrumentation cannot do: propagate context across every boundary you built by hand, and attach the attributes only your code knows.

The diagram is the whole obligation. Solid propagation across four hops, one of which is a queue, and a span tree that survives it.

UserLLMAgentToolDataDecisionHumanGuardrail

Propagating across the queue is the hop that breaks

FRAMEWORK-SPECIFICThe @opentelemetry/api surface shown is the JS one. The Python, Go and Java APIs expose the same inject/extract/startSpan concepts with different names and different context-management idioms — Go threads ctx explicitly, Java uses a try-with-resources Scope. The envelope pattern is identical everywhere.

HTTP propagation is usually handled by an instrumented client. The queue is not: the message is data you constructed, so the trace context has to be data you put in it and data you take out. Forget it and the worker's spans become the roots of unrelated traces, which looks fine on every dashboard.

One subtlety: the worker's span should be a *child* of the enqueue span, but it may start minutes later. That is legitimate — trace duration is wall-clock from root to last span, and a trace that spans a queue is long by nature. If that is unhelpful for your workload, use a span link rather than pretending the hop did not happen.

Injecting and extracting trace context around a queue

1import { context, propagation, trace, SpanKind } from '@opentelemetry/api'
2 
3// --- Producer: inject the active context into the message envelope ---
4async function enqueue(topic: string, payload: unknown) {
5  const carrier: Record<string, string> = {}
6  propagation.inject(context.active(), carrier)   // writes 'traceparent'
7 
8  await queue.send(topic, {
9    meta: { traceContext: carrier, enqueuedAt: Date.now() },
10    payload,
11  })
12}
13 
14// --- Consumer: extract it and make the job span a child ---
15queue.subscribe(topic, async (msg) => {
16  const parent = propagation.extract(context.active(), msg.meta.traceContext ?? {})
17 
18  await context.with(parent, async () => {
19    const span = tracer.startSpan(`job ${topic}`, { kind: SpanKind.CONSUMER })
20    // Attributes only this code knows:
21    span.setAttribute('messaging.queue_wait_ms', Date.now() - msg.meta.enqueuedAt)
22    span.setAttribute('messaging.attempt', msg.attempt)
23    try {
24      await context.with(trace.setSpan(context.active(), span), () => handle(msg.payload))
25      span.setStatus({ code: 1 })
26    } catch (e) {
27      span.recordException(e as Error)
28      span.setStatus({ code: 2 })     // searchable: failed traces
29      throw e
30    } finally {
31      span.end()
32    }
33  })
34})

queue_wait_ms is computed from two different machines' clocks, so treat it as indicative rather than exact. It is still the only number that makes queue wait visible at all, and its *trend* is trustworthy even when its absolute value is not.

Which signal answers which question

The three signals are not competitors and are constantly treated as though they were. Each is cheap at what it does and hopeless at the others, and the practical skill is knowing which one to open first.

Backend work usually flows in one direction: a metric tells you something is wrong, a trace tells you where, and a log tells you what the code was doing when it got there.

Question	Signal	Why the others fail	Cost model
Is anything wrong right now?	Metrics	Logs and traces are per-event; aggregating them for an alert is slow and costly	Per series, flat in request count (The Metrics a Backend Must Emit)
Which hop spent the time?	Traces	Metrics have no structure; log timestamps cross unsynchronised clocks	Per span, sampled (Reading the Waterfall)
What happened to this one request?	Logs	Metrics aggregate by construction; traces are sampled and carry no decision detail	Per event, high volume (What a Backend Should Actually Log)
Did this specific customer succeed?	Logs, by correlation id	Traces may have been sampled away; metrics cannot be filtered to one principal	Per event (Correlation Ids That Survive Every Hop)
Is the queue keeping up?	Metrics (depth + age)	A trace shows one message, not the backlog	Per series (Queue Backlog)
Why is p99 worse than p50?	Traces of slow requests + histograms	A mean tells you nothing; a single log line cannot show the distribution	Tail sampling makes this affordable (Tail Latency: Why p50 Being Fine Does Not Help)

How to build it

Most important first.

Propagate context on every outbound path, including queue messages and any custom transport. An un-propagated hop does not produce a broken trace; it produces two unrelated traces, which is worse because nothing looks wrong (Correlation Ids That Survive Every Hop).
Adopt one propagation standard across all services. Mixed formats produce silently disconnected traces.
Let auto-instrumentation create the structural spans; spend your effort on attributes: tenant.id, route, cache.hit, db.rows_returned, retry.attempt, queue.wait_ms.
Create manual spans only for meaningful units of work — a batch enrichment, a pricing computation, an external retry loop. A span per function makes traces unreadable and expensive.
Record errors on spans with the status and the error category, so a trace search for failed traces works (An Error Taxonomy That Maps Cause to Response).
Prefer tail sampling if the platform supports it, keeping all error and slow traces plus a small percentage of successes. Otherwise head-sample and accept that the trace you want may be gone.
Put traceId on every log line so the two signals cross-reference (Structured Logging).
Instrument queue wait explicitly. Enqueue time in the message envelope, compared with dequeue time, is the only way the wait becomes visible — but treat the value as approximate, because it spans two clocks.

What can go wrong

Failure modes

Context lost at an async boundary — a detached promise, a thread pool, a library that does not propagate — so downstream spans become the roots of new traces.
A missing hop in the middle: service B does not propagate, so A's trace ends at B and C's spans are orphans. The waterfall looks complete and is missing a second.
Span explosion from over-instrumentation: thousands of spans per trace makes the UI unusable and the ingest expensive.
Attribute cardinality: unlike metric labels, high-cardinality span attributes are normal and useful — but attaching a whole request body to a span is still a cost and a leak.
Head sampling at 1% meaning the one trace you need is almost certainly not stored, discovered only when you go looking for it.
Clock skew making a child span appear to start before its parent, which some UIs render as a negative-duration gap and everyone misreads as a bug in the tool.
The mitigation failing: a tracing SDK whose exporter blocks when the collector is unreachable, adding latency to every request because your observability backend is down.

What can race

Concurrent child spans under one parent are exactly what a trace is good at showing, and exactly what a shared mutable "current span" variable destroys. Current-span tracking must be request-scoped, like any other ambient context (Request Context Propagation).
A span exported before its children complete produces a partial trace. Async exporters must be flushed on shutdown or the last traces before a deploy are lost (Graceful Shutdown).

Security

Span attributes go to your tracing backend, frequently a third party. Never attach tokens, authorization headers, full request bodies or raw query parameters (Secrets in Logs).
SQL statements as span attributes are common and useful — attach the *statement*, never the bound parameter values, which are user data and often personal (Sensitive Data Classification).
An inbound traceparent from an untrusted caller is attacker-controlled: it can be used to link their traffic into your traces or to force a sampling decision. Accept from trusted peers; regenerate at the public edge.
Trace URLs are effectively internal architecture diagrams. A trace UI exposed without authentication maps your entire service topology (Public Exposure, Read With Context).

Misreads

"Tracing replaces logging." A trace shows structure and timing. It does not show the twelve fields you would have logged about the decision the code made (What a Backend Should Actually Log).
"We installed the SDK, so we have tracing." You have structural spans. Whether the trace crosses your queue boundary is a separate question and usually the answer is no.
"The trace shows service B is slow." It shows B's span was long. That may be B waiting on C, on a lock, or on a connection — read the children before assigning blame.
"Trace id and correlation id are the same." They overlap and differ in the one property that matters during a support ticket: trace ids are sampled away and correlation ids are not.
"A gap in the waterfall means the tool is broken." A gap is a measurement: time you did not instrument.

Operating it

How you see it in production

Check trace completeness: what fraction of traces contain spans from every service you expect on that path? A number below one names a propagation gap.
Count orphan spans — spans whose parent id is not present in the trace. A rise means someone broke propagation in a deploy.
Look at the unexplained gap inside your own root span. Consistent unexplained time is either uninstrumented code or resource waiting, and both are worth a manual span.
When a trace shows a fast service and a slow caller, measure at both ends of the hop. The difference is network plus queueing, which is not in either service's own numbers (Queueing: Why Systems Get Slow Before They Get Broken).

What changes at 10x and 100x

At 10x, sampling stops being optional. The decision is which traces to keep, and the answer is almost always errors and slow ones.
At 100x, the collector tier becomes infrastructure with its own capacity, failure modes and cost. Your service must degrade gracefully when it is unavailable — never block a request on an exporter.
With many services, the propagation contract must be enforced by a shared library or a service mesh. Per-team implementations diverge and the divergence is invisible until a trace is missing a hop.
Trace storage grows with span count, not request count. Over-instrumentation is the cost driver, and it is much easier to add spans than to remove them.

What this costs

Tracing has per-request overhead — context propagation, span creation, attribute serialization and export. Small per request, real at high rates, and paid on every request including the fast ones.
Sampling means the trace you want may not exist. Tail sampling fixes this and needs buffering infrastructure that head sampling does not.
Auto-instrumentation is fast to adopt and produces generic spans; meaningful attributes are manual work that nobody schedules.
A tracing backend is another production dependency with its own outages, and it usually goes down at the same time as everything else.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

GENERALSpans, traces and context propagation are vendor-neutral concepts and predate every current tool.
FRAMEWORK-SPECIFICOpenTelemetry is used as the example: it defines the API, the SDK, the W3C propagators and the OTLP wire format, and most backends now ingest it. Vendor SDKs that predate it use their own propagation headers and are not interoperable without a bridge — mixing them is the usual cause of a trace that stops halfway.
RUNTIME-SPECIFICContext propagation piggybacks on the runtime's ambient-context mechanism, so it inherits its gaps: AsyncLocalStorage on Node loses context across some native callbacks and worker threads; Python contextvars do not cross into a ThreadPoolExecutor without copying; Go passes ctx explicitly and never loses it silently but requires every signature to carry it.
SIMPLIFIEDSpan links, baggage, exemplars and metric-trace correlation are omitted here. The depth on trace anatomy, sampling strategy and waterfall reading lives in Observability & Performance (Distributed Tracing).

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

Architecturedistributed-tracing logs-metrics-traces

Performancedistributed-tracing trace-anatomy span-relationships trace-sampling trace-waterfall context-propagation opentelemetry critical-path n-plus-one-traces fan-out-latency

Cloudinfrastructure-observability

Domains that do not exist yet

— Distributed Systems — clock skew, causal ordering and why a distributed timeline assembled from local clocks is an approximation rather than a measurement.