Carrying the Trace Across the Gap
Trace context travels in-band with the work: a header on the HTTP call, a field on the queue message, an argument to the job. Every hop that forgets to carry it cuts the trace in half — and the caller looks like it was idle for 400 ms.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
In-band context, out-of-band spans
Two things travel, by different routes, and confusing them causes most propagation bugs. Spans go out-of-band: each service ships its own spans to a collector on its own schedule. Context — the trace id, the current span id, sampling decision and any baggage — travels in-band, riding along with the actual request. For HTTP that is the traceparent header; for a queue it is a message attribute; for an in-process async task it is whatever the runtime uses to carry ambient state.
This split is why tracing works at all across services that never talk to each other, and it is also why a single missing header is unrecoverable. If the callee never learns the trace id, no amount of clever backend processing can reunite the two halves — timestamps and service names are not enough to distinguish your request from the four hundred others in the same second. The trace is not "degraded"; it is two unrelated traces forever.
The sampling flag riding in the same header is the subtle part: the decision to record a trace is usually made at the entry point and propagated, so every downstream service records consistently. A hop that regenerates context instead of continuing it re-rolls that decision, producing traces that are complete in some services and missing in others — which looks like a flaky backend and is actually a propagation bug.
Where it breaks, in order of frequency
The ranking is consistent across organizations. Queues break first, because a message body is a domain object and nobody thinks of it as a transport — the producer serializes an order, the consumer deserializes an order, and the context was never part of the schema. Custom HTTP clients break second: a hand-rolled fetch wrapper or a client constructed before instrumentation was installed simply never injects the header. Thread and process boundaries break third, when work is handed to a pool and the ambient context does not follow it. Third-party callbacks and webhooks break last and permanently — you cannot make someone else's server carry your header.
Each break has the same signature and a distinct fix. The signature is a caller span with a large unexplained duration and an orphan root elsewhere. The fix for queues is putting context in the message envelope explicitly, which also gives you queue wait time for free: consumer start minus producer end is exactly how long the message sat, the honest version of Depth Is Not an Emergency; Age Is.
A note on jobs specifically: propagating context to a job is not the same as making the job a child span. Carry the context so the two are *navigable*; use a link so the tree stays honest, per Parents, Children and Links. Teams that conflate these end up either with no connection at all or with 40-second root spans.
1// producer — inside the request's span context2const carrier: Record<string, string> = {}3propagation.inject(context.active(), carrier) // writes traceparent (+ tracestate)4 5await queue.send({6 body: order, // the domain payload7 attributes: { ...carrier, enqueued_at: Date.now() },8})9 10// consumer — start a NEW trace, linked to the producer11const parentCtx = propagation.extract(context.active(), msg.attributes)12const link = trace.getSpanContext(parentCtx)13 14tracer.startActiveSpan('receipt-worker process', {15 kind: SpanKind.CONSUMER,16 links: link ? [{ context: link }] : [], // link, not parent17 attributes: {18 // consumer start minus enqueue time = how long it waited in the queue19 'messaging.queue_wait_ms': Date.now() - msg.attributes.enqueued_at,20 },21}, async (span) => { /* ... */ })Measuring propagation health
Propagation is infrastructure, and like all infrastructure it needs a metric rather than a vibe. The best single number is orphan root spans per service: spans of kind SERVER or CONSUMER with no parent and no link, in a service that is never an entry point. If receipt-worker produces 8,000 orphan roots an hour, propagation from the queue is broken, and you know it without anyone noticing during an incident.
The second number is trace completeness: for a sample of entry-point traces, how many distinct services appear, compared to the number you expect from the service map. A sudden drop after a deploy is a regression in a client library or a framework upgrade that replaced an instrumented HTTP client with a bare one.
Both numbers belong on the observability team's own dashboard, and both should alert. The failure mode of tracing is silent: nothing errors, dashboards still render, and you only discover the gap at 3 a.m. when the trace you needed stops at a boundary. Treating propagation as a monitored dependency rather than a one-time setup task is the difference between tracing that works during incidents and tracing that works during demos.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| orphan_root_spans{service="receipt-worker"} | 7,900/h | Nearly every job starts a fresh, unconnected trace | smoking gun |
| orphan_root_spans{service="checkout-api"} | 12/h | Expected — a few health checks and direct probes | normal |
| trace_services_per_trace p50 | 3 (was 6) | Traces are terminating early, halved since the deploy | smoking gun |
| span_export_errors | 0 | Spans reach the collector fine — this is not an export problem | normal |
| checkout-api self time p99 | 460 ms | Large unexplained gap: the work is happening, it is just untraced | suspect |
Key points
- Context travels in-band with the request (
traceparent, message attributes); spans travel out-of-band to a collector — a missing header is unrecoverable at the backend. - Queues break propagation most often, because the message body is treated as a domain object and the context was never part of its schema.
- A broken hop shows up as a caller span with large unexplained self time plus orphan root spans in the callee — one bug, two symptoms, in different places.
- The sampling decision rides in the same context, so a hop that regenerates rather than continues it produces traces that are complete in some services and absent in others.
- Propagating context to a job and parenting the job are different decisions: carry the context, but use a link so the tree stays honest.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Deploy → HTTP client: a library upgrade replaces the instrumented client with a bare one, so
traceparentstops being injected. - 2Caller → callee: the callee receives no context and starts a fresh trace as an orphan root.
- 3Trace backend → engineer: the caller shows a 460 ms childless span; the callee's work exists but in an unrelated trace.
- 4Engineer → wrong conclusion: "checkout-api got slow", followed by a profiling session on a service that was blocked on the network the entire time.
- • "The trace ends here, so the request ended here." The request continued; only the context did not.
- • "Spans are missing, so the collector is dropping them." Check export errors first — propagation failures and export failures look similar on a trace list and have nothing in common.
- • "This service is slow — look at that 460 ms span." A childless span in a service that makes network calls is a missing-instrumentation signature until proven otherwise.
- • "We propagate on HTTP, so we are done." HTTP is the hop everyone gets right. Queues, thread pools and scheduled jobs are where traces actually die.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Count orphan root spans per service per hour, excluding genuine entry points — the single best propagation health metric.
- • Track distinct services per trace at p50 for a known entry point and alert on drops after deploys.
- • Measure self time on suspiciously fat spans: a 460 ms span with no children in a service that obviously calls out is a propagation gap, not slow code.
- • For queues, record `enqueued_at` in the envelope so consumer-start minus enqueue gives real queue wait ([[queue-age]]).
- • Put trace context in the message envelope as a first-class part of the schema, injected on produce and extracted on consume.
- • Centralize outbound HTTP through one instrumented client and ban bare clients in review — most propagation regressions are a new client, not new code.
- • Explicitly carry context across thread-pool and process boundaries in runtimes where ambient context does not follow the work.
- • For inbound third-party callbacks you cannot control, accept the break and correlate on a business id you supplied (order id, idempotency key) instead of pretending it is one trace.
- • Orphan roots for the affected service should fall to the level explained by genuine entry points, within one deploy cycle.
- • Distinct services per trace should return to the expected count for a sampled entry-point trace.
- • The formerly childless 460 ms span should now decompose into the calls it was making.
- • Queue wait time should start reporting non-zero, plausible values — it is derived from the same envelope you just fixed.
- • Envelope fields add a small amount to every message and require both producer and consumer to agree on the schema.
- • Centralized HTTP clients constrain teams that want per-call configuration; the propagation win usually justifies it, but say so out loud.
- • Explicit context passing is verbose in runtimes where implicit propagation is idiomatic, and reviewers will push back.
- • Correlating third-party callbacks on business ids is weaker than a real trace and needs its own query conventions.
- • Alert on orphan root spans per service; it is the canary for every future propagation break.
- • Contract-test propagation: an integration test that sends a request through the queue and asserts the consumer span carries a link to the producer.
- • Add trace-context fields to the message schema definition so a new producer cannot omit them silently.
- • Re-check completeness after framework and client-library upgrades — that is when this breaks.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe orphan-root and completeness numbers are invented to show the reading; absolute values depend entirely on your service topology and traffic.
- RUNTIME-SPECIFICWhether context follows work across an async boundary automatically depends on the runtime and its context mechanism; the failure modes differ substantially between them.