Sampling Without Throwing Away the Evidence
At 10,000 requests a second, tracing everything is a second production system. Sampling is inevitable; the question is whether you keep a random 1% — which discards almost every slow and failed request — or keep the ones that matter.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The arithmetic that forces the decision
Work it through with stated assumptions. Ten thousand requests per second, thirty spans per request, roughly 500 bytes per span after compression: that is 150 MB/s, or about 13 TB per day, of telemetry describing a system that probably serves less user data than that. The cost is not only storage — it is network egress from every pod, CPU spent serializing, and a backend that must ingest and index it all. This is why sampling is not a compromise imposed by cheap management; it is the normal operating condition of tracing at scale.
The naive response is uniform head sampling: decide at the entry point, keep 1%, propagate the decision so the whole trace is consistent. It is simple, cheap, and requires no buffering. It is also, for debugging purposes, close to useless — because the requests you need are exactly the rare ones. If 0.5% of requests error, uniform 1% sampling retains 1% of those errors: five per hundred thousand. The customer with a trace id in their support ticket will find nothing.
The alternative is deciding *after* seeing the outcome. Tail sampling buffers a trace until it completes, then applies rules: keep everything that errored, everything slower than a threshold, everything from a tenant under investigation, plus a small uniform baseline so you retain a picture of normal. The cost is a buffering component with memory proportional to in-flight traces, and a delay before traces become queryable.
| Strategy | Decision point | Keeps slow/errored traces? | Cost |
|---|---|---|---|
| None (100%) | n/a | All of them | ~13 TB/day at the assumptions above — usually prohibitive |
| Uniform head, 1% | At the entry point, before anything happens | 1% of them — the wrong 99% survives | Lowest; no buffering, no extra component |
| Rate-limited head | Entry point, N traces/sec per endpoint | Still blind to outcome, but protects low-traffic endpoints | Low; keeps rare endpoints visible |
| Tail sampling | After the trace completes | All errors, all slow traces, by rule | Buffering component, memory ∝ in-flight traces, ingest delay |
| Head + tail hybrid | Coarse head filter, then tail rules | Nearly all that matter | Most complex; the usual answer at scale |
Consistency, and why the decision must propagate
Whatever the strategy, the decision must be consistent across the whole trace. If each service samples independently at 1%, the probability of retaining a complete eight-service trace is one in a hundred million; what you get instead is a large pile of trace fragments, each showing one service in isolation — the worst outcome, since you pay for storage and cannot answer a single "where did the time go" question.
This is why the sampling flag rides in the propagated context alongside the trace id (Carrying the Trace Across the Gap). The entry point decides, everyone downstream obeys. It also means a service that regenerates context rather than continuing it silently re-rolls the dice, producing exactly the fragmentation described above — one more reason propagation health deserves its own metric.
Tail sampling complicates this because the decision comes late: services must export spans optimistically and let the collector decide what to keep. That works, at the cost of shipping spans you will discard — the network cost of tail sampling is closer to unsampled than people expect, and only the storage cost drops. Worth knowing before promising a cost reduction to whoever asked for one.
What sampling costs you analytically
Sampled traces cannot produce trustworthy totals, and this is the mistake that outlives every sampling migration. "How many requests errored" answered from a tail-sampled store is meaningless: errors are over-retained by design, so the ratio in your trace store bears no relationship to the ratio in production. Any number that must be complete — error rate, request count, SLI compliance, anything you bill or alert on — has to come from metrics, which are aggregated before sampling and therefore count everything (SLIs: Measuring What the User Actually Feels).
Biased retention also skews the shapes you see. If you keep all traces over 1 second and a uniform 0.1% otherwise, browsing "recent traces" shows a system that looks catastrophically slow. That is not a bug, but it fools people daily, and it argues for keeping the baseline sample large enough to see normal alongside the exceptional.
The practical target: retain 100% of errors, 100% above a latency threshold set near your SLO boundary, a configurable slice per tenant for investigations, and enough uniform baseline to characterize normal — then set retention windows by class, since a slow trace matters for weeks and a baseline trace matters for hours.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| retention rate, errored traces | 100% | Every failure is debuggable from a support ticket | normal |
| retention rate, traces > 1 s | 100% | The tail is fully preserved | normal |
| retention rate, baseline | 0.2% | Enough to characterize normal for comparison | normal |
| collector buffer memory | 78% of limit | In-flight trace buffering near capacity — evictions imminent | suspect |
| traces dropped at collector | 4.1% | Buffer evictions silently losing traces before the rules run | smoking gun |
| ingest delay p99 | 38 s | Traces queryable well after an incident starts | suspect |
Key points
- Tracing everything is a second production system: 10k rps × 30 spans × 500 B is roughly 13 TB/day, which forces a sampling decision.
- Uniform head sampling keeps the wrong requests — at 1% it retains 1% of your errors, which is none of the ones a customer will ask about.
- The sampling decision must propagate, or independent per-service decisions shred every trace into unusable fragments.
- Tail sampling keeps errors and slow traces by deciding after completion, at the cost of a buffering collector, ingest delay, and network cost close to unsampled.
- Sampled traces cannot produce totals: error rate, request count and SLI compliance must come from metrics, which count everything.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Cost review → team: tracing spend flagged as unsustainable, so the sample rate is dropped to a uniform 1%.
- 2Sampling → retention: 1% uniform retains ~1% of the 0.5% of requests that error, i.e. five in a hundred thousand.
- 3Incident → engineer: the customer supplies a trace id from a failed checkout and the backend returns nothing.
- 4Team → conclusion: tracing is judged unhelpful, when the design discarded exactly the evidence it was bought for.
- • "We sample 1%, so we see 1% of problems." You see 1% of *requests*. For rare events, that is close to 0% of problems.
- • "Tail sampling will cut our tracing bill." It cuts storage. Services still emit and ship every span, so network and serialization costs barely move.
- • "Our error rate from traces is 4%." Not if errors are preferentially retained. Read error rate from metrics only.
- • "The traces look terrible — everything is slow." Biased retention makes a healthy system look sick when browsing recent traces. Compare against the baseline slice.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Report retention rate separately for errored, slow and baseline traces — the overall sampling percentage hides everything that matters.
- • Track collector buffer utilization and evicted traces; evictions drop traces *before* the keep rules apply, defeating the whole design.
- • Measure ingest delay p99, since a trace that becomes queryable 40 s late is not available when an incident starts.
- • Estimate cost as spans/sec × bytes/span and compare against the storage bill; a large divergence means attribute bloat ([[trace-anatomy]]).
- • Adopt outcome-based retention: keep all errors, all traces above a latency threshold near the SLO boundary, plus a small uniform baseline.
- • Make the sampling decision at the entry point and propagate it, so traces are never fragmented across services.
- • Set retention windows per class — slow and errored traces for weeks, baseline traces for hours — instead of one blanket window.
- • Reduce bytes per span before reducing trace count: dropping request-body attributes is often a larger saving than any sampling change ([[trace-anatomy]]).
- • Take a known-failed request id from a support ticket and confirm its trace is retrievable — the only test that matters.
- • Confirm dropped-at-collector count is near zero after sizing the buffer, otherwise the rules never ran.
- • Compare storage spend before and after against retention rates by class, so the saving is attributed to the right change.
- • Verify error rate read from metrics still matches its historical value; if it moved, someone is computing it from traces.
- • Tail sampling adds a stateful component to the telemetry path, with its own memory limits, failure modes and operational burden.
- • Ingest delay means traces are not available at the very start of an incident, when they would be most useful.
- • Aggressive retention rules bias the visible population, so browsing traces gives a distorted impression of system health.
- • Keeping all errors is expensive precisely during an incident, when error volume spikes — the pipeline must survive its own busiest hour.
- • Alert on collector buffer utilization and evicted traces — silent eviction is the failure mode that makes tail sampling look broken.
- • Alert on retention rate for errored traces dropping below 100%, which catches rule and configuration regressions.
- • Document, in the runbook, that totals come from metrics — the misuse recurs with every new team member.
- • Review bytes-per-span quarterly; attribute bloat creeps back and quietly re-inflates the bill.
Accuracy
Performance numbers are conditional. These are the conditions.
- ESTIMATEDThe ~13 TB/day figure is derived from stated assumptions (10k rps, 30 spans/request, 500 B/span compressed). Change any assumption and it moves by an order of magnitude.
- ENVIRONMENT-SPECIFICWhether tail sampling is practical depends on your collector topology: it requires all spans of a trace to reach the same buffering instance, which constrains load balancing.