What is tail-based sampling in distributed tracing and when should you use it?

Tail-based sampling is a tracing strategy where the decision about whether to keep or discard a trace is made after the entire trace is complete, not at the moment the root span starts. This contrasts with head-based sampling, where a random coin flip at the entry point determines whether the trace is recorded — before you know if anything interesting will happen.

The problem with head-based sampling is that it discards traces randomly, including most of the interesting ones. If 1% of requests produce errors, and you sample 10% of all traces, you will keep only ~0.1% of your error traces. The errors — the cases you most need to debug — are systematically underrepresented.

Tail-based sampling solves this by buffering spans in a collector (like the OpenTelemetry Collector's tail sampling processor) until the trace is complete. Then the sampling policy is evaluated: keep all traces that contain an error, keep all traces with p99 latency exceeded, keep 1% of the healthy fast traces. This ensures errors and slow traces are always captured at 100%, while routine traffic is sampled down.

The trade-off is infrastructure complexity: the collector must hold spans in memory long enough for late-arriving spans to complete the trace (typically 10–30 seconds), requiring significant RAM and careful timeout tuning. If the collector crashes mid-window, partial traces are lost.

Use tail-based sampling in production microservices where error rates are low (less than 5%) and capturing all error traces is a hard requirement for debugging.