SherlockLiu Logo SherlockLiu
Back to all posts
Engineering

Practical OpenTelemetry — Part 10: Sampling and Deployment Models

SL
• Aug 22, 2026 • 12 min read
Practical OpenTelemetry — Part 10: Sampling and Deployment Models

Chapter 10 of Practical OpenTelemetry answers the question every adoption eventually hits once instrumentation is actually flowing: how do we afford this at scale? Two threads run through it — trace sampling (three strategies, each with a real, distinct tradeoff) and the four places a Collector can physically live in your infrastructure.

Trace Sampling: Not All Traces Are Equal

Sampling exists because storing every trace is both unaffordable and pointless — Part 1’s “more than 95% of full-volume telemetry would be of no debugging interest” argument, applied concretely. But how you sample determines whether you keep the 5% that matters.

The book distinguishes two families:

  • Probability sampling — every element has a pre-established chance of selection. Simple, predictable, statistically honest.
  • Non-probability sampling (purposive) — biased toward elements with interesting characteristics: errors, slow requests. Better debugging value, but you can’t know the true probability beforehand, which breaks any downstream statistical math built on top of it.

Head-Based Sampling: Decide at Creation

Head-based sampling decides in-process, at span creation — cheap, and it reduces the tracer’s own overhead. Two API concepts interact here:

  • IsRecording — the span is passed to processors, but may still not be exported (useful for span-to-metrics pipelines that need the span object without paying export cost).
  • Sampled — the propagated flag. A span can be recording but not sampled.

Built-in samplers: AlwaysOn, AlwaysOff, TraceIdRatioBased (deterministic — the same trace ID always lands on the same side of the ratio), and ParentBased — the default and the recommendation. ParentBased replicates the parent’s decision, falling back to a root sampler only for genuinely new traces; this prevents orphaned spans — child spans sampled while their parent wasn’t, which break the causal tree a reader is trying to follow.

Consistent Probability Sampling: The p and r Values

Plain head-based sampling has a hidden flaw: a parent’s decision distorts a child service’s effective sampling probability, which breaks adjusted counts — the inverse-probability math used to estimate a trace’s true population size from a sample. The fix, in the experimental ConsistentProbabilityBased sampler plus W3C tracestate:

  • p-value (0–63) — the negative log2 of the sampling probability (1 = 50%, 2 = 25%, 3 = 12.5%…); powers of two only.
  • r-value (0–63) — generated once at the trace root (commonly, leading zeros of the trace ID), propagated unmodified through the whole trace.
  • Decision: sample if p ≤ r.

The same r-value travels with the entire trace, so services configured with different p-values still produce consistent, possibly partial traces — service A sampling at 12.5% and service B at 50% means every span B keeps is a strict superset of what A keeps. On the wire: tracestate: ot=r:3;p:2.

Collector-Side Probability Sampling

The probabilistic_sampler processor hashes the trace ID with a configurable hash_seed and sampling_percentage. Two caveats: the hash_seed must be identical across every collector that could see spans from the same trace (the constraint that matters most in gateway deployments), and it honors the legacy OpenTracing sampling.priority attribute for migrations. It’s the simplest scheme — no state held, no routing required — which is exactly when to reach for it.

Tail-Based Sampling: Decide at the End

Here’s the counterintuitive one: probability sampling is random, but errors and slow requests are precisely what you want to keep — and “good” traces are what you want to drop. Tail-based sampling decides after the whole trace completes, based on its actual characteristics.

The costs:

  • The tail_sampling processor must hold all spans of in-flight traces in memory for a decision_wait window.
  • All spans of a trace must reach the same Collector instance — solved either by a single non-scaling replica, or two-tier routing by trace ID via the loadbalancing exporter.
  • Memory must be bounded: how many traces in flight, max trace duration.

The payoff: telemetry that concentrates on the transactions that actually matter for debugging. The book’s summary line — “sometimes, less is more” — is the whole thesis in four words.

Diagram comparing three trace sampling strategies — head-based, consistent probability, and tail-based — with their tradeoffs, forming an adoption ladder from simplest to most advanced. Three Sampling Strategies Head-Based decide at span creation in-process, cheap ParentBased: mirror parent + root ratio sampler default · recommended start Consistent Probability sample if p ≤ r r from trace root, propagated tracestate: ot=r:3;p:2 accurate adjusted counts experimental · partial traces OK Tail-Based decide after trace completes keep errors + slow, drop good needs same-collector routing biases spans → use metrics for KPIs advanced · gateway only Which to choose? Start here ParentBased + ratio sampler zero infra, complete traces Upgrade when you need accurate population estimates from sampled traces Adopt when storage cost > routing complexity and you run a gateway Whatever you sample, remember: metrics for aggregates, traces for debugging.

Figure: the three sampling strategies form an adoption ladder — head-based first, consistent when you need math, tail-based when you need discrimination.

Four Deployment Models

Chapter 10 also covers where the Collector — and your SDKs — physically live:

Model Shape Pros Cons
Collector-less SDK exports straight to backend Simplest; nothing extra to run No central config; app bears aggregation cost; exporter saturation hurts the app
Node agent Collector daemonset per node Replaces node-exporter/kube-state-metrics-style tooling (hostmetrics, kubeletstats, k8scluster receivers); smaller blast radius Apps still manage their own export config
Sidecar agent Collector injected per pod Pipelines decoupled from apps early; enrichment config independent of app lifecycle Pod recreation to change pipelines; real resource cost at scale (the book’s own figure: ~100m CPU / 128MB per pod); no global view
Gateway Central funnel cluster Global sampling and policies; central “last hop” shields backend migrations from every team Operational complexity; needs a TargetAllocator for pull-based scraping at scale

The gateway is where tail-based sampling actually lives in production — it’s the only topology where “all spans of a trace on one instance” is tractable to guarantee. The book’s tip is worth repeating verbatim: provide a common organizational telemetry endpoint resolving to the best gateway, so teams never configure backend URLs directly and a future migration touches one DNS record instead of every service’s config.

Where This Leaves the Book

Sampling and deployment topology are the last technical decisions in the book. Everything past this point — Part 11 and Part 12 — is about people: how you actually get an organization to adopt any of the last nine posts’ worth of tooling without a mutiny, and how you keep the resulting telemetry worth trusting once it’s live. The book’s own transition is a useful one to sit with: the hardest part of an observability rollout was never the sampler math.

Next: Practical OpenTelemetry — Part 11: Minimizing Adoption Friction — the enablement framework, migration shims, and why automatic instrumentation should always win by default.

References

Comments

Related Blogs