8.1 Head-Based Sampling & Built-in Samplers

8.1 Head-Based Sampling & Built-in Samplers

Quick Answer: Head-based sampling makes the sampling and recording decision at the very start ("head") of a trace when the root span is initiated, before child operations execute, latencies are measured, or errors are thrown. The decision sets the low-order bit (0x01) of TraceFlags within the immutable SpanContext, which is propagated across microservices via the W3C traceparent header. OpenTelemetry SDKs provide three sampling result states (DROP, RECORD_ONLY, RECORD_AND_SAMPLE) and the classic built-in samplers AlwaysOn, AlwaysOff, TraceIdRatioBased, and the production-standard ParentBased wrapper.

In modern microservice ecosystems processing hundreds of thousands of transactions per second, capturing 100% of all distributed traces is economically and operationally prohibitive. Emitting, transmitting, serializing, indexing, and storing billions of spans consumes massive network egress bandwidth, exhausts collector CPU cycles, and inflates cloud storage invoices by tens of thousands of dollars each month. To maintain manageable overhead while preserving deep diagnostic observability, engineering teams must implement sampling strategies.

In the OpenTelemetry architecture, sampling falls into two distinct operational models: head-based sampling and tail-based sampling. This section delves into head-based sampling, the SDK-level decision contract, the W3C TraceFlags bitmask, and the built-in samplers specified by the OpenTelemetry standard.

---

Fundamentals of Head-Based Sampling: The Ingress Decision

Head-based sampling is defined by where and when the sampling decision occurs: at the ingress point ("head") of a distributed request, when the root span is initially created.

Incoming Request (HTTP / gRPC)
        │
        ▼
┌──────────────────────────────────────────────┐
│ Microservice Ingress: Root Span Creation     │
│ Sampler.shouldSample() is invoked            │
│                                              │
│ Decision: [RECORD_AND_SAMPLE] (low bit = 1)   │
│        or [DROP]              (low bit = 0)   │
└──────────────────────┬───────────────────────┘
                       │
        ┌──────────────┴──────────────┐
        ▼                             ▼
[TraceFlags = 0x01]           [TraceFlags = 0x00]
Span recorded & exported      Span dropped / ignored
Context injected downstream   Context injected downstream

The Operational Advantage: Maximum Ingress Efficiency

The primary advantage of head-based sampling is resource efficiency:

• Minimal CPU Overhead: The sampling algorithm evaluates a single mathematical or rule-based check once at the root span inception.
• Zero Downstream Wastage: When a trace is dropped at the head, downstream services that respect parent sampling do not waste CPU cycles serializing span JSON/Protobuf payloads or buffering spans in memory.
• Immediate Network Savings: Network bandwidth between microservices and observability collectors is conserved because unselected traces never leave application memory.

The Fundamental Limitation: Lack of Foresight

The fundamental limitation of head-based sampling is that the decision is made before the request executes:

• When a user clicks "Submit Order", the root span begins.
• At that exact microsecond, the SDK cannot know whether a downstream database connection will deadlock after 8 seconds.
• The SDK cannot know whether an inventory microservice will throw an unhandled NullPointerException.
• The SDK cannot know whether a payment gateway will return an HTTP 500 status code.

If the head sampler decides to drop the trace because a probabilistic 5% ratio was configured, that trace is gone forever. When the downstream database deadlocks 3 hops later, no trace exists to debug the incident. Overcoming this limitation requires understanding the SDK decision states and configuring the right samplers.

---

The Sampling Contract: The TraceFlags Low Bit

When an OpenTelemetry SDK starts a span, the Tracer delegates the decision to the configured Sampler implementation by invoking its shouldSample() method. The result of this decision is immutably stored in the span's SpanContext.

The Anatomy of TraceFlags

A SpanContext contains four core members:

1. TraceId (16 bytes / 32 hexadecimal characters)
2. SpanId (8 bytes / 16 hexadecimal characters)
3. TraceFlags (1 byte / 8-bit bitmap)
4. TraceState (list of key-value pairs for vendor routing)

The TraceFlags byte is the standard cross-process vehicle for communicating sampling decisions. The sampling decision is the least significant bit (mask 0x01); W3C Trace Context Level 2 also defines the random flag (0x02), which newer probability samplers use:

TraceFlags Byte: [ 0 | 0 | 0 | 0 | 0 | 0 | 0 | S ]
                                               │
                                               └── S = Sampled Bit (Bit 0)
                                                   0 = Not Sampled (0x00)
                                                   1 = Sampled     (0x01)

• 01 (Sampled): Indicates that the trace was selected for recording and export. Downstream services extracting this context recognize that the parent was sampled and should record child spans accordingly.
• 00 (Not Sampled): Indicates that the trace was not selected for export. Downstream services extracting this context recognize that the parent was not sampled and drop child spans.

Wire Serialization in W3C traceparent

When context is propagated across network boundaries (such as HTTP headers), TraceFlags is serialized as the final 2-character hexadecimal field of the traceparent header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             │  │                                │                │
          Version  TraceId (16 bytes)              SpanId (8 bytes)  TraceFlags (01 = Sampled)

If the low bit is 00 (traceparent: 00-4bf9...-00f0...-00), the downstream recipient extracts isSampled() == false.

---

The SDK Decision Model: SamplingResult States

In application code and custom SDK plugins, the Sampler interface returns a structured SamplingResult object containing a Decision enum, an optional set of span attributes to append, and an updated TraceState.

The OpenTelemetry specification defines exactly three SamplingDecision enum values:

┌────────────────────────────────────────────────────────────────────────┐
│                     SamplingDecision Enum Values                       │
├────────────────────┬────────────────────┬──────────────────────────────┤
│ Decision State     │ isRecording()      │ Low Bit of TraceFlags (Wire) │
├────────────────────┼────────────────────┼──────────────────────────────┤
│ DROP               │ false              │ 0 (0x00)                     │
│ RECORD_ONLY        │ true               │ 0 (0x00)                     │
│ RECORD_AND_SAMPLE  │ true               │ 1 (0x01)                     │
└────────────────────┴────────────────────┴──────────────────────────────┘

1. DROP

• The span is completely discarded. Calling span.isRecording() returns false.
• Any operations attempting to set attributes (span.setAttribute()), add span events (span.addEvent()), or record exceptions are immediate no-ops, consuming zero memory or CPU cycles.
• The span is never dispatched to registered SpanProcessor instances (such as BatchSpanProcessor) and is never exported.
• The low bit of TraceFlags is 0 (0x00).

2. RECORD_ONLY

• The span records execution data in memory: span.isRecording() returns true.
• Attributes, events, links, and status changes are populated and processed by active SpanProcessor.onStart() and SpanProcessor.onEnd() callbacks.
• Crucial Detail: The low bit of TraceFlags is set to 0 (0x00). The span is not exported to the telemetry backend.
• Primary Use Case: Generating real-time in-process metrics (e.g., span-to-metrics counters or latency histograms calculated by custom processors) across 100% of requests without incurring the massive network egress bandwidth and storage cost of exporting full trace graphs.

3. RECORD_AND_SAMPLE

• Full observability enabled: span.isRecording() returns true.
• All attributes, events, and status updates are recorded in memory.
• The low bit of TraceFlags is set to 1 (0x01).
• The completed span is forwarded to registered SpanProcessors and queued for network transmission by exporters (e.g., OTLP gRPC/HTTP exporter).

---

Built-in OpenTelemetry Samplers

These four classic samplers are the ones every SDK exposes through OTEL_TRACES_SAMPLER:

                               Built-in Samplers
                                       │
         ┌──────────────────┬──────────┴──────────┬──────────────────┐
         ▼                  ▼                     ▼                  ▼
    AlwaysOn           AlwaysOff          TraceIdRatioBased     ParentBased
  (100% Export)       (0% Export)         (Probabilistic N%)   (Tree Wrapper)

1. AlwaysOn (Environment identifier: always_on)

• Behavior: Unconditionally returns RECORD_AND_SAMPLE for every span evaluated.
• Configuration: Requires no parameters or arguments.
• Use Cases: Development environments, automated CI/CD integration testing, low-traffic critical microservices (e.g., an internal authentication authority processing 10 requests per minute), or regulatory audit compliance systems where missing a single transaction is unacceptable.
• Drawbacks: Disastrous in high-throughput production systems; leads to memory exhaustion, network saturation, and exorbitant storage bills.

2. AlwaysOff (Environment identifier: always_off)

• Behavior: Unconditionally returns DROP for every span evaluated.
• Configuration: Requires no parameters or arguments.
• Use Cases: Completely disabling tracing in non-critical environments, emergency circuit breaking during severe collector outages, or silencing high-volume batch data pipelines where tracing is undesired.

3. TraceIdRatioBased(ratio) (Environment identifier: traceidratio)

• Behavior: Probabilistically samples a configurable percentage of root traces based on a floating-point ratio between 0.0 (0%) and 1.0 (100%). For example, a ratio of 0.05 samples approximately 5% of traces.
• The Algorithm: The decision must be deterministic: the sampler derives it from the TraceId (commonly by comparing part of the ID, often its lower 64 bits, with ratio times the maximum value) rather than from a random number generator. A sampler with a higher ratio must also sample every trace that a lower-ratio sampler would sample.
• Compatibility Warning: The specification never fixed the exact algorithm, so SDKs in different languages (or different SDK versions) can reach different decisions for the same trace ID. The specification therefore recommends using it only as the root sampler inside ParentBased.
• Deprecation: TraceIdRatioBased is now deprecated in favor of ProbabilitySampler (Development status). ProbabilitySampler uses the W3C Level 2 randomness: it compares the trace ID's rightmost 56 bits (R) with a rejection threshold (T) derived from the ratio, samples when R ≥ T, and records the threshold as th in the OpenTelemetry tracestate entry, so every SDK makes the same decision. SDKs must keep TraceIdRatioBased working unchanged until at least January 1, 2027.
• Critical Pitfall: Used alone on downstream services, TraceIdRatioBased ignores the parent's sampled flag and decides independently. With different ratios, or with SDKs that hash differently, a child span can be dropped while its parent was sampled, creating broken traces.

4. ParentBased(root_sampler) (Environment identifiers: parentbased_always_on, parentbased_traceidratio, etc.)

• The Production Standard Wrapper: In production distributed systems, you should almost never configure TraceIdRatioBased or AlwaysOn directly as your standalone sampler. You must wrap them inside ParentBased!
• Core Architectural Purpose: ParentBased is a composite sampler that respects the upstream caller's sampling decision for all child spans, guaranteeing that a distributed trace is captured as an unbroken, contiguous tree.

Incoming Span Creation
         │
         ├───── Has Parent Context? ──────┐
         │ (Yes)                          │ (No - Root Span)
         ▼                                ▼
Is Parent Sampled?               Delegate to root_sampler
 ├── YES (01) ──> SAMPLE CHILD    (e.g., TraceIdRatioBased(0.10))
 └── NO  (00) ──> DROP CHILD

Two further samplers appear in the specification. JaegerRemoteSampler polls a Jaeger-compatible endpoint for per-service sampling strategies. AlwaysRecord wraps another sampler and turns its DROP decisions into RECORD_ONLY, so processors (for example span-to-metrics processors) see every span while only sampled spans are exported.

Configurable Delegate Paths in ParentBased

While ParentBased has intelligent defaults, the OpenTelemetry specification allows engineering teams to customize five distinct delegation scenarios:

---

Built-in Samplers Architectural Reference

---

Real-World Production Scenario: Preventing "Swiss Cheese" Traces

Consider an enterprise banking platform with three microservices in series: API Gateway → Account Service → Ledger Service.

The Failure Mode: Unwrapped Ratio Sampling

The operations team configures all three services with standalone TraceIdRatioBased(0.10):

1. An incoming customer deposit hits the API Gateway.
2. The API Gateway evaluates its local TraceIdRatioBased sampler. The hash lands under the 10% threshold: RECORD_AND_SAMPLE (TraceFlags 01).
3. The API Gateway calls the Account Service via HTTP, passing traceparent: 00-4bf9...-00f0...-01.
4. The Account Service also uses a standalone TraceIdRatioBased(0.10), so it ignores the parent's sampled flag and makes its own decision. It runs a different language SDK from the gateway, and because the ratio algorithm was never standardized, its hash of the same TraceId does not match the gateway's: its decision is effectively independent.
5. The Ledger Service goes further and uses a ratio of 0.02. Even where hashing agrees, a 2% sampler drops most traces that a 10% sampler kept.
6. The result in the observability backend is a fragmented "swiss cheese" trace: an API Gateway root span exists, but downstream account and ledger spans are missing.

The Architectural Resolution

The team updates all services to use ParentBased(TraceIdRatioBased(0.10)):

• The API Gateway acts as the root boundary, sampling 10% of new transactions.
• Whenever the API Gateway samples a trace, both Account Service and Ledger Service detect remoteParentSampled and execute their default AlwaysOn delegate.
• 100% of the child spans for that transaction are captured, delivering an unbroken, fully correlated distributed execution DAG.