8.2 Tail-Based Sampling vs Head-Based Sampling

8.2 Tail-Based Sampling vs Head-Based Sampling

Quick Answer: Tail-based sampling defers the decision to sample or drop a trace until the very end ("tail") of the request lifecycle, after all child spans have finished executing and all telemetry attributes, status codes, and durations are known. This solves the fundamental flaw of head-based sampling, where probabilistic sampling drops 99% of rare errors and slow requests. Because distributed traces span multiple microservice runtimes, tail-based sampling cannot be implemented inside a single application SDK; it requires an out-of-process intermediate pipeline—specifically the OpenTelemetry Collector's tail_sampling processor—coupled with a trace-id-aware loadbalancing exporter to route all spans of a trace to the identical Collector instance.

While head-based sampling delivers exceptional performance and minimal CPU overhead at the application runtime, it imposes a severe observability compromise. In cloud environments where 99.9% of user requests succeed normally and only 0.1% experience catastrophic failure or unacceptable tail latency, probabilistic head sampling operates blindly, dropping the vast majority of the very incidents on-call engineers need to diagnose. Resolving this conflict requires shifting the decision boundary from the application ingress to an intelligent, out-of-process sampling tier.

---

The Mathematical Dilemma of Head-Based Sampling

To understand why tail-based sampling is an indispensable requirement for enterprise systems, consider the mathematical reality of probabilistic head-based sampling in high-scale architectures.

The Anomaly Blind Spot

Suppose an e-commerce platform processes 10,000,000 requests per day. The system maintains a healthy 99.9% success rate, meaning that 0.1% of transactions (10,000 requests) experience an unhandled exception, database lock timeout, or HTTP 500 internal server error.

To control tracing ingestion and storage expenses, the engineering team configures head-based sampling at a 1% ratio (TraceIdRatioBased(0.01) wrapped in ParentBased):

• Total Traces Sampled: 10,000,000 × 0.01 = 100,000 traces recorded and exported.
• Error Traces Captured: 10,000 × 0.01 = 100 error traces captured.
• Error Traces Discarded: 10,000 × 0.99 = 9,900 error traces permanently lost!

Head-Based Sampling (1% Probabilistic Ratio):

Total Errors Occurred: [ 10,000 Failure Events ]
├── Captured in Backend:  100 Traces  ( 1% )  <-- Tiny statistical sample
└── Dropped at Ingress: 9,900 Traces  (99% )  <-- Complete blind spot!

Because the sampling decision was locked in at the root span when the request first touched the API Gateway, 99 out of every 100 failure events were discarded before the error ever occurred. If an SRE investigates an incident that impacted 50 users in a specific geographical region, there is a high probability that zero traces were captured for that incident.

To guarantee capturing all 10,000 error traces using head-based sampling, the platform would have to increase the sampling ratio to 100%, forcing the organization to ingest, process, and pay for 9,990,000 uninteresting "200 OK" traces.

---

Mechanics of Tail-Based Sampling: Retrospective Decision Making

Tail-based sampling flips the sampling paradigm by deferring the decision until the transaction has completed across all participating microservices.

Microservices A, B, C emit 100% of spans via OTLP
                       │
                       ▼
       ┌────────────────────────────────┐
       │ OpenTelemetry Collector Buffer  │
       │ Holds spans for decision_wait  │
       │ (e.g., 30-second trace window) │
       └───────────────┬────────────────┘
                       │
                       ▼
             Evaluate All Spans:
             ├── Any span status == ERROR?      ──► KEEP (100% Export)
             ├── Any span duration > 2000ms?    ──► KEEP (100% Export)
             ├── Any http.response.status_code == 500? ──► KEEP (100% Export)
             ├── customer.tier == "enterprise"? ──► KEEP (100% Export)
             └── Normal 200 OK fast traces      ──► SAMPLE at 0.1% Ratio

The Core Workflow

1. Full Generation: Application microservices do not drop spans at ingress. Every service creates spans for 100% of requests and exports them immediately over the local network via OTLP.
2. In-Memory Buffering: An intermediate collector cluster ingests the spans and holds them in an in-memory buffer for a configurable evaluation window (the decision_wait duration, typically 10 to 30 seconds).
3. DAG Assembly: Spans sharing the same TraceId are assembled into their respective trace tree.
4. Policy Inspection: Once the decision window expires or the trace root completes, a comprehensive policy engine inspects every span in the trace.
5. Selective Retention: If any span in the trace matches an error rule, a latency threshold, or a high-value customer attribute, the entire distributed trace (root span to leaf span) is forwarded to storage. If the trace represents a routine, fast success, it is sampled down to a tiny fraction (e.g., 0.1%) or discarded.

---

Architectural Requirements: Why Tail Sampling Belongs in the Collector

A natural question is: Why can't tail-based sampling be implemented directly inside the application SDK?

There are two insurmountable architectural reasons:

1. Microservice Memory Isolation: In a distributed system, a single trace traverses multiple distinct operating system processes, container pods, and serverless runtimes. Service A (the API Gateway) completes its root span in 15 milliseconds and sends the response to the user. Service C (the background billing worker) might encounter a database lock 4 seconds later on a completely different Kubernetes node. Service A cannot look into Service C's memory space to determine whether Service C will encounter an error.
2. Application Memory Protection: Buffering thousands of spans in application heap memory while waiting for asynchronous distributed transactions to finish introduces massive garbage collection overhead, memory bloat, and the risk of Out-Of-Memory (OOM) application crashes. Telemetry processing must never destabilize core business applications.

Therefore, tail-based sampling is an out-of-process concern. It is implemented in the intermediate telemetry tier using the OpenTelemetry Collector—specifically using the tail_sampling processor available in the OpenTelemetry Collector Contrib distribution.

Configuration of the tail_sampling Processor

The tail_sampling processor in the OpenTelemetry Collector Contrib repository exposes rich policy primitives:

processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 50000
    expected_new_traces_per_sec: 2000
    policies:
      # Policy 1: Retain 100% of traces with an explicit Error status
      - name: drop-errors-never
        type: status_code
        status_code: { status_codes: [ ERROR ] }

      # Policy 2: Retain 100% of slow requests (tail latency > 2500ms)
      - name: slow-traces-filter
        type: latency
        latency: { threshold_ms: 2500 }

      # Policy 3: Retain 100% of server errors via HTTP status code
      - name: http-5xx-rules
        type: numeric_attribute
        numeric_attribute:
          key: http.response.status_code
          min_value: 500
          max_value: 599

      # Policy 4: Retain all enterprise VIP customer requests
      - name: vip-tenants
        type: string_attribute
        string_attribute:
          key: customer.tier
          values: [ enterprise, platinum ]

      # Policy 5: Sample remaining routine 200 OK traces at 1%
      - name: routine-traffic-probabilistic
        type: probabilistic
        probabilistic: { sampling_percentage: 1.0 }

• decision_wait (default 30s): How long the Collector buffers a trace, counted from its first span, before evaluating the policies. Spans that arrive later follow the stored decision while the trace is still in memory; decision_cache settings keep decisions (not spans) for longer so that late spans still follow them.
• num_traces (default 50000): The maximum number of distinct traces held in memory; when it is exceeded, the oldest traces are evicted, which bounds memory but can drop data under heavy load.
• Policy logic: A trace is kept if any policy decides to sample it; and policies require several conditions together, and drop policies explicitly discard matching traces.
• expected_new_traces_per_sec: Pre-allocates internal hash table memory to eliminate dynamic allocation pauses under heavy ingress.

---

The Trace Co-Location Problem & The Two-Tier Architecture

The central engineering challenge of tail-based sampling in a scaled production environment is trace co-location.

The Fatal Flaw of Standard Round-Robin Load Balancing

Suppose you deploy three instances of the OpenTelemetry Collector (Collector 1, Collector 2, Collector 3) behind a standard Kubernetes ClusterIP Service or AWS Application Load Balancer:

• Service A generates the root span for TraceId: abc-123 and sends it to the Kubernetes service VIP. The load balancer routes it to Collector 1.
• Service B generates a child span for TraceId: abc-123 and sends it to the VIP. The load balancer routes it to Collector 2.
• Service C generates a child span that throws an HTTP 500 error for TraceId: abc-123. The load balancer routes it to Collector 3.

                        Kubernetes Service (Round-Robin)
                                      │
           ┌──────────────────────────┼──────────────────────────┐
           ▼                          ▼                          ▼
      Collector 1                Collector 2                Collector 3
┌────────────────────┐     ┌────────────────────┐     ┌────────────────────┐
│ Root Span          │     │ Child Span         │     │ Error Span         │
│ (Trace abc-123)    │     │ (Trace abc-123)    │     │ (Trace abc-123)    │
│                    │     │                    │     │                    │
│ Decision: DROP!    │     │ Decision: DROP!    │     │ Decision: KEEP!    │
│ (No error seen)    │     │ (No error seen)    │     │ (Error seen)       │
└────────────────────┘     └────────────────────┘     └────────────────────┘

The Disaster: Because the spans for TraceId: abc-123 were scattered across three different collector memories:

• Collector 1 evaluates only the root span. It sees no error and drops the root span!
• Collector 2 evaluates only the middle span and drops it!
• Collector 3 sees the error and decides to keep its span—but when it exports to the backend, the trace has no root span, no HTTP request URL, and no client metadata! The trace is completely severed.

The Mandatory Solution: Two-Tier Collector Topology

To make tail-based sampling function in a horizontally scalable architecture, all spans sharing the same TraceId must arrive at the exact same Collector instance. This requires deploying a two-tier collector topology using the loadbalancing exporter:

1. Tier 1: Agent / Ingress Tier (Stateless Routing)
   • Deployed as a Kubernetes DaemonSet on every node or as a local container sidecar.
   • Applications send OTLP to the node-local agent (through the node's host IP) or, with sidecars, to localhost:4317 / localhost:4318.
   • Tier 1 collectors do not run the tail_sampling processor. Instead, they configure the loadbalancing exporter.
   • The loadbalancing exporter reads the TraceId of every span, computes a consistent hash, and routes all spans with that TraceId to a deterministic member of Tier 2.
2. Tier 2: Tail-Sampling Gateway Tier (Stateful Evaluation)
   • Deployed as an independent, scalable pool of OpenTelemetry Collector gateway instances.
   • Every span belonging to TraceId: abc-123 lands exclusively on Gateway Instance 2.
   • Gateway Instance 2 buffers the spans, reconstructs the full distributed trace DAG, executes the tail-sampling policies, and exports complete traces to the storage backend.

Head-Based vs. Tail-Based Sampling: Comprehensive Comparison

The following comparison table summarizes the technical differences, resource profiles, and architectural trade-offs: