10.3 Standard Exporters & Delivery Guarantees

10.3 Standard Exporters & Delivery Guarantees

Quick Answer: Exporters are the egress components of the OpenTelemetry Collector that translate internal pdata structures into destination formats and transmit them over the network to external observability platforms. The canonical exporters are otlp (gRPC) and otlphttp (HTTP), while debug (which supersedes the deprecated logging exporter) outputs telemetry to stdout for local inspection. Exporter reliability relies on two foundational features: sending_queue (a FIFO queue that decouples ingestion from export latency, with optional disk-backed persistent storage) and retry_on_failure (exponential backoff for transient errors such as HTTP 429/503 and gRPC UNAVAILABLE). With retries enabled, delivery of queued data is at-least-once, so network retries can introduce duplicates at the backend, while non-retryable client errors (HTTP 400, 401) are dropped immediately to prevent queue poison pills; only a persistent queue survives a crash.

In an enterprise observability pipeline, telemetry must be routed reliably to disparate destinations: distributed tracing backends (such as Jaeger, Tempo, or AWS X-Ray), metric time-series databases (such as Prometheus, Mimir, or Google Cloud Monitoring), and log search clusters (such as Elasticsearch or OpenSearch). Exporters decouple internal Collector processing from the network transport requirements of these external systems.

---

Core Exporters

Know the canonical exporters, their key configuration parameters, verbosity levels, and operational behavior.

1. The otlp & otlphttp Exporters (The Canonical Standards)

The primary method for exporting telemetry from an OpenTelemetry Collector to downstream Collectors or OTLP-native observability platforms is via the OpenTelemetry Protocol exporters:

• otlp Exporter (gRPC): Transmits binary Protobuf payloads over HTTP/2 multiplexed gRPC connections (default destination port 4317). It provides maximum network throughput, low CPU overhead, and persistent connection multiplexing.
• otlphttp Exporter (HTTP): Transmits Protobuf or JSON payloads over standard HTTP/1.1 or HTTP/2 POST requests (default destination port 4318, sending to paths /v1/traces, /v1/metrics, or /v1/logs). It is ideal when routing telemetry through corporate HTTP proxies, cloud Application Load Balancers (ALBs), or API gateways that do not support gRPC streaming.

exporters:
  # Canonical OTLP gRPC exporter
  otlp/backend:
    endpoint: tempo.internal.net:4317
    tls:
      ca_file: /etc/otel/certs/ca.pem
      cert_file: /etc/otel/certs/client.pem
      key_file: /etc/otel/certs/client.key
    compression: gzip # Supported: gzip, zstd, none
    timeout: 10s
    headers:
      Authorization: "Bearer ${env:API_KEY}"

  # Canonical OTLP HTTP exporter
  otlphttp/external:
    endpoint: https://otlp.monitoring.vendor.com
    encoding: proto # Supported: proto, json
    compression: gzip
    timeout: 15s

2. The debug Exporter (Local Troubleshooting)

During development, staging verification, or production debugging, operators must inspect raw telemetry items entering or exiting a pipeline. The debug exporter prints telemetry payloads directly to the Collector's standard output (stdout) or standard error (stderr).

exporters:
  debug:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 100

Debug Verbosity Levels

• basic (default): Emits concise one-line summaries displaying the count of processed spans, metric data points, or log records and their associated service names. Extremely lightweight; suitable for low-overhead production sanity checks.
• normal: Emits key metadata including span names, trace IDs, timestamps, and primary attributes. Provides a balance between visibility and log volume.
• detailed: Emits the complete, unabridged internal pdata structure, printing every span event, link, attribute key-value pair, resource attribute block, and raw metric histogram bucket. Warning: Generates massive log volume; should only be enabled in testing or isolated staging environments.

3. The prometheus Exporter (Pull-Based Metric Egress)

While most exporters push data outward, the prometheus exporter hosts an HTTP server that exposes a /metrics scrape endpoint. External Prometheus or VictoriaMetrics servers can then scrape the Collector on a configured interval.

• Default Endpoint: 0.0.0.0:8889/metrics
• Metric Mapping: Translates OpenTelemetry pmetric.Metrics into Prometheus exposition format.
• Distinction: Do not confuse the prometheus exporter (a pull server hosted on port 8889) with the prometheusremotewrite exporter (a push client that dispatches metrics via HTTP POST to a remote Prometheus Remote Write endpoint).

---

Exporter Resilience & Reliability Features

In enterprise networks, downstream observability backends experience network partitions, transient rate-limiting spikes, and maintenance outages. Without robust resilience mechanisms, transient backend disruptions would cause immediate telemetry loss or trigger cascading backpressure that stalls upstream microservices.

The OpenTelemetry Collector provides two tightly integrated resilience mechanisms on all standard push exporters: sending_queue and retry_on_failure.

exporters:
  otlp/resilient:
    endpoint: apm.internal.net:4317
    sending_queue:
      enabled: true
      num_consumers: 16
      queue_size: 10000
      storage: file_storage # Optional: enables persistent disk-backed queuing
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 5m

1. sending_queue (In-Memory & Persistent Buffering)

The sending_queue decouples pipeline ingestion from exporter network transmission using a First-In, First-Out (FIFO) queue:

• enabled: Activates the queuing mechanism (default: true).
• num_consumers: The number of concurrent worker goroutines that pull batches from the queue and execute network RPC calls to the destination backend (default: 10). Increasing consumer workers improves export throughput over high-latency WAN connections.
• queue_size: The queue's capacity (default: 1000, counted in requests, i.e. incoming batches, because the default sizer is requests; items and bytes sizers also exist). When the queue is full, new data is rejected (counted by otelcol_exporter_enqueue_failed_*), unless block_on_overflow: true makes the caller wait.
• wait_for_result (default false): With the default, the pipeline reports success as soon as data is queued; the upstream client is not told whether the backend later accepts it.
• batch: Newer releases can batch inside the exporter (sending_queue::batch, disabled by default; batch: {} enables it with a 200 ms flush timeout and a minimum of 8192 items), an alternative to the batch processor.
• storage (Persistent Queuing): References a storage extension (such as file_storage). When configured, queued batches are written to durable local disk storage rather than RAM. If the Collector pod crashes, runs out of memory, or restarts during a network partition, buffered telemetry is restored from disk upon boot and dispatched without data loss.

2. retry_on_failure (Exponential Backoff)

When an exporter attempts to deliver a batch and receives a network timeout or transient server error, retry_on_failure automatically schedules retries using an exponential backoff algorithm with jitter:

• enabled: Activates automated retries (default: true).
• initial_interval: The duration to wait before executing the first retry attempt (default: 5s).
• max_interval: The upper ceiling on the backoff duration (default: 30s). Each subsequent retry increases the wait interval exponentially ($t_{\text{next}} = \min(\text{max\_interval}, t_{\text{current}} \times 1.5)$) plus random jitter to prevent thundering herd problems.
• max_elapsed_time: The maximum total cumulative time spent retrying a single batch (default: 5m). If the backend remains unreachable after max_elapsed_time expires, the batch is permanently discarded, the failure is counted in otelcol_exporter_send_failed_spans (or the metric-point and log-record equivalents), and the worker moves to the next batch.

3. Retryable vs. Non-Retryable Errors (Poison Pill Avoidance)

Exporters sort backend HTTP and gRPC status codes into two groups:

---

Delivery Guarantees: At-Least-Once vs. Best-Effort

Understanding the delivery guarantees provided by the OpenTelemetry Collector is vital for architecting resilient distributed systems:

At-Least-Once Delivery Semantics

With retry_on_failure enabled, the Collector's delivery to the backend is at-least-once for data it still holds: a batch leaves the queue only after a success or a permanent error. How far that guarantee extends depends on the queue:

1. With the default wait_for_result: false, the receiver acknowledges the client once the data is accepted into the pipeline and queued, not when the backend confirms it. An in-memory queue loses whatever it holds if the process crashes; only a persistent queue (storage: file_storage) carries queued data across a restart.
2. A batch is only removed from the queue after the downstream backend responds with a success code (200 OK or gRPC OK) or a permanent error.
3. Duplicate Generation Scenario: Suppose the Collector transmits a batch of 5,000 spans to an APM backend. The backend successfully processes and writes the spans to its database, but a network router blips before the HTTP 200 OK response packet reaches the Collector. The Collector experiences a socket timeout, assumes delivery failed, and retries the batch. The backend receives the batch a second time, resulting in duplicate spans.
4. Handling Duplicates: Observability backends must handle deduplication using unique span IDs and trace IDs, or accept minor metric over-counting during network recovery events.

Best-Effort Delivery Semantics

Under certain failure modes, the Collector operates under Best-Effort delivery, where telemetry may be discarded to preserve system stability:

• When memory reaches the memory_limiter threshold and incoming data is dropped to prevent an OOM crash.
• When the sending_queue fills to maximum capacity (queue_size exceeded) during a prolonged outage.
• When a batch exceeds max_elapsed_time in the retry loop.
• When non-retryable errors (HTTP 400/401) are dropped.
• When the process crashes while data sits in an in-memory queue.