7.2 Metric Readers & Log Processors

7.2 Metric Readers & Log Processors

Quick Answer: While traces and logs represent discrete, event-driven records processed by SpanProcessors and LogRecordProcessors, metrics represent continuous numeric state orchestrated by MetricReaders. A MetricReader bridges the MeterProvider to a MetricExporter and controls when metric collection occurs. OpenTelemetry provides two primary metric reader architectures: PeriodicExportingMetricReader (which runs a background timer to push metrics to an exporter at fixed intervals, default 60000ms / 60s) and pull-based readers like PrometheusMetricReader (which exposes an HTTP scrape endpoint like /metrics to aggregate and serialize metrics on demand). Furthermore, Metric Readers configure Aggregation Temporality Selectors to determine whether instruments emit Cumulative or Delta data points.

The three primary telemetry signals, traces, metrics, and logs, have fundamentally different temporal and operational characteristics. Understanding why metrics utilize Readers rather than Processors is essential for mastering the OpenTelemetry SDK architecture.

---

Why Metrics Use "Readers" Instead of "Processors"

To understand the OpenTelemetry pipeline architecture, consider the nature of the data being transmitted:

• Traces and Logs are Event-Driven: When an HTTP request completes, span.end() is invoked. When an error occurs, logger.error() is called. Each event creates a discrete record that must be immediately processed, buffered, and exported. Therefore, traces and logs employ Processors (SpanProcessor and LogRecordProcessor) that react immediately to lifecycle triggers (onEnd or emit).
• Metrics are State-Based and Continuous: When an application executes counter.add(1) or histogram.record(0.045), it does not generate an individual event to be sent across the network. If an application handles 50,000 requests per second, sending 50,000 discrete metric events would overwhelm networks and storage backends. Instead, metric instruments update in-memory thread-safe accumulators (such as atomic counters or histogram bucket sketches). The OpenTelemetry SDK needs a component to periodically "read" these accumulators, aggregate them across time windows, and export the compressed summary. That component is the MetricReader.

Signal     Pipeline Component    Lifecycle Trigger            Operational Model
Traces  -> SpanProcessor      -> span.end()                -> Event-driven stream
Logs    -> LogRecordProcessor -> logger.emit()             -> Event-driven stream
Metrics -> MetricReader       -> Periodic timer or scrape   -> Continuous state sampling

---

The Architecture of a MetricReader

A MetricReader is registered with the MeterProvider. It serves as the conductor of the metric pipeline. Whenever a collection is triggered, the MetricReader coordinates a multi-step sequence within the SDK:

1. Lock-Free Accumulator Read: The reader queries all registered synchronous instruments (Counter, UpDownCounter, Histogram) and captures their current in-memory values.
2. Callback Execution: The reader invokes all registered Asynchronous (Observable) Callback functions (e.g., ObservableGauge, ObservableCounter). This allows applications to poll external runtime state (such as JVM heap usage, CPU percentage, or disk queue depth) precisely when a collection cycle occurs.
3. View Evaluation: The reader passes raw instrument measurements through any configured View definitions, which can rename metrics, alter histogram bucket boundaries, or drop unwanted attribute keys to manage cardinality.
4. Temporality Selection: The reader converts raw values into either Cumulative or Delta data points based on the exporter's configured AggregationTemporalitySelector.
5. Export Handoff: The reader delivers the finalized batch of MetricData to the registered MetricExporter (or serializes it for an HTTP scrape response).

---

Push-Based vs. Pull-Based Metric Readers

The OpenTelemetry Metrics SDK supports two distinct collection and delivery paradigms:

+-----------------------------------------------------------------------------------+
|                      PeriodicExportingMetricReader (PUSH)                         |
|                                                                                   |
|  [MeterProvider] <--- Timer (export_interval_millis: 60s)                         |
|         │                                                                         |
|         ▼                                                                         |
|    collect() ---> [OTLP MetricExporter] ---> HTTP POST/gRPC ---> [OTel Collector] |
+-----------------------------------------------------------------------------------+

+-----------------------------------------------------------------------------------+
|                         PrometheusMetricReader (PULL)                             |
|                                                                                   |
|  [MeterProvider]                                                                  |
|         │                                                                         |
|         ▼                                                                         |
|    collect() <--- HTTP GET /metrics <--- [Prometheus Scraper Server]              |
+-----------------------------------------------------------------------------------+

1. Push Model: PeriodicExportingMetricReader

The PeriodicExportingMetricReader is the standard push-based reader in OpenTelemetry. It encapsulates a dedicated background scheduler thread and a configured MetricExporter (such as OTLPMetricExporter over gRPC or HTTP).

Key Tuning Parameters

• export_interval_millis (Environment: OTEL_METRIC_EXPORT_INTERVAL):
  • Default: 60000ms (60 seconds).
  • Governs how frequently the background timer fires to collect metrics across all instruments and dispatch them to the exporter.
  • Tuning considerations: Lowering this interval to 15000ms (15s) provides higher-resolution alerting for volatile services, but quadruples network bandwidth and storage ingestion rates.
• export_timeout_millis (Environment: OTEL_METRIC_EXPORT_TIMEOUT):
  • Default: 30000ms (30 seconds).
  • Governs the maximum duration the reader permits for a collection and export RPC before timing out and aborting.

2. Pull Model: PrometheusMetricReader

The PrometheusMetricReader implements the pull-based model canonical to the Prometheus ecosystem:

• It does not run an internal periodic timer to push metrics out.
• Instead, it exposes or binds to an HTTP server endpoint (conventionally /metrics on port 9464 or the application's primary management port).
• When an external Prometheus server or OpenTelemetry Collector scrape job initiates an HTTP GET /metrics request, the PrometheusMetricReader intercepts the request, invokes collect() across the MeterProvider, translates the data into Prometheus text exposition format (or OpenMetrics format), and returns the HTTP 200 response.

Multiple Readers on a Single MeterProvider

A powerful capability of the OpenTelemetry SDK is that a single MeterProvider can host multiple MetricReaders simultaneously. For example, an enterprise service can register:

1. A PeriodicExportingMetricReader configured to push OTLP metrics every 30 seconds to a centralized OpenTelemetry Collector gateway.
2. A PrometheusMetricReader bound to port 9464 allowing local Kubernetes Prometheus operators to scrape metrics directly.

Both readers pull from the same underlying instruments without interfering with one another.

---

Aggregation Temporality Selectors

Aggregation Temporality (introduced in Section 5.3) belongs here too, because the reader and exporter decide it. Temporality defines the temporal reference window represented by a metric measurement.

Cumulative vs. Delta Temporality

Time Window:         0s - 10s       10s - 20s      20s - 30s
Events in Window:    5 requests     8 requests     3 requests

Cumulative Output:   5              13             16            (Sum grows monotonically)
Delta Output:        5              8              3             (Reports only window delta)

Why Temporality Matters

• Prometheus requires Cumulative: Prometheus was architecturally designed around cumulative counters. In Prometheus, if a counter resets to 0, Prometheus interprets it as a process restart and automatically handles counter resets in rate calculations (via rate() or increase()). If delta values were exposed as Prometheus counters, every interval would look like a counter reset, which breaks rate queries.
• Some push backends prefer Delta: Delta points are self-contained per interval, so an aggregator can sum them across many nodes without tracking per-series state or detecting restarts, and the SDK can forget idle series after each export.

How the MetricReader Controls Temporality

KEY PRINCIPLE: Application developers never configure temporality when authoring code! You do not set temporality on a Counter or Histogram. Instead, temporality is governed by the MetricReader / MetricExporter via an AggregationTemporalitySelector function.

When a MetricReader collects measurements, it queries its exporter's temporality selector. The selector maps instrument kinds (Counter, UpDownCounter, Histogram) to their desired temporality:

• OTLPMetricExporter can be configured via environment variable OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE to cumulative (default), delta, or lowmemory (delta for synchronous Counter and Histogram only).
• PrometheusMetricReader always mandates Cumulative temporality for monotonic sums and histograms, because Prometheus cannot digest Delta metrics.

---

Log Record Processors

As explored in the logging architecture, the LoggerProvider manages emitted log records through registered LogRecordProcessor instances. Log processors mirror the design of span processors:

1. SimpleLogRecordProcessor

• Immediately and synchronously dispatches each log record to the configured LogRecordExporter on the application caller thread during emit().
• Anti-Pattern for Production: Blocking application threads on synchronous network I/O for every logger.info() call severely degrades application throughput.
• Target Use Cases: Serverless functions (AWS Lambda, where execution environments freeze immediately upon handler exit), command-line tools, and unit testing.

2. BatchLogRecordProcessor

• Buffers incoming log records into an internal thread-safe bounded memory queue.
• A background worker thread drains the queue and exports records in batches.
• Tuning Parameters:
  • Maximum queue size (OTEL_BLRP_MAX_QUEUE_SIZE): records held in the buffer (default: 2048).
  • Schedule delay (OTEL_BLRP_SCHEDULE_DELAY): interval between batch exports (default: 1000ms, shorter than the span processor's 5000 ms).
  • Maximum export batch size (OTEL_BLRP_MAX_EXPORT_BATCH_SIZE): records per export (default: 512).
  • Export timeout (OTEL_BLRP_EXPORT_TIMEOUT): default 30000ms.

---

Cross-Signal Pipeline Comparison

The following table synthesizes the architectural differences across all three telemetry signal pipelines in the OpenTelemetry SDK:

---

Practical Code Implementation: Metric and Log Pipelines

The following Python example demonstrates how to configure both a PeriodicExportingMetricReader and a BatchLogRecordProcessor with custom tuning parameters:

from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter

# 1. Configure OTLP Metric Pipeline
metric_exporter = OTLPMetricExporter(endpoint="http://otel-collector:4317", insecure=True)

# Periodic reader collecting and pushing metrics every 30s
metric_reader = PeriodicExportingMetricReader(
    exporter=metric_exporter,
    export_interval_millis=30000,  # Collect every 30 seconds
    export_timeout_millis=10000    # 10 second timeout
)

meter_provider = MeterProvider(metric_readers=[metric_reader])

# 2. Configure OTLP Logging Pipeline
log_exporter = OTLPLogExporter(endpoint="http://otel-collector:4317", insecure=True)

# Batch log processor buffering up to 4096 records
log_processor = BatchLogRecordProcessor(
    exporter=log_exporter,
    max_queue_size=4096,
    schedule_delay_millis=2000,    # Flush batch every 2 seconds
    max_export_batch_size=512,
    export_timeout_millis=10000
)

logger_provider = LoggerProvider()
logger_provider.add_log_record_processor(log_processor)