11.3 Advanced Connectors

11.3 Advanced Connectors

Quick Answer: A Connector is a dual-personality component introduced in modern versions of the OpenTelemetry Collector. It simultaneously acts as an exporter at the end of one pipeline and as a receiver at the beginning of another pipeline. Connectors allow cross-signal telemetry derivation—such as generating Prometheus RED metrics from distributed trace spans—within a single collector process memory space, completely eliminating the network latency, CPU serialization overhead, and operational fragility of chaining multiple collector instances together.

In early OpenTelemetry Collector architectures, pipelines operated in strict isolation: a traces pipeline could only ingest traces and export traces; a metrics pipeline could only ingest metrics and export metrics. If an engineering team wanted to extract aggregated performance metrics from trace spans, they had to deploy a complex architecture: Collector A would export traces over the network via OTLP to Collector B, which would ingest those spans via an OTLP receiver and convert them into metrics. This multi-hop design doubled CPU usage, wasted network bandwidth, and introduced artificial failure domains. Connectors were introduced to solve this architectural problem permanently.

---

The Architecture of a Connector: The Dual Personality

A Connector bridges two pipelines inside the Collector's internal runtime. Because both pipelines reside within the same Go process heap, telemetry is passed between them via in-memory pointers, achieving zero network overhead and zero serialization penalty.

+-----------------------------------------------------------------------------------+
|                         OpenTelemetry Collector Process                           |
|                                                                                   |
|  +-----------------------------------------------------------------------------+  |
|  | Pipeline 1 (Traces)                                                         |  |
|  |   [otlp receiver] ---> [batch processor] ---> [spanmetrics CONNECTOR]       |  |
|  +-------------------------------------------------------│---------------------+  |
|                                                          │ (In-memory handoff)     |
|  +-------------------------------------------------------▼---------------------+  |
|  | Pipeline 2 (Metrics)                                                        |  |
|  |   [spanmetrics CONNECTOR] ---> [batch processor] ---> [prometheus exporter] |  |
|  +-----------------------------------------------------------------------------+  |
+-----------------------------------------------------------------------------------+

Pipeline Placement Rules

In the collector configuration file (config.yaml), a connector is defined once in the top-level connectors: block. However, to function, it must appear twice in service.pipelines:

1. It must be listed under the exporters: list of the upstream feeder pipeline.
2. It must be listed under the receivers: list of the downstream consumer pipeline.

If a connector is defined under connectors: but omitted from either the upstream exporters or downstream receivers, the Collector fails schema validation at startup and refuses to launch.

---

Three Widely Used Connectors

The Collector ships many connectors (forward in Core; failover, roundrobin, servicegraph, and others in Contrib). Three are used most often:

                               Core Connectors
                                      │
         ┌────────────────────────────┼────────────────────────────┐
         ▼                            ▼                            ▼
    spanmetrics                     count                       routing
(Traces -> Metrics)           (Any -> Metrics)              (Any -> Same Signal)
RED Metrics & Latencies      Volume & Chargeback Billing   Multi-Tenant Pipeline Steering

---

1. The spanmetrics Connector

The spanmetrics connector is the most widely deployed connector in production cloud architectures. It consumes distributed trace spans and computes real-time RED metrics (Rate, Errors, Duration) across all instrumented services.

What Metrics Does It Generate?

• traces.span.metrics.calls (Counter): The number of spans, with the default dimensions service.name, span.name, span.kind, and status.code. A Prometheus exporter shows it as traces_span_metrics_calls_total.
• Error Rate: Spans whose status is Error land in the status.code="Error" series of the calls metric, so error percentages come straight from the counter without querying trace storage.
• traces.span.metrics.duration (Histogram): Span durations in milliseconds by default (shown by Prometheus as traces_span_metrics_duration_milliseconds_bucket and related series), used for p50, p90, and p99 latency objectives.

The traces.span.metrics prefix is the default namespace setting; older configurations produced names such as calls_total and duration_milliseconds.

Key Features and Configuration Attributes

• Dimension Extraction: By default, spanmetrics records core span attributes. SREs can extract additional span or resource attributes (e.g., http.route, http.response.status_code, rpc.method, cloud.region) into metric dimensions using the dimensions: configuration block; every extra dimension multiplies the number of series.
• Custom Latency Buckets: Teams can tailor histogram boundaries with histogram.explicit.buckets (default [2ms, 4ms, 6ms, 8ms, 10ms, 50ms, 100ms, 200ms, 400ms, 800ms, 1s, 1400ms, 2s, 5s, 10s, 15s]) or switch to histogram.exponential.
• Unsampled Golden Signals (Key Advantage): In high-throughput architectures, teams frequently configure head-based or tail-based sampling to drop 90%–99% of spans to control storage costs. If metrics are derived after sampling, request counts and error rates become severely skewed. By placing the spanmetrics connector before tail-based sampling, the collector computes 100% accurate, unsampled RED metrics while still discarding 95% of trace data downstream!

---

2. The count Connector

The count connector measures the volume of telemetry flowing through a pipeline. It can consume spans, metric datapoints, or log records and emit summary count metrics at regular intervals.

Production Use Cases

• Pipeline Health Monitoring: Verifying that collector pipelines are actively receiving and processing telemetry without silently stalling.
• Internal Chargeback & Cost Allocation: Measuring the exact telemetry footprint (number of spans, log records, or metric points) generated by individual development teams, departments, or Kubernetes namespaces (e.g., grouping counts by k8s.namespace.name or service.name).
• Capacity Planning: Forecasting storage ingestion requirements based on telemetry volume trends.

Generated Metrics

• trace.span.count: Total spans received per dimension over time.
• metric.datapoint.count: Total metric data points processed.
• log.record.count: Total log records ingested.

---

3. The routing Connector

The routing connector dynamically inspects incoming telemetry attributes and steers data to distinct downstream pipelines. Unlike spanmetrics (which translates traces into metrics), the routing connector routes data within the same signal type (traces to traces, logs to logs, or metrics to metrics).

Key Routing Mechanisms

• OTTL Condition Routing: Each table entry holds an OTTL condition (or a statement of the form route() where ...) that selects its pipelines, for example resource.attributes["deployment.environment.name"] == "production". Records that match no entry go to default_pipelines.
• Move or Copy: Each routing-table entry has an action. The default, move, sends matched data to its pipelines and removes it from later evaluation; copy lets the same data match further routes too.

Production Use Cases

• Data Sovereignty and Compliance (GDPR): Directing EU customer telemetry to European storage backends and US customer telemetry to North American backends based on geo.country.
• Tiered Multi-Tenancy: Routing enterprise paying customers to high-availability, high-retention storage tiers while routing free-tier customers to sampled, low-cost object storage.
• Error Isolation: Routing error traces (status.code == STATUS_CODE_ERROR) to high-priority alert queues while directing successful spans to standard batch aggregators.

---

Comprehensive YAML Architecture: Connecting Traces to Metrics

The following configuration illustrates an enterprise OpenTelemetry Collector setup utilizing both spanmetrics and routing connectors:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

connectors:
  # 1. Generate RED metrics from distributed traces
  spanmetrics:
    histogram:
      explicit:
        buckets: [5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s]
    dimensions:
      - name: http.response.status_code
      - name: http.route
    aggregation_temporality: "AGGREGATION_TEMPORALITY_CUMULATIVE"

  # 2. Route traces based on environment
  routing:
    default_pipelines: [traces/dev]
    table:
      - condition: resource.attributes["deployment.environment.name"] == "production"
        pipelines: [traces/prod]

processors:
  batch:
    send_batch_size: 1024
    timeout: 1s

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  otlp/prod:
    endpoint: "prod-jaeger:4317"
    tls: { insecure: true }
  otlp/dev:
    endpoint: "dev-jaeger:4317"
    tls: { insecure: true }

service:
  pipelines:
    # Ingress Traces Pipeline: Ingests OTLP, feeds spanmetrics and routing connectors
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [spanmetrics, routing] # spanmetrics acts as EXPORTER here

    # Downstream Metrics Pipeline: Receives derived metrics from spanmetrics
    metrics/from_spans:
      receivers: [spanmetrics]          # spanmetrics acts as RECEIVER here
      processors: [batch]
      exporters: [prometheus]

    # Production Traces Destination
    traces/prod:
      receivers: [routing]
      processors: [batch]
      exporters: [otlp/prod]

    # Development Traces Destination
    traces/dev:
      receivers: [routing]
      processors: [batch]
      exporters: [otlp/dev]