8.3 OpenTelemetry SDK Configuration & Environment Variables

8.3 OpenTelemetry SDK Configuration & Environment Variables

Quick Answer: The OpenTelemetry SDK adheres strictly to 12-factor application methodology by providing code-free configuration via standardized OTEL_* environment variables. This enables operators to modify service identity, exporters, protocols, endpoints, samplers, and processor buffer sizing without modifying or recompiling application code. Crucially, signal-specific variables (such as OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) strictly override generic fallback variables (OTEL_EXPORTER_OTLP_ENDPOINT), and OTEL_SERVICE_NAME takes strict precedence over any service.name declared inside OTEL_RESOURCE_ATTRIBUTES.

In modern cloud-native deployment patterns, application binaries and container images are immutable artifacts promoted sequentially through development, staging, canary, and production environments. Hardcoding telemetry configurations—such as backend endpoints, authentication tokens, sampling ratios, or batching queues—into source code violates basic operational practices. OpenTelemetry resolves this by defining a cross-language specification for SDK configuration through operating system environment variables, plus a YAML file format for richer setups (covered at the end of this section).

The specification defines one common set of environment variables with shared parsing and precedence rules. Language SDKs implement most of them; the specification's compliance matrix records the gaps for each language.

---

Essential General SDK Environment Variables

Before exploring signal-specific transport pipelines, candidates must master the fundamental environment variables that establish service identity, context propagation, and diagnostic visibility.

1. OTEL_SERVICE_NAME

• Purpose: Defines the logical service name associated with all telemetry emitted by the process (e.g., checkout-service, auth-worker).
• Default: If service.name is not set anywhere, the SDK uses unknown_service: plus the process executable name (e.g., unknown_service:java), or plain unknown_service if the executable name is unavailable.
• Critical Precedence Rule: If both OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES define a service name, OTEL_SERVICE_NAME takes absolute precedence!

  # In this scenario, service.name resolves to 'order-processor'
  export OTEL_SERVICE_NAME="order-processor"
  export OTEL_RESOURCE_ATTRIBUTES="service.name=legacy-billing,deployment.environment.name=prod"
  

2. OTEL_RESOURCE_ATTRIBUTES

• Purpose: A comma-separated list of key-value pairs defining resource entity attributes that describe the host, container, cloud platform, and application version.
• Syntax: key1=val1,key2=val2
• Example:

  export OTEL_RESOURCE_ATTRIBUTES="service.version=3.4.1,deployment.environment.name=production,host.id=ip-10-0-4-12,k8s.cluster.name=prod-us-east"
  

• Character Escaping: Values containing commas, spaces, equals signs, or other reserved characters must be percent-encoded (for example %2C for a comma); quoting or backslash escaping is not part of the format.

3. OTEL_LOG_LEVEL

• Purpose: Sets the diagnostic log verbosity of the internal OpenTelemetry SDK itself (controls the SDK's internal logging for initialization, connection dropped, or queue overflow warnings). It does not filter or control the application's business log output!
• Accepted Values: Language-specific level names such as debug, info, warn, and error (Node.js, for example, also accepts none, verbose, and all).
• Default: info.
• Operational Troubleshooting: When an application starts but spans never arrive at the Collector, setting OTEL_LOG_LEVEL=debug immediately exposes whether the OTLP connection failed, timed out, or encountered TLS certificate validation rejections.

4. OTEL_PROPAGATORS

• Purpose: Specifies a comma-separated list of context propagators to register into the global composite TextMapPropagator.
• Default: "tracecontext,baggage" (W3C Trace Context and W3C Baggage).
• Accepted Identifiers:
  • tracecontext — W3C Trace Context (traceparent, tracestate).
  • baggage — W3C Baggage (baggage).
  • b3 — Zipkin B3 Single Header format (b3: {TraceId}-{SpanId}-{SamplingState}-{ParentSpanId}).
  • b3multi — Zipkin B3 Multi-Header format (X-B3-TraceId, X-B3-SpanId, X-B3-Sampled).
  • jaeger — Legacy Jaeger native format (uber-trace-id).
  • xray — AWS X-Ray format (X-Amzn-Trace-Id).
  • ottrace — OpenTracing legacy format.
  • none — Disables context propagation entirely.
• Example:

  # Support both modern W3C standards and legacy Zipkin headers
  export OTEL_PROPAGATORS="tracecontext,baggage,b3multi"
  

---

Signal-Specific Protocol & Endpoint Configuration

A central configuration skill is understanding how the OpenTelemetry SDK resolves endpoints, protocols, and headers across different telemetry signals.

The General vs. Signal-Specific Precedence Hierarchy

The specification defines a two-tier configuration model: Generic Fallback Variables and Signal-Specific Override Variables. The rule is absolute: Signal-specific variables always take precedence over generic variables.

Signal-Specific Endpoint (e.g., OTEL_EXPORTER_OTLP_TRACES_ENDPOINT)
                            │
                      Is it set?
                      ├── YES ──► Use Signal-Specific Value (exact URL)
                      └── NO  ──► Fall back to Generic Variable
                                  (OTEL_EXPORTER_OTLP_ENDPOINT)

Endpoint Resolution & URL Path Concatenation

Understanding how endpoints and paths are resolved requires knowing whether the transport is gRPC or HTTP:

1. Default Ports:

   • OTLP / gRPC default port: 4317
   • OTLP / HTTP default port: 4318

2. Generic Endpoint Path Concatenation (HTTP only): When you configure the generic OTEL_EXPORTER_OTLP_ENDPOINT for an HTTP protocol (such as http/protobuf), the SDK automatically appends standard signal path suffixes to the base URL:

   • Traces: {generic_endpoint}/v1/traces
   • Metrics: {generic_endpoint}/v1/metrics
   • Logs: {generic_endpoint}/v1/logs

   Example: If OTEL_EXPORTER_OTLP_ENDPOINT="http://my-collector:4318", the SDK automatically sends traces to http://my-collector:4318/v1/traces.

3. Signal-Specific Endpoint (No Path Concatenation): When a signal-specific variable is provided, the SDK does not append any path suffix. It uses the provided URL verbatim!

   • Example: If OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://trace-gw:4318/custom/trace/path", the SDK sends traces to that exact URL without appending /v1/traces.

Protocol Configuration

• Generic Variable: OTEL_EXPORTER_OTLP_PROTOCOL
• Signal Overrides: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, OTEL_EXPORTER_OTLP_METRICS_PROTOCOL, OTEL_EXPORTER_OTLP_LOGS_PROTOCOL
• Valid Options:
  • grpc — OTLP over gRPC (HTTP/2 with binary Protobuf payloads, default port 4317).
  • http/protobuf — OTLP over HTTP with binary Protobuf payloads (default port 4318).
  • http/json — OTLP over HTTP with JSON payloads (useful for debugging, default port 4318).

Headers, Timeouts & Compression

• Headers: Configured as comma-separated key-value pairs.
  • Generic: OTEL_EXPORTER_OTLP_HEADERS="api-key=secret123,x-tenant=engineering"
  • Signal-specific: OTEL_EXPORTER_OTLP_TRACES_HEADERS="api-key=trace-secret"
• Timeout: Maximum time in milliseconds to wait before an export network request fails.
  • Generic: OTEL_EXPORTER_OTLP_TIMEOUT (Default: 10000 = 10 seconds).
  • Signal-specific: OTEL_EXPORTER_OTLP_TRACES_TIMEOUT
• Compression: Compression algorithm applied to OTLP network payloads.
  • Generic: OTEL_EXPORTER_OTLP_COMPRESSION (Accepted: gzip, none. Default: none or language-dependent).
  • Signal-specific: OTEL_EXPORTER_OTLP_TRACES_COMPRESSION

---

Sampler Configuration via Environment Variables

Instead of hardcoding sampler classes in code, the OpenTelemetry SDK allows complete sampling control via two standardized variables:

1. OTEL_TRACES_SAMPLER: Defines the sampler algorithm.
2. OTEL_TRACES_SAMPLER_ARG: Supplies the optional numeric parameter required by ratio-based samplers.

Standard Sampler Environment Values

# Production deployment example: Sample 5% of root traces, respect parent everywhere else
export OTEL_TRACES_SAMPLER="parentbased_traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.05"

---

Tuning the BatchSpanProcessor via Environment Variables

The BatchSpanProcessor is the standard production processor that accepts spans from active tracers, queues them in an in-memory buffer, and dispatches them asynchronously to the configured exporter in batches. Fine-tuning the BatchSpanProcessor via environment variables is essential to prevent application latency degradation and memory exhaustion.

Tracer.end() ──► [ BatchSpanProcessor Queue (OTEL_BSP_MAX_QUEUE_SIZE) ]
                                      │
                      Trigger: OTEL_BSP_SCHEDULE_DELAY
                           or: OTEL_BSP_MAX_EXPORT_BATCH_SIZE
                                      ▼
                         [ OTLP Exporter Network Request ]

Key Batching Environment Variables

• OTEL_BSP_SCHEDULE_DELAY:
  • Meaning: The delay interval in milliseconds between consecutive buffer flush cycles.
  • Default: 5000 ms (5 seconds).
  • Tuning: Lowering this value (e.g., to 1000 ms) forces more frequent export cycles, reducing in-memory span residency at the expense of more frequent network requests.
• OTEL_BSP_MAX_QUEUE_SIZE:
  • Meaning: The maximum number of spans that can sit in the internal buffer awaiting export.
  • Default: 2048 spans.
  • Tuning: If an application experiences sudden traffic surges and the internal queue reaches capacity, subsequent completed spans are dropped to protect host memory. During high-throughput bursts, operators increase this to 8192 or 16384 if host RAM permits.
• OTEL_BSP_MAX_EXPORT_BATCH_SIZE:
  • Meaning: The maximum number of spans dispatched in a single exporter network payload.
  • Default: 512 spans.
  • Constraint: Must be less than or equal to OTEL_BSP_MAX_QUEUE_SIZE.
  • Tuning: Increasing batch size improves network efficiency by amortizing HTTP/gRPC frame overhead across more spans.
• OTEL_BSP_EXPORT_TIMEOUT:
  • Meaning: The maximum duration in milliseconds that an export batch attempt may take before being cancelled.
  • Default: 30000 ms (30 seconds).

---

Master Reference: Top 20 Standard OpenTelemetry Environment Variables

The following table consolidates 20 core environment variables:

---

Declarative Configuration with OTEL_CONFIG_FILE

Environment variables are flat, so some settings (such as per-view options or several exporters for one signal) cannot be expressed with them. The declarative configuration specification solves this with a YAML file:

• Point the SDK at the file with OTEL_CONFIG_FILE=/path/to/otel-config.yaml (the older OTEL_EXPERIMENTAL_CONFIG_FILE name is deprecated).
• When OTEL_CONFIG_FILE is set, all other OTEL_* environment variables are ignored, except those the file references through substitution syntax such as ${OTEL_EXPORTER_OTLP_ENDPOINT} or ${VAR:-default}.
• The configuration schema itself is stable (experimental properties carry a /development suffix), but SDK support is still experimental; the Java agent is the most complete implementation today.

file_format: "1.2"
resource:
  attributes_list: ${OTEL_RESOURCE_ATTRIBUTES}
propagator:
  composite:
    - tracecontext:
    - baggage:
tracer_provider:
  sampler:
    parent_based:
      root:
        always_on:
  processors:
    - batch:
        exporter:
          otlp_http:
            endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT:-http://localhost:4318}/v1/traces

One more switch belongs in every toolkit: OTEL_SDK_DISABLED=true makes the SDK a no-op for all signals (propagators configured through OTEL_PROPAGATORS still work), which is useful for quickly ruling telemetry in or out during an incident.

Summary of Key Takeaways

• Configuration through standardized environment variables decouples telemetry control from application source code.
• Signal-specific exporter variables strictly override general OTLP endpoint and protocol settings.
• Service naming follows a strict hierarchy: OTEL_SERVICE_NAME takes precedence over service.name inside OTEL_RESOURCE_ATTRIBUTES.
• Production distributed tracing typically uses parentbased_traceidratio to preserve full traces while keeping data volumes within budget.
• A declarative YAML file (OTEL_CONFIG_FILE) replaces the flat environment variables entirely when it is set.
• The BatchSpanProcessor queue and batch limits must be tuned in concert to avoid dropping spans under high-throughput production load.