11.1 OTTL Concepts & Syntax Grammar

11.1 OTTL Concepts & Syntax Grammar

Quick Answer: The OpenTelemetry Transformation Language (OTTL) is a declarative domain-specific language (DSL) purpose-built for the OpenTelemetry Collector. It provides a standardized grammar to inspect, modify, enrich, and filter in-memory telemetry data (pdata) across traces, metrics, and logs. An OTTL statement follows the canonical form function(target, arguments...) where condition. Every statement executes within a designated Context (such as resource, scope, span, spanevent, metric, datapoint, or log), which establishes the root of the telemetry tree and dictates which paths and metadata fields can be queried or mutated.

In modern enterprise observability architectures, raw telemetry emitted by diverse applications rarely arrives in an ideal state. Microservices may emit legacy attribute keys, sensitive credentials, unparsed JSON log payloads, or mismatched semantic conventions. Before OpenTelemetry standardized transformation, engineering teams were forced to chain multiple disparate Collector processors—such as attributesprocessor, resourceprocessor, filterprocessor, metricstransformprocessor, and logstransformprocessor—each with its own idiosyncratic configuration syntax, varying regex engines, and separate memory allocations. OTTL was introduced to replace this fragmented ecosystem with a single, highly performant, unified transformation engine.

---

Why OTTL Was Created: Unified Pipeline Mutation

In the OpenTelemetry Collector architecture, telemetry is represented internally using Go memory structures called pdata (Pluggable Data / Pipeline Data). pdata provides high-performance, memory-efficient abstractions over the underlying OpenTelemetry Protocol (OTLP) Protocol Buffers.

Before OTTL, transforming pdata across pipeline stages presented severe operational challenges:

• Configuration Fragmentation: A team wanting to mask a credit card number in a log body had to use regex in logstransformprocessor, while dropping a span attribute required attributesprocessor, and appending a cluster identifier required resourceprocessor.
• Serialization Overhead: Chaining five separate processors introduced cumulative CPU overhead, as each processor independently traversed and re-validated pipeline data structures.
• Inconsistent Filtering Syntax: Filtering logic differed between traces (which used trace-specific Boolean trees) and metrics (which used metric label matching filters).

OTTL resolves these issues with one grammar shared by many components (the transform, filter, and routing components, and tail-sampling conditions). Statements are parsed once at startup into Go functions that operate directly on pdata, so no configuration is re-interpreted for each record.

---

The Anatomy of OTTL Grammar

An OTTL transformation is expressed through statements, expressions, paths, and predicates.

+-----------------------------------------------------------------------------------+
|                             OTTL Statement Grammar                                |
+-----------------------------------------------------------------------------------+
|  set(attributes["customer.tier"], "gold") where attributes["customer.id"] == 101   |
|  │   │                            │       │     │                                 |
|  │   │                            │       │     └─ Condition Predicate (Boolean)  |
|  │   │                            │       └─────── Where Keyword                  |
|  │   │                            └─────────────── Value Expression (Literal)     |
|  │   └──────────────────────────────────────────── Target Path Expression         |
|  └──────────────────────────────────────────────── Invocable Transformation Function
+-----------------------------------------------------------------------------------+

1. Invocable Functions: Editors and Converters

Every statement begins with an invocable function name followed by comma-separated arguments in parentheses: function_name(arg1, arg2, ...). OTTL has two kinds of functions:

• Editors have lowercase names (set, delete_key, keep_keys, replace_pattern, merge_maps) and change telemetry. A statement always starts with an editor.
• Converters have UpperCamelCase names (Concat, ParseJSON, IsMatch, Int, ExtractPatterns) and return a value. They appear as arguments to editors or inside where conditions and never change data on their own.

2. Path Expressions

A Path points to a specific field or map entry within the pdata hierarchy. OTTL paths adhere to strict formatting conventions:

• Dot Notation (.): Used to traverse fixed struct fields defined in the OpenTelemetry data model (e.g., span.name, span.kind, log.severity_number, status.code).
• Bracket Notation (["key"]): Used to access key-value collections such as attribute maps or resource metadata (e.g., attributes["http.response.status_code"], resource.attributes["k8s.pod.name"]).
• Context-Qualified Paths: Current OTTL also accepts paths that start with the context name, such as span.attributes["region"], log.body, or resource.attributes["service.name"]. With these, the transform processor's basic configuration can infer the context of each statement automatically.
• Keys inside bracket notation must be enclosed in double quotes.

3. Conditions (Predicates) and the where Clause

A statement may optionally include a where clause. When present, the function executes only if the condition evaluates to true for the record being processed. If the condition evaluates to false, the record passes through unmodified without error.

OTTL supports rich logical and relational operators:

• Equality / Inequality: ==, !=
• Relational Comparisons: <, <=, >, >= (applicable to numbers and timestamps)
• Logical Operators: and, or, not
• Grouping: Parentheses ( ... ) to enforce precedence
• Nil Evaluation: Checking whether an attribute exists: attributes["auth.token"] != nil

Example of a compound condition:

set(attributes["environment"], "production") where attributes["env"] == "prod" or (resource.attributes["cloud.provider"] == "aws" and attributes["region"] == "us-east-1")

---

The OTTL Execution Context Hierarchy

In OpenTelemetry, telemetry is structured in a strict object hierarchy: a batch contains Resources, each Resource contains Instrumentation Scopes, each Scope contains Telemetry Items (Spans, Metric DataPoints, or Logs), and Spans/Metrics contain nested sub-items (SpanEvents, Histogram Buckets).

An OTTL statement must operate within a designated Context. The context sets the local operational root. Paths resolve relative to this context, while parent objects remain accessible via explicit qualifiers.

+-----------------------------------------------------------------------------------+
|                         OTTL Context Resolution Hierarchy                         |
+-----------------------------------------------------------------------------------+
|  Resource Context                                                                 |
|  └── resource.attributes, resource.dropped_attributes_count                       |
|      │                                                                            |
|      ├── Scope Context                                                            |
|      │   └── instrumentation_scope.name, instrumentation_scope.version            |
|      │       │                                                                    |
|      │       ├── Span Context                                                     |
|      │       │   ├── name, kind, trace_id, span_id, status.code, attributes[...]  |
|      │       │   └── SpanEvent Context                                            |
|      │       │       └── name, time_unix_nano, attributes[...]                    |
|      │       │                                                                    |
|      │       ├── Metric / DataPoint Context                                       |
|      │       │   ├── metric.name, metric.unit, metric.type                        |
|      │       │   └── value_int, value_double, time_unix_nano, attributes[...]     |
|      │       │                                                                    |
|      │       └── Log Context                                                      |
|      │           └── body, severity_number, severity_text, attributes[...]        |
+-----------------------------------------------------------------------------------+

The Seven Standard OTTL Contexts

1. resource Context:

   • Scope: Operates once per Resource entity across all signals.
   • Local Paths: attributes, dropped_attributes_count.
   • Use Case: Normalizing host, container, cloud, or cluster metadata before downstream signal processing.

2. scope Context:

   • Scope: Operates at the Instrumentation Scope boundary (the library or package that emitted the telemetry).
   • Local Paths: name, version, attributes, dropped_attributes_count.
   • Use Case: Overriding instrumentation library versions or scoping flags.

3. span Context:

   • Scope: Operates on individual distributed trace spans.
   • Local Paths: name, kind, start_time_unix_nano, end_time_unix_nano, trace_id, span_id, parent_span_id, trace_state, status.code, status.message, attributes.
   • Parent Paths: Full read/write access to resource.attributes and instrumentation_scope.name.
   • Use Case: Redacting sensitive URLs in span attributes, fixing span names, updating span status.

4. spanevent Context:

   • Scope: Operates on discrete events embedded within a span.
   • Local Paths: name, time_unix_nano, attributes, dropped_attributes_count.
   • Parent Paths: Access to the parent span's fields via span.* and resource.*.
   • Use Case: Scrubbing stack trace arguments or masking parameters recorded in span exception events.

5. metric Context:

   • Scope: Operates on metric metadata across all data points of that metric.
   • Local Paths: name, description, unit, type.
   • Parent Paths: Access to resource.attributes.
   • Use Case: Renaming metric instruments or standardizing metric unit strings (e.g., converting ms to s).

6. datapoint Context:

   • Scope: Operates on individual metric measurements (Gauge, Sum, Histogram, Summary).
   • Local Paths: value_int, value_double, start_time_unix_nano, time_unix_nano, flags, attributes, exemplars.
   • Parent Paths: Access to metric metadata via metric.* and resource metadata via resource.*.
   • Use Case: Adding or dropping label dimensions on specific metric data points, converting values.

7. log Context:

   • Scope: Operates on individual log records.
   • Local Paths: time_unix_nano, observed_time_unix_nano, severity_number, severity_text, body, flags, trace_id, span_id, attributes.
   • Parent Paths: Access to resource.attributes and instrumentation_scope.name.
   • Use Case: Parsing raw string bodies into structured JSON attributes, mapping severity levels, masking PII.

---

OTTL Context & Path Reference Matrix

The following reference table summarizes the contexts, target signals, accessible paths, and common field paths:

---

OTTL Data Types and Literal Representation

OTTL is a strictly typed language. When executing comparisons or setting values, literals and types must conform to supported Go primitives:

Important Exam Distinction: Nil vs. Empty String

In OTTL, an attribute key that does not exist evaluates to nil. An attribute that exists with an empty string value evaluates to "".

• To check if a key is missing: attributes["user.email"] == nil
• To check if a key is present and contains text: attributes["user.email"] != nil and attributes["user.email"] != ""