13.3 Schema Management & Telemetry Evolution

13.3 Schema Management & Telemetry Evolution

Quick Answer: As OpenTelemetry semantic conventions mature, attribute and metric names evolve (such as transitioning from http.status_code to http.response.status_code, and http.method to http.request.method). Without schema management, these modifications break dashboards, alerting rules, and downstream pipelines. OpenTelemetry solves this through the Schema File Specification, which defines versioned transformations (rename_attributes) identified by a unique schema_url. SDKs embed these schema URLs into Resources and Instrumentation Scopes (getTracer(name, version, schemaUrl)). At runtime, the Collector bridges versions using the transform processor with OTTL to dual-write both legacy and modern attributes, ensuring seamless zero-downtime migrations.

Observability is an evolving engineering discipline. Over the lifecycle of an enterprise platform, underlying operating systems, cloud providers, runtime libraries, and telemetry frameworks undergo continuous modernization. In OpenTelemetry, this evolution is codified in the Semantic Conventions—the standardized naming catalogs governing resource attributes, span names, metric instruments, and log record fields.

However, standardization introduces a major operational challenge: what happens when standard semantic conventions change? If an attribute name shifts from http.status_code to http.response.status_code, every Grafana dashboard panel, Prometheus alerting rule, SLO calculation, and data warehouse query that expects the old attribute name breaks instantly. Schema management is the set of standards, specifications, and runtime transformation mechanisms that enable telemetry to evolve smoothly without breaking downstream consumers.

---

The Telemetry Evolution Problem

In early versions of the OpenTelemetry specification, semantic conventions were largely experimental. As conventions achieved Stable status, the specification committee refined attribute names to ensure consistency, clarity, and domain separation across HTTP, database, messaging, RPC, and cloud infrastructure telemetry.

Historical Examples of Semantic Convention Shifts

The Enterprise Blast Radius

Consider an enterprise observability platform ingesting telemetry from 500 microservices across 20 independent engineering squads. If an automated CI/CD pipeline upgrades the OpenTelemetry Java instrumentation agent across these services, the agent immediately begins emitting http.response.status_code instead of http.status_code.

The immediate consequences include:

• Silent Alert Failures: Prometheus alerting queries filtering on {http_status_code=~"5.."} return zero series, leaving on-call teams completely blind to production outages.
• Empty Dashboard Panels: Grafana dashboards displaying application error budgets drop to 0% error rate, giving a false sense of stability.
• Broken Stream Processing: Kafka streams or vector search jobs indexing span attributes fail schema validation checks and crash.

To prevent this operational chaos, OpenTelemetry created a formal, declarative schema evolution framework.

---

OpenTelemetry Schema File Specification

An OpenTelemetry Schema File is a declarative, version-controlled YAML or JSON document that defines the precise transformations required to map telemetry from one semantic convention version to another. Rather than requiring developers to write bespoke migration scripts, schema files provide a standardized format that automated tools, SDKs, and Collector processors can parse and execute.

Core Elements of the Schema File

1. file_format: The version of the schema file format itself (currently 1.1.0).
2. schema_url: The URL where the file is published; its version must match the highest version in the file (e.g., https://opentelemetry.io/schemas/1.22.0).
3. versions: One entry per schema version. Each entry is divided into sections: all, resources, spans, span_events, metrics, and logs.
4. changes: Each section lists the transformations needed to move from the previous version:
   • rename_attributes: An attribute_map of old key to new key (in the metrics section, an optional apply_to_metrics list limits it to named metrics).
   • rename_metrics: A map of old metric name to new metric name.
   • rename_events (span events) and split (metrics) cover the remaining cases.

A schema file can only express mechanical transformations such as renames and metric splits. Splitting the old http.target into url.path and url.query cannot be written that way, which is why it is absent from the published schema files.

Schema File Example

The following excerpt, adapted from the published schema file, shows the HTTP renames of semantic conventions v1.21.0 and the HTTP metric renames of v1.22.0:

file_format: 1.1.0
schema_url: https://opentelemetry.io/schemas/1.22.0
versions:
  1.22.0:
    metrics:
      changes:
        - rename_metrics:
            http.server.duration: http.server.request.duration
            http.client.duration: http.client.request.duration
  1.21.0:
    spans:
      changes:
        - rename_attributes:
            attribute_map:
              http.method: http.request.method
              http.status_code: http.response.status_code
              http.scheme: url.scheme
              http.url: url.full
              net.host.name: server.address
              net.host.port: server.port
              http.client_ip: client.address
  1.20.0:

---

Emitting Schema URLs in Telemetry

For schema evolution to work at runtime, downstream consumers must know which schema version was used to produce a given telemetry payload. The OpenTelemetry specification accomplishes this by embedding the schema_url directly into the OpenTelemetry Protocol (OTLP) data model at two distinct levels:

1. Resource Level (Resource.schema_url): Identifies the schema version used for resource attributes (service.name, host.id, k8s.pod.name).
2. Instrumentation Scope Level (ScopeSpans.schema_url, ScopeMetrics.schema_url, ScopeLogs.schema_url): Identifies the schema version used by the specific library or tracer producing spans, metrics, and logs.

+-------------------------------------------------------------------------+
|                    OTLP Payload Schema Hierarchy                        |
+-------------------------------------------------------------------------+
  ResourceSpans
  ├── Resource (Attributes: service.name="order-svc", host.id="i-123")
  │   └── schema_url: "https://opentelemetry.io/schemas/1.24.0"
  │
  └── ScopeSpans (Scope: name="io.opentelemetry.http", version="1.32.0")
      ├── schema_url: "https://opentelemetry.io/schemas/1.24.0"
      └── Spans
          ├── Span 1 (Attributes: http.response.status_code=200)
          └── Span 2 (Attributes: http.response.status_code=500)

Setting Schema URLs in Language SDKs

When instantiating resources or acquiring tracers and meters, developers and instrumentation authors should explicitly provide the schema URL.

Java SDK Example

// 1. Defining Schema URL on the Resource
Resource resource = Resource.getDefault()
    .merge(Resource.create(
        Attributes.of(AttributeKey.stringKey("service.name"), "checkout-service"),
        "https://opentelemetry.io/schemas/1.24.0"
    ));

// 2. Defining Schema URL on the Tracer (Instrumentation Scope)
Tracer tracer = openTelemetry.tracerBuilder("com.example.checkout") // Instrumentation Scope Name
    .setInstrumentationVersion("1.4.0")                                 // Instrumentation Scope Version
    .setSchemaUrl("https://opentelemetry.io/schemas/1.24.0")            // Schema URL
    .build();

Go SDK Example

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/trace"
)

// Acquiring a tracer with an explicit schema URL
tracer := otel.GetTracerProvider().Tracer(
    "com.example.checkout",
    trace.WithInstrumentationVersion("1.4.0"),
    trace.WithSchemaURL("https://opentelemetry.io/schemas/1.24.0"),
)

When telemetry is exported over OTLP, the SDK stamps the schema_url string into the protobuf message envelope. Downstream collectors and APM backends inspect this field to determine whether translation is required.

---

Schema Translation & Evolution at Runtime

In enterprise production environments, microservices cannot all be upgraded simultaneously. A large organization will run a heterogeneous mix of services emitting telemetry across multiple schema versions for months.

To bridge this gap without breaking dashboards or forcing immediate code rewrites, teams use two tools. At the source, instrumentations that support it can emit both old and new names during a migration (for HTTP, OTEL_SEMCONV_STABILITY_OPT_IN=http/dup). In the pipeline, the Collector can translate telemetry: the Contrib schema processor (alpha stability) applies published schema files based on each payload's schema_url, and the transform processor can write explicit OTTL rules.

The Dual-Writing Pattern (Zero-Downtime Migration Standard)

The industry standard for zero-downtime schema evolution is the dual-writing pattern. During a planned migration window, the OpenTelemetry Collector's transform processor evaluates incoming spans. If a span arrives with modern attributes, the processor populates the legacy attribute; if a span arrives with legacy attributes, it populates the modern attribute.

By ensuring that both attributes exist simultaneously on every span, legacy dashboards continue functioning uninterrupted while observability teams build, test, and validate new dashboards.

processors:
  transform/schema_migration:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          # 1. Backport: If modern attribute exists, populate legacy attribute
          - set(attributes["http.status_code"], attributes["http.response.status_code"]) where attributes["http.response.status_code"] != nil
          - set(attributes["http.method"], attributes["http.request.method"]) where attributes["http.request.method"] != nil
          - set(attributes["http.url"], attributes["url.full"]) where attributes["url.full"] != nil

          # 2. Forward-port: If legacy attribute exists, populate modern attribute
          - set(attributes["http.response.status_code"], attributes["http.status_code"]) where attributes["http.status_code"] != nil and attributes["http.response.status_code"] == nil
          - set(attributes["http.request.method"], attributes["http.method"]) where attributes["http.method"] != nil and attributes["http.request.method"] == nil
          - set(attributes["url.full"], attributes["http.url"]) where attributes["http.url"] != nil and attributes["url.full"] == nil

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, transform/schema_migration, batch]
      exporters: [otlp]

---

Long-Term Schema Governance Best Practices

To maintain high data quality and operational stability across large-scale platforms, organizations should adopt four fundamental schema governance practices:

1. Maintain a Central Schema Registry: Host corporate semantic conventions in a Git repository. Publish internal schema files documenting custom enterprise attributes (e.g., tenant.id, billing.tier) alongside standard OpenTelemetry conventions.
2. Enforce Semantic Conventions in CI/CD: Integrate automated linters and unit tests into application build pipelines. Verify that custom instrumentation code uses approved attribute keys and provides valid schema_url metadata before merging pull requests.
3. Structured Deprecation Windows: Establish formal deprecation cycles (e.g., 90 days). When upgrading semantic conventions, run dual-writing in the Collector for the duration of the window, notifying dashboard and alert owners to migrate their queries.
4. Automate Query Auditing: Query the Prometheus and Grafana APIs to audit which dashboards and alert rules still reference deprecated attribute names before removing dual-writing rules from Collector pipelines.