3.2 The Core Trace Model: Spans, SpanContext & Links

3.2 The Core Trace Model: Spans, SpanContext & Links

Distributed tracing provides visibility into the end-to-end path of a transaction as it traverses complex microservice architectures, serverless functions, database queries, and message brokers. To represent these distributed journeys without vendor bias, OpenTelemetry establishes a standardized Trace Data Model.

---

Anatomy of a Span: The Atomic Unit of Tracing

A Span is the foundational building block of a distributed trace. It represents a single contiguous segment of work performed by an individual system or component over a measurable interval of time. Spans within a trace assemble into a Directed Acyclic Graph (DAG) of parent-child relationships that together describe the entire causal tree of a distributed transaction.

[Trace: 4bf92f3577b34da6a3ce929d0e0e4736]

Span A (Root, SERVER) -----------------------------------------------------> [220ms]
  |-- Span B (CLIENT) ---------> [45ms] (Outgoing HTTP Call)
  |     `-- Span C (SERVER) ---> [40ms] (Remote Auth Service)
  |-- Span D (INTERNAL) -------> [15ms] (Token Validation)
  `-- Span E (CLIENT) -----------------------------------------> [130ms] (Database Query)

Core Fields of a Span

Every span generated by an OpenTelemetry SDK contains a standardized set of structural fields:

1. Span Name: A low-cardinality, human-readable string identifying the operation being performed. The OpenTelemetry specification mandates that span names must represent the class or category of work rather than an ephemeral instance.
   • Valid Span Names (Low Cardinality): GET /api/v1/users/{id}, SELECT orders, checkout.process_payment.
   • Invalid Span Names (High Cardinality Anti-Pattern): GET /api/v1/users/91823, SELECT orders WHERE id = '824982'. Injecting dynamic identifiers into span names destroys backend indexing and query aggregation.
2. TraceID: A globally unique 16-byte (128-bit) identifier represented as a 32-character lowercase hexadecimal string (e.g., 4bf92f3577b34da6a3ce929d0e0e4736). All spans participating in the same distributed transaction share the identical TraceID.
3. SpanID: A unique 8-byte (64-bit) identifier represented as a 16-character lowercase hexadecimal string (e.g., 00f067aa0ba902b7). Every span within a trace has a distinct SpanID.
4. ParentSpanID: An 8-byte (16-character hex) reference pointing to the SpanID of the invoking (parent) operation. For a Root Span (the initial span initiating a distributed trace), the ParentSpanID is empty or zeroed (0000000000000000 / null).
5. Start and End Timestamps: High-precision Unix epoch timestamps recorded at the instant the operation begins and concludes (typically recorded in nanoseconds or microseconds depending on host OS monotonic clock precision). A span's duration is strictly calculated as EndTime - StartTime.
6. SpanKind: An enumeration defining the relationship of the span to network and process boundaries.
7. InstrumentationScope: Identifies the library or package name and version that generated the span (e.g., name: "io.opentelemetry.contrib.mongodb", version: "1.22.0").
8. Attributes, Events, Links, and Status: Contextual metadata, zero-duration milestone annotations, cross-trace causal links, and execution outcome codes.

---

The SpanKind Enumeration

In a distributed system, an operation's architectural role determines how its latency and error rates affect service-level objectives (SLOs). OpenTelemetry captures this semantics through the SpanKind enum:

         [Caller Process]                             [Callee Process]
  +---------------------------+                +---------------------------+
  | SpanKind.CLIENT           |  HTTP Request  | SpanKind.SERVER           |
  | (Measures round-trip      | -------------> | (Measures server-side     |
  |  including network lag)   | <------------- |  execution processing)    |
  +---------------------------+  HTTP Response +---------------------------+

The specification defines five distinct SpanKind values:

1. SpanKind.SERVER

Represents the synchronous handling of an incoming request from an external client or upstream service (e.g., an HTTP server endpoint, a gRPC service method, or an incoming RPC). The span begins when the server receives the request payload and ends when the response is dispatched. A SERVER span is typically the child of a remote CLIENT span.

2. SpanKind.CLIENT

Represents a synchronous outgoing call to an external dependency or service (e.g., an outgoing HTTP client call, a gRPC client invocation, a Redis query, or an SQL database statement). The span starts when the client prepares the request and ends when the full response has been read. It measures the full round-trip duration, which includes client serialization, network latency, server processing, and deserialization.

3. SpanKind.PRODUCER

Represents the creation and publication of a message to an asynchronous messaging system or message broker (e.g., publishing a topic event to Apache Kafka, sending a message to RabbitMQ, or enqueueing an Amazon SQS task). The span encompasses the message serialization and transport handoff to the broker. Crucially, a PRODUCER span covers creating and sending the message; it does not wait for any consumer to process it.

4. SpanKind.CONSUMER

Represents the asynchronous retrieval and processing of a message from a message broker (e.g., polling an SQS queue, processing a Kafka partition record, or dequeuing a Celery task). The CONSUMER span begins when processing starts on the received message and ends when processing concludes. Because of asynchronous queuing, the CONSUMER span is decoupled in time from the originating PRODUCER span.

5. SpanKind.INTERNAL

Represents an internal operation completely confined within the local process boundary. This is the default kind when no SpanKind is explicitly specified. It is used for function executions, in-memory computations, algorithmic subroutines, batch loops, or local cryptographic operations.

SpanKind Comparison Matrix

---

SpanContext: The Distributed Boundary Bridge

A critical distinction is the architectural difference between a Span and a SpanContext:

• A Span is a complete, mutable, heavy SDK-internal object containing start and end timestamps, mutable attribute dictionaries, event arrays, link lists, status codes, and references to the parent span.
• A SpanContext is a compact, immutable, serializable packet of metadata containing only the minimal identifiers required to preserve distributed trace continuity across process and network boundaries.

When a service makes an outgoing network call to a downstream microservice, it does not transmit the full span. Instead, the OpenTelemetry propagator serializes only the SpanContext into wire protocol headers (such as the W3C traceparent header). The downstream service extracts this SpanContext and sets it as the parent context of its newly created SERVER span.

Structure of SpanContext

A compliant SpanContext contains four mandatory fields:

1. TraceID: 16 bytes (32 hex characters). Preserves the global trace identity.
2. SpanID: 8 bytes (16 hex characters). Represents the span that generated or forwarded this context (which becomes the ParentSpanID for downstream child spans).
3. TraceFlags: 1 byte (8 bits, represented as 2 hex characters). Bitwise flags controlling trace behavior. Bit 0 (0x01) is the Sampled flag. If set (01), the span was selected for recording and export by the head sampler; if unset (00), the span is not sampled. W3C Trace Context Level 2, which the OpenTelemetry specification follows, also defines bit 1 (0x02) as the Random flag, signalling that the trace ID's rightmost 7 bytes are random.
4. TraceState: An immutable list of opaque, comma-separated key-value pairs (e.g., congo=t61rcWkgMzE,rojo=00f067aa0ba902b7, supporting up to 32 key-value members). It is designed to carry vendor-specific routing metadata and legacy tracing system state across distributed boundaries without modifying the standard TraceID.

SpanContext Predicates

• IsValid: Returns true if both TraceID and SpanID are non-zero. A SpanContext with an empty/all-zero TraceID or SpanID is considered invalid.
• IsRemote: A boolean flag indicating whether the SpanContext was extracted from an incoming wire carrier (e.g., incoming HTTP headers) rather than being generated locally within the current process memory.

---

Span Links: Causal Graphs Beyond Parent-Child Trees

In standard distributed tracing, spans form hierarchical trees where each child span has exactly one parent span. While this model perfectly represents synchronous request-response interactions, it fails in several modern distributed computing patterns:

1. Batch Processing (Fan-In): A background worker pulls 100 individual messages from an Amazon SQS queue or Kafka partition. Each message was produced by a completely different user transaction in a distinct distributed trace. If the worker processes all 100 messages in a single bulk database transaction within one span, which of the 100 traces is the parent? A span cannot have 100 parents!
2. Scatter-Gather Workflows (Fan-Out Aggregations): A master worker initiates parallel subtasks across multiple pipelines and combines the results in an aggregation stage that completes long after the original dispatch traces ended.
3. Asynchronous Continuations: An event handler triggers follow-up actions minutes after the original HTTP request concluded and closed its root span.

Trace A (Order Checkout)   --> [Producer Span 1 Context]
                                      \ (Span Link 1)
Trace B (Payment Webhook)  --> [Producer Span 2 Context] ---> [Batch Consumer Span]
                                      / (Span Link 2)         (Kind: CONSUMER)
Trace C (Inventory Sync)   --> [Producer Span 3 Context]

The OpenTelemetry Solution: Span Links

To solve this without compromising the single-parent tree constraint, OpenTelemetry provides Span Links. A Span Link is a causal pointer from a newly created span to an external SpanContext. A span can have zero, one, or hundreds of Span Links.

Characteristics of Span Links

• Contains Target SpanContext: A link points to another span's SpanContext (which can belong to the same trace or a completely foreign trace).
• Optional Link Attributes: A link can carry its own key-value attributes providing context about the relationship (e.g., messaging.batch.message_id = "msg_9941", link.reason = "batch_consumer").
• Creation Timing: Supply links at span creation whenever the linked contexts are already known (for example tracer.Start(ctx, name, trace.WithLinks(links...))), because a head sampler can only consider what exists at creation. The current API also provides AddLink to attach a link after the span has started, for contexts discovered later.

Through Span Links, visualization backends can render graph edges linking the batch consumer span back to all 100 original customer traces, providing comprehensive causal debugging without corrupting latency calculations in the originating trees.