2.2 Resource Attributes & Entity Modeling

Resource Attributes & Entity Modeling

When troubleshooting an incident in a modern microservices architecture, inspecting an isolated span or metric is rarely sufficient on its own. For instance, knowing that an HTTP GET /checkout request suffered a 500 error is incomplete without knowing which service failed, what version was running, which physical host or pod processed the request, and in which cloud region the workload was deployed.

In OpenTelemetry, this critical context is provided by the Resource. A Resource is an immutable object that models the computing entity generating telemetry data. It captures static, environment-level identity and infrastructure metadata, ensuring every span, metric data point, and log record produced by an application is automatically contextualized without duplicating static metadata across every individual span.

---

Resource Attributes vs. Span & Metric Attributes

A common area of confusion is the distinction between Resource Attributes and Span/Metric Attributes:

Architectural Principle: Never attach static entity details (such as deployment.environment.name="production" or k8s.pod.name) as span attributes on each child span. Storing static attributes on spans causes substantial network bandwidth and storage bloat. Static metadata belongs strictly in the Resource.

---

The Primary Service Identity Attributes

The OpenTelemetry Resource specification establishes standard attributes for identifying services and workloads. Among these, four core attributes define a service's identity in APM platforms:

1. service.name (The Primary Identifier)

service.name is the single most critical attribute in OpenTelemetry. It defines the logical name of the application or service (such as "order-service", "checkout-worker", or "auth-api").

• APM backends, service maps, and error tracking systems rely on service.name to index and group telemetry.
• If an application initializes the OpenTelemetry SDK without specifying service.name, the specification requires the fallback "unknown_service:" followed by the process executable name (for example unknown_service:java), or plain "unknown_service" when the executable name is unavailable.
• Seeing "unknown_service" in an APM tool indicates that the service name was not configured.

2. service.namespace (Logical Grouping)

In enterprise environments with hundreds of services, service.namespace provides a logical grouping or organizational hierarchy (such as "billing", "ecommerce", or "fulfillment"). APM tools use this attribute to categorize services into functional domains.

3. service.instance.id (Unique Physical Instance)

While service.name identifies the logical software component, service.instance.id uniquely identifies the specific running process, container, or virtual machine instance (for example, a Kubernetes pod UID or a generated UUID: "62c95e1e-18e3-4d4b-9c3f-723a9b1c0982"). This allows SREs to isolate memory leaks, crash loops, or degraded latency to a specific faulty instance.

4. service.version (Deployment and Canary Tracking)

service.version captures the software release version (such as "2.4.1", "v1.0.0-rc3", or a git commit hash: "git:e4b2d1c"). This attribute is critical for:

• Canary analysis: Comparing error rates between release versions during automated progressive rollouts.
• Deployment tracking: Determining precisely which commit introduced a latency regression.

---

Resource Detectors

Rather than requiring engineers to manually code infrastructure properties into application binaries, OpenTelemetry provides Resource Detectors. Resource detectors are plugins or SDK modules that interrogate the local operating environment at startup to extract system and cloud metadata automatically.

Core resource detectors include:

• Host Detector: Inspects system calls to extract host.name, host.id, host.arch (e.g., "amd64", "arm64"), and os.type (e.g., "linux", "darwin").
• Process Detector: Interrogates runtime process information to capture process.pid, process.executable.name, process.command_line, process.runtime.name (e.g., "cpython", "openjdk"), and process.runtime.version.
• Container Detector: Inspects /proc/self/cgroup or container environment files to discover container.id and container.image.name.
• Kubernetes metadata: SDKs do not query the Kubernetes API by default. Pod attributes such as k8s.pod.name, k8s.namespace.name, k8s.node.name, and k8s.pod.uid usually arrive through Downward API environment variables (the OpenTelemetry Operator injects them into OTEL_RESOURCE_ATTRIBUTES) or are added later by the Collector's k8sattributes processor.
• Cloud Detectors:
  • AWS Detector: Queries the AWS Instance Metadata Service (IMDSv2) for EC2 metadata (cloud.provider="aws", cloud.region, cloud.availability_zone, host.id="i-0123..."), ECS task ARNs, or Lambda execution environments.
  • GCP Detector: Interrogates the Compute Engine metadata server to detect cloud.provider="gcp", cloud.account.id, cloud.availability_zone, host.id, and gcp.gce.instance.name.
  • Azure Detector: Queries the Azure Instance Metadata Service (IMDS) to discover cloud.provider="azure", cloud.region, host.id (the VM ID), and azure.vm.size.

---

Configuring Resources via Environment Variables

OpenTelemetry standardizes environment variables to configure resource metadata without modifying application code or rebuilding container images:

1. OTEL_SERVICE_NAME

A dedicated shortcut for declaring service.name:

export OTEL_SERVICE_NAME="order-service"

2. OTEL_RESOURCE_ATTRIBUTES

A comma-separated list of key=value pairs defining arbitrary resource attributes:

export OTEL_RESOURCE_ATTRIBUTES="service.name=order-service,service.version=1.4.2,deployment.environment.name=production,service.namespace=checkout"

Key rules for OTEL_RESOURCE_ATTRIBUTES:

• Pairs are separated by commas (,).
• Keys and values are separated by an equal sign (=).
• Values containing commas, whitespace, or special characters must be URL percent-encoded (e.g., %20 for space, %2C for comma).
• If OTEL_SERVICE_NAME is set, it takes precedence over any service.name defined within OTEL_RESOURCE_ATTRIBUTES.

---

Resource Merging and Precedence Rules

When an OpenTelemetry SDK initializes, multiple sources contribute resource attributes: default SDK attributes, one or more resource detectors, environment variables, and explicit application code. The SDK merges these sources into a single immutable Resource object.

[ Default SDK Attributes ]
          ↓ (merged into)
[ Auto-Detected Attributes (Host, Cloud, K8s) ]
          ↓ (merged into)
[ OTEL_RESOURCE_ATTRIBUTES Environment Variables ]
          ↓ (merged into)
[ OTEL_SERVICE_NAME Environment Variable ]
          ↓ (merged into)
[ Programmatic Resource Defined in Application Code ]

Precedence Hierarchy (Lowest to Highest)

1. SDK Built-in Defaults: Minimal telemetry metadata, such as telemetry.sdk.language, telemetry.sdk.name, and telemetry.sdk.version.
2. Resource Detectors: Infrastructure attributes discovered from host, container, or cloud metadata endpoints.
3. Environment Variables (OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME): External configuration applied at deployment or container runtime. In the standard SDK auto-configuration these values are merged after the detectors, so they win on a collision.
4. Explicit Application Code: Resource instances passed directly to the provider builder in source code. The specification states that user-provided resource information has higher priority than OTEL_RESOURCE_ATTRIBUTES.

When a key collision occurs during a merge (for example, if both the cloud detector and application code define host.name), the higher precedence source overwrites the lower precedence source.

Schema URLs in Resources (schema_url)

Every Resource can carry a schema_url referencing the version of semantic conventions governing its attributes (e.g., "https://opentelemetry.io/schemas/1.24.0"). When two resources are merged:

• If both resources have identical schema_urls, the merged resource retains that URL.
• If one resource has a schema_url and the other has an empty schema_url, the non-empty URL is preserved.
• If both resources carry non-empty but different schema_urls, the specification calls this a merging error and leaves the result implementation-specific, so keep schema URLs consistent across detectors and code.

---

Practical Configuration Examples

Python SDK Resource Configuration (Programmatic)

from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_VERSION
from opentelemetry.sdk.trace import TracerProvider

# Construct custom immutable resource
custom_resource = Resource.create({
    SERVICE_NAME: "payment-gateway",
    SERVICE_VERSION: "3.1.0",
    "deployment.environment.name": "production",
    "team.owner": "checkout-core",
})

# Initialize TracerProvider with merged resource
provider = TracerProvider(resource=custom_resource)

Collector Resource Processor Configuration (YAML)

In addition to SDK-side configuration, the OpenTelemetry Collector can enrich or modify resource attributes centrally using the resource processor:

processors:
  resource:
    attributes:
      - key: deployment.environment.name
        value: "production"
        action: upsert
      - key: k8s.cluster.name
        value: "us-east-prod-eks"
        action: insert
      - key: unwanted.debug.attr
        action: delete