8.4 Zero-Code Agents: Configuration, Kubernetes Injection & Troubleshooting
Key Takeaways
An OpenTelemetry agent is a zero-code package attached to a process (Java agent JAR, Python opentelemetry-instrument, Node.js auto-instrumentations register module, .NET automatic instrumentation, or eBPF tools such as OBI) that bundles the SDK, instrumentations, and exporters.
Agents read the standard OTEL_* settings, and each language adds switches to disable individual instrumentations, such as Java's OTEL_INSTRUMENTATION_[NAME]_ENABLED and Python's OTEL_PYTHON_DISABLED_INSTRUMENTATIONS.
The OpenTelemetry Operator injects agents through an Instrumentation resource plus inject-java, inject-python, inject-nodejs, inject-dotnet, or inject-go pod annotations, and only at pod creation.
Go injection uses a privileged eBPF sidecar and requires the target executable path through the otel-go-auto-target-exe annotation or the Instrumentation resource.
When an agent is attached, application code should use only the OpenTelemetry API for custom spans; building a second SDK in code causes duplicate pipelines.
8.4 Zero-Code Agents: Configuration, Kubernetes Injection & Troubleshooting
Quick Answer: In the API and SDK domain, an agent is a zero-code instrumentation package attached to a running process: the Java agent JAR, Python's
opentelemetry-instrumentwrapper, Node.js'sauto-instrumentations-node/registermodule, the .NET automatic instrumentation, or an eBPF instrumenter such as OBI. An agent bundles the SDK, instrumentation libraries, and exporters, and it reads the sameOTEL_*settings as any SDK. In Kubernetes, the OpenTelemetry Operator injects agents through anInstrumentationresource and pod annotations. Do not confuse this with the Collector agent deployment pattern in Chapter 12.
Section 2.3 compared zero-code and code-based instrumentation as a strategy. This section covers the operational side of the Agents competency: how each agent is attached and configured, how to switch individual instrumentations on and off, how the Operator injects agents, and how to troubleshoot an agent that produces no data.
What an Agent Contains
An agent adds three things to a process that has no OpenTelemetry code of its own:
- The API and an SDK, configured automatically from environment variables, system properties, or (where supported) a declarative configuration file.
- Instrumentation libraries for supported frameworks and clients (HTTP servers and clients, database drivers, messaging clients, gRPC).
- Exporters, usually OTLP, plus the standard propagators (
tracecontext,baggageby default).
Because the agent uses the standard configuration, the same variables work everywhere: OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_TRACES_SAMPLER, and OTEL_PROPAGATORS.
Attaching and Configuring Agents by Language
| Runtime | How the agent is attached | Selecting instrumentations | Notes |
|---|---|---|---|
| Java | -javaagent:/path/opentelemetry-javaagent.jar, or the same flag in JAVA_TOOL_OPTIONS | OTEL_INSTRUMENTATION_[NAME]_ENABLED=false turns one library off; OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=false turns all off so you can enable a few; OTEL_JAVAAGENT_ENABLED=false disables the agent | Configuration sources, highest priority first: system properties, environment variables, a properties file (otel.javaagent.configuration-file), then customizer SPIs. Extensions load from otel.javaagent.extensions. |
| Python | pip install opentelemetry-distro opentelemetry-exporter-otlp, then opentelemetry-bootstrap -a install, then run opentelemetry-instrument python app.py | OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=redis,kafka | opentelemetry-bootstrap installs instrumentation packages only for libraries already present in the environment; the instrumentors monkey-patch those libraries at startup. |
| Node.js | node --require @opentelemetry/auto-instrumentations-node/register app.js, or the same flag in NODE_OPTIONS | OTEL_NODE_ENABLED_INSTRUMENTATIONS="http,express" or OTEL_NODE_DISABLED_INSTRUMENTATIONS | OTEL_NODE_RESOURCE_DETECTORS limits which resource detectors run. |
| .NET | Run the installer script, then source instrument.sh, which sets the CLR profiler and startup-hook environment variables | OTEL_DOTNET_AUTO_* settings | Works with .NET and .NET Framework 4.6.2 or later. |
| Go and others (eBPF) | Go auto-instrumentation or OpenTelemetry eBPF Instrumentation (OBI) runs beside the process with elevated Linux privileges | Configured through the eBPF tool | OBI observes Java, .NET, Go, Python, Ruby, Node.js, C, C++, and Rust workloads and produces spans and RED metrics without touching the code. |
Adding Custom Telemetry on Top of an Agent
The agent does not stop you from writing code-based instrumentation. For Java, add the opentelemetry-api dependency and create spans or use the @WithSpan annotation; the agent connects those API calls to its own SDK. The common mistake is to also build and register a second SDK in application code, which produces duplicate or conflicting pipelines. When an agent is present, application code should only use the API.
Kubernetes Injection with the OpenTelemetry Operator
The Operator automates agent attachment in Kubernetes:
- Create an
Instrumentationcustom resource that holds shared settings: the exporter endpoint, propagators, sampler, and per-language images. - Annotate a namespace or pod template with the language to inject, for example
instrumentation.opentelemetry.io/inject-java: "true". The other annotations areinject-python,inject-nodejs,inject-dotnet,inject-go, andinject-sdk(which only sets SDK environment variables). - When a pod is created, the Operator's mutating admission webhook adds an init container that copies the agent into the pod and sets the environment variables (
JAVA_TOOL_OPTIONS,OTEL_EXPORTER_OTLP_ENDPOINT,OTEL_SERVICE_NAME, resource attributes, and so on).
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: demo-instrumentation
spec:
exporter:
endpoint: http://demo-collector:4318
propagators:
- tracecontext
- baggage
sampler:
type: parentbased_traceidratio
argument: "1"
Operational points to remember:
- Injection happens only at pod creation. Annotating a Deployment that is already running does nothing until its pods are recreated (for example with a rollout restart).
- Match the protocol to the port. The Operator's .NET instrumentation exports OTLP over
http/protobufby default, so its endpoint must be the Collector's HTTP port (4318); check each language's default before pointing it at 4317. - Go is different. Go injection adds an eBPF sidecar that needs elevated privileges (
privileged: true, running as root) and the target binary path, supplied with theinstrumentation.opentelemetry.io/otel-go-auto-target-exeannotation or in theInstrumentationresource; without it, injection is aborted.
Troubleshooting an Agent That Produces No Data
| Symptom | Likely cause | Check or fix |
|---|---|---|
Service appears as unknown_service:java | OTEL_SERVICE_NAME not set | Set it in the pod spec or the Instrumentation resource |
| No spans from a Python framework | Instrumentation package missing | Re-run opentelemetry-bootstrap -a install after installing the framework |
| Spans exported nowhere | Wrong endpoint or protocol/port mismatch | Temporarily set OTEL_TRACES_EXPORTER=console to print spans locally; confirm 4317 is gRPC and 4318 is HTTP |
| Annotated pods show no agent | Pods created before the annotation, or no matching Instrumentation resource | Restart the workload; confirm the resource exists in the namespace |
| Too much overhead or noisy spans | Every supported library is instrumented | Disable specific instrumentations (Java OTEL_INSTRUMENTATION_[NAME]_ENABLED=false, Python OTEL_PYTHON_DISABLED_INSTRUMENTATIONS, Node.js OTEL_NODE_DISABLED_INSTRUMENTATIONS) |
| Duplicate spans | Both an agent and a manually built SDK are active | Keep only the agent's SDK; application code uses the API alone |
A platform team creates an Instrumentation resource and adds instrumentation.opentelemetry.io/inject-java: "true" to the pod template of a Deployment whose pods have been running for a week. An hour later, those pods still send no telemetry. What is the most likely explanation?
The Java agent only supports pods that also run a Collector sidecar
The annotation must be placed on the Collector rather than on the application pods
Java injection requires the otel-go-auto-target-exe annotation to point at the JAR file
The Operator injects the agent through a mutating webhook when a pod is created, so the existing pods must be recreated (for example with a rollout restart)
A Java service runs with the OpenTelemetry Java agent. JDBC spans are too noisy, but the team wants to keep HTTP and messaging instrumentation. Which configuration does this?
Set OTEL_INSTRUMENTATION_JDBC_ENABLED=false, which suppresses only the JDBC instrumentation
Set OTEL_JAVAAGENT_ENABLED=false, which disables the agent
Set OTEL_TRACES_SAMPLER=always_off so that JDBC spans are never sampled
Remove -javaagent from JAVA_TOOL_OPTIONS and add JDBC tracing by hand
A Python service is started with opentelemetry-instrument. HTTP client spans appear, but after the team later adds Flask to the project, no Flask server spans are produced. What should they do?
Switch the exporter protocol from http/protobuf to grpc
Run opentelemetry-bootstrap -a install again so that the matching Flask instrumentation package is installed, then restart the service
Set OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=flask to force Flask instrumentation on
Replace opentelemetry-instrument with a manually configured TracerProvider
Sections you finish are checked off in the contents.