12.3 OpenTelemetry Ingestion, OpenPipeline & Dynatrace Workflows Automation

Modern enterprise IT is characterized by the coexistence of vendor-neutral open-source standards and high-performance commercial observability platforms. The Cloud Native Computing Foundation (CNCF) OpenTelemetry (OTel) framework has emerged as the global standard for vendor-neutral telemetry instrumentation. Dynatrace provides native, first-class support for OpenTelemetry ingestion, combining open data collection with the analytical power of the Grail data lakehouse, OpenPipeline streaming data governance, and Dynatrace Workflows for closed-loop autonomous remediation.

---

OpenTelemetry Architecture & Native Ingestion Pathways

OpenTelemetry standardizes the collection of three primary telemetry signals: Traces, Metrics, and Logs. Dynatrace supports the standard OpenTelemetry Protocol (OTLP) over both OTLP/HTTP (binary Protobuf or JSON) and OTLP/gRPC.

+-----------------------------------------------------------------------------------------+
|                         OPENTELEMETRY INGESTION ARCHITECTURE                            |
+-----------------------------------------------------------------------------------------+
|                                                                                         |
|  [Custom App (OTel SDK)]   [Kubernetes Pod (OTel Collector)]   [AWS Lambda (OTel Layer)]|
|            \                           |                              /                 |
|             \                          |                             /                  |
|              +-------------------------+----------------------------+                   |
|                                        |                                                |
|                                        v (OTLP / gRPC or HTTP)                          |
|                     +--------------------------------------+                            |
|                     |       ENVIRONMENT ACTIVEGATE         |                            |
|                     |  • Ingress Proxy & Message Router    |                            |
|                     |  • Token Authentication Validation   |                            |
|                     +--------------------------------------+                            |
|                                        |                                                |
|                                        v (HTTPS / TLS 443)                              |
|                     +--------------------------------------+                            |
|                     |         DYNATRACE PLATFORM           |                            |
|                     |  +--------------------------------+  |                            |
|                     |  |         OPENPIPELINE           |  |                            |
|                     |  |  Filter -> Mask -> Route       |  |                            |
|                     |  +--------------------------------+  |                            |
|                     |  |       GRAIL LAKEHOUSE          |  |                            |
|                     |  |   Logs, Traces, Metrics        |  |                            |
|                     |  +--------------------------------+  |                            |
|                     |  |       DAVIS AI ENGINE          |  |                            |
|                     |  +--------------------------------+  |                            |
|                     |  |     DYNATRACE WORKFLOWS        |  |                            |
|                     +--+--------------------------------+--+                            |
+-----------------------------------------------------------------------------------------+

Direct Ingestion Endpoints and API Tokens

Telemetry can be transmitted directly to Dynatrace SaaS or routed through a local Environment ActiveGate acting as a high-performance proxy. ActiveGate endpoints alleviate WAN bandwidth by bundling connections and avoiding direct public cloud access from private subnets.

• Traces Endpoint: https://{your-activegate-or-environment}/api/v2/otlp/v1/traces
• Metrics Endpoint: https://{your-activegate-or-environment}/api/v2/otlp/v1/metrics
• Logs Endpoint: https://{your-activegate-or-environment}/api/v2/otlp/v1/logs

Authentication requires a Dynatrace API Token configured with specific, least-privilege token scopes:

• openTelemetryTrace.ingest — Authorizes ingestion of OTLP traces.
• metrics.ingest — Authorizes ingestion of OTLP and custom metrics.
• logs.ingest — Authorizes ingestion of OTLP and structured log records.

Hybrid Distributed Tracing: OneAgent + OpenTelemetry Interoperability

A critical requirement in large enterprises is the ability to combine OneAgent automatic bytecode instrumentation with OpenTelemetry custom spans without creating disjointed traces. Dynatrace accomplishes this via native support for the W3C Trace Context specification (traceparent and tracestate HTTP headers):

1. When a OneAgent-monitored Java service calls a downstream Go microservice instrumented with OpenTelemetry SDK, OneAgent injects standard W3C traceparent headers into the outgoing HTTP request.
2. The Go microservice extracts the W3C context, starts a child span, and transmits the span via OTLP to Dynatrace.
3. Dynatrace automatically correlates the OpenTelemetry span with the parent transaction using the shared 128-bit trace_id and 64-bit parent_span_id.
4. The result is a single, uninterrupted PurePath distributed trace spanning both proprietary OneAgent sensors and open-source OpenTelemetry spans.

---

OpenPipeline: Ingress Stream Processing and Data Governance

As telemetry volumes explode, organizations face mounting storage costs and compliance challenges from unmasked Personally Identifiable Information (PII). OpenPipeline is Dynatrace's high-throughput stream processing engine that operates directly at the data ingress boundary—before telemetry is committed to long-term Grail storage or evaluated by Davis AI.

Core Stages of OpenPipeline Execution

OpenPipeline intercepts incoming telemetry streams (logs, metrics, business events, traces) and processes records sequentially through defined pipeline stages:

Incoming Stream ---> [ ROUTING ] ---> [ FILTERING ] ---> [ TRANSFORMATION ] ---> [ MASKING ] ---> [ STORAGE (GRAIL) ]

1. Routing: Inspects metadata (e.g., source host group, Kubernetes namespace, cloud account, log severity) and routes the record to dedicated processing pipelines or storage tables.
2. Filtering (Drop Rules): Discards low-value, high-volume telemetry at line rate. For example, dropping high-frequency HTTP 200 OK health-check access logs or verbose DEBUG log streams from non-production containers. Dropped records do not incur Grail storage costs or consume Davis Data Units (DDUs).
3. Transformation & Normalization: Parses unstructured text into structured JSON fields, normalizes non-standard attribute keys to match OpenTelemetry semantic conventions, and enriches records with contextual metadata (e.g., mapping IP addresses to corporate data centers).
4. Masking & Security Compliance: Identifies sensitive data patterns using regular expressions—such as credit card numbers (PCI-DSS), Social Security numbers, passwords, and authorization tokens—and permanently redacts or hashes them before persistence. Because masking occurs at stream ingress, sensitive data is never written to disk, guaranteeing strict GDPR, HIPAA, and SOC 2 compliance.

---

Dynatrace Workflows: Event-Driven Automation

Observability without automated action leaves organizations vulnerable to slow manual incident response. Dynatrace Workflows is an enterprise automation and orchestration engine built directly into the platform, providing visual canvas orchestration, low-code action blocks, and custom TypeScript/JavaScript execution tasks.

Workflow Trigger Mechanisms

Workflows are event-driven and can be activated through multiple trigger types:

• Davis Problem Events: Fires automatically when Davis AI detects an anomaly, when a problem transitions states (e.g., opened, updated with new root cause, or resolved), or when specific severity thresholds are met.
• Schedules (Time-Based / Cron): Executes periodically at defined intervals (e.g., running hourly capacity audits or daily compliance checks).
• DQL Threshold Monitors: Evaluates a Dynatrace Query Language (DQL) query against Grail storage and triggers when query output breaches a predefined threshold.
• On-Demand & Webhooks: Triggered manually by operators via the Dynatrace console or invoked externally by CI/CD pipelines and external webhooks.

Integrated Ecosystem Connectors

Dynatrace Workflows feature deep native connectors across modern IT enterprise ecosystems:

• Incident Management & ChatOps: ServiceNow (creating/updating incidents and change requests), Jira, PagerDuty, Slack, and Microsoft Teams.
• Configuration Management & Infrastructure: Ansible Automation Platform (launching automation controller job templates), Terraform Cloud, AWS Systems Manager, and Kubernetes (executing pod restarts or scaling deployments).
• Custom Logic & Cloud Functions: Executing custom JavaScript/TypeScript code within secure platform sandboxes, invoking AWS Lambda, Azure Functions, or triggering generic HTTP/REST webhooks.

---

Closed-Loop Autonomous Remediation Architecture

A primary goal of Site Reliability Engineering is achieving closed-loop self-healing—detecting an incident, identifying the root cause, applying remediation, and verifying recovery without human intervention.

+-----------------------------------------------------------------------------------------+
|                     CLOSED-LOOP AUTO-REMEDIATION WORKFLOW LIFECYCLE                     |
+-----------------------------------------------------------------------------------------+
|                                                                                         |
|  1. DAVIS AI PROBLEM DETECTION                                                          |
|     • Davis detects response time degradation on 'Payment-Service'.                     |
|     • Root cause identified: Memory leak causing GC thrashing on Container Pod A.      |
|                                                                                         |
|  2. WORKFLOW TRIGGER ACTIVATION                                                         |
|     • Event trigger matches problem category: 'RESOURCE_SATURATION'.                    |
|     • Workflow initializes with execution context (affected Entity ID, Problem ID).    |
|                                                                                         |
|  3. DIAGNOSTIC COLLECTION & ENRICHMENT                                                  |
|     • Task executes DQL query to extract heap dump summaries and container logs.       |
|     • Creates a high-priority incident in ServiceNow containing diagnostic telemetry.   |
|                                                                                         |
|  4. AUTOMATED REMEDIATION ACTION                                                        |
|     • Workflow invokes an Ansible Automation controller job via REST API.              |
|     • Ansible triggers a graceful rolling restart of the saturated microservice pod.    |
|                                                                                         |
|  5. VERIFICATION & AUDIT CLOSURE                                                        |
|     • Workflow pauses (sleep timer) and queries Davis problem status via DQL.           |
|     • If Davis marks the problem RESOLVED, Workflow posts resolution comment to Jira,   |
|       notifies Slack #sre-ops, and closes the ServiceNow ticket autonomously.          |
+-----------------------------------------------------------------------------------------+

---

Technology Comparison Matrix