7.3 Event Ingestion, Event Types & Event Hierarchy in Problem Evolution

In Dynatrace, performance monitoring transcends static time-series graphs through the ingestion and analysis of discrete Telemetry Events. Events capture state transitions, threshold violations, configuration deployments, and operational failures occurring across monitored entities. Rather than treating all events with equal urgency, Dynatrace applies a deterministic Event Severity Hierarchy.

This hierarchy dictates how Davis opens problem tickets, how problem severity levels escalate during cascading outages, and how external notifications are dispatched. Furthermore, event correlation enables modern CI/CD pipelines to feed deployment metadata directly into Dynatrace, empowering Davis to connect software releases directly to performance regressions.

Mastering event ingestion pathways, the severity hierarchy, and problem card evolution is essential for the Dynatrace Certified Associate exam.

---

Telemetry Events: The Building Blocks of Problem Intelligence

An Event in Dynatrace represents an instantaneous occurrence or a time-bounded state change on a specific monitored entity (such as a Host, Process Group, Service, or Application).

Ingestion Pathways

Events flow into Dynatrace from multiple instrumentation vectors:

1. Native OneAgent Sensors: The OneAgent continuously monitors operating system metrics, process lifecycles, and network sockets, automatically generating events for host reboots, process crashes, CPU saturation, out-of-memory kills, and network packet drops.
2. ActiveGate Cloud & Virtualization Plugins: ActiveGates polling VMware vCenter, AWS CloudWatch, Azure Monitor, or GCP generate infrastructure events (e.g., hypervisor host migrations, AWS RDS failovers, Azure VM auto-scaling).
3. Dynatrace Events API (v2): Modern DevOps toolchains push external events programmatically using the REST endpoint POST /api/v2/events. This allows CI/CD tools (Jenkins, GitLab CI, GitHub Actions, ArgoCD) to annotate Dynatrace entities with deployment, configuration change, or remediation metadata.
4. Synthetic Monitors: Browser and HTTP synthetic monitors generate availability and performance events when multi-location validation tests fail.

---

The Dynatrace Event Severity Hierarchy

Dynatrace organizes all events into five primary severity categories. This hierarchy is strictly enforced: higher-severity events take precedence over lower-severity events when Davis determines overall problem severity.

+---------------------------------------------------------------------------------------------------+
|                             DYNATRACE EVENT SEVERITY HIERARCHY                                    |
+---------------------------------------------------------------------------------------------------+
|  1. AVAILABILITY (Highest Severity)                                                               |
|     • Complete outage of service or infrastructure (Process crash, Host reboot, HTTP 503).        |
|     • Opens problem immediately with P1 / Critical priority.                                      |
|                                                │                                                  |
|                                                ▼                                                  |
|  2. ERROR                                                                                         |
|     • Spike in failed requests or unhandled exceptions (HTTP 5xx, database transaction rollback). |
|     • Opens problem if baseline failure rate thresholds are breached.                             |
|                                                │                                                  |
|                                                ▼                                                  |
|  3. PERFORMANCE                                                                                   |
|     • Response time degradation, high queue latency, slow database queries.                       |
|     • Opens problem if dynamic baseline or static response time threshold is breached.            |
|                                                │                                                  |
|                                                ▼                                                  |
|  4. RESOURCE_CONTENTION (RESOURCE)                                                                |
|     • Infrastructure exhaustion (Disk full, Memory saturation, CPU starvation).                   |
|     • Opens problem or merges into active incident as the underlying root cause.                  |
|                                                │                                                  |
|                                                ▼                                                  |
|  5. CUSTOM_ALERT                                                                                  |
|     • User-configured metric threshold violations (e.g., custom metric > 1,000).                  |
|                                                │                                                  |
|                                                ▼                                                  |
|  6. INFO / CONFIGURATION / DEPLOYMENT (Lowest Severity - Contextual Only)                         |
|     • CI/CD deployments, configuration changes, feature flag toggles, planned restarts.          |
|     • NEVER opens a problem ticket on its own; correlated by Davis as Root Cause Context!         |
+---------------------------------------------------------------------------------------------------+

Comprehensive Event Hierarchy Analysis

Exam Key Point: The DCA exam heavily tests the behavior of INFO (Configuration / Deployment) events. Remember this cardinal rule: INFO events NEVER open a problem ticket and NEVER trigger an alert notification on their own. Instead, Davis uses them as contextual evidence to determine whether a recent release or configuration update caused a subsequent performance or error event.

---

The Special Role of INFO & Deployment Events

In modern GitOps and Continuous Deployment workflows, software is released multiple times per day. When an outage occurs minutes after a canary rollout, connecting the failure to the release is the most critical step in mean time to resolution (MTTR).

Structure of the Events API v2 Payload

DevOps pipelines deliver deployment metadata to Dynatrace by executing an HTTP POST to /api/v2/events. A standard deployment event payload includes:

{
  "eventType": "CUSTOM_INFO",
  "title": "Deployment of Order Service v2.4.1",
  "entitySelector": "type(SERVICE),tag(service:order-service)",
  "properties": {
    "remediation": "https://argocd.internal/rollback/order-svc",
    "git.commit": "9f8d32b",
    "ci.buildUrl": "https://jenkins.internal/job/order-svc/104",
    "approver": "release-lead@company.com"
  }
}

How Davis Correlates Deployments to Outages

When Davis opens a Problem ticket for a downstream service degradation (e.g., HTTP 500 errors), it queries the Smartscape topology graph for any events occurring in temporal proximity to the incident. If an INFO deployment event was registered on that service or its underlying process group within the preceding window, Davis highlights the deployment directly in the Root Cause section of the Problem Card as the probable catalyst for the degradation.

---

The Problem Evolution Timeline & Anatomy of a Problem Card

When an incident unfolds across a microservice landscape, it rarely begins and ends with a single static error. Dynatrace visualizes the progression of complex failures through Problem Evolution.

+---------------------------------------------------------------------------------------------------+
|                                 ANATOMY OF A DYNATRACE PROBLEM CARD                              |
+---------------------------------------------------------------------------------------------------+
| PROBLEM #1842: High Failure Rate on 'Order Service'                                               |
| Status: OPEN | Severity: ERROR | Duration: 14 mins | Impact: 2 Applications, 4 Services        |
+---------------------------------------------------------------------------------------------------+
| [1. BUSINESS IMPACT SECTION]                                                                      |
|   • Web Application: 1,420 Real User sessions impacted.                                           |
|   • Mobile App (iOS/Android): Cart checkout failures detected.                                    |
+---------------------------------------------------------------------------------------------------+
| [2. VISUAL PROBLEM EVOLUTION TIMELINE]                                                            |
|   14:00 ─── [INFO] Deployment event: 'Order Service v2.4.1' registered via Events API             |
|   14:02 ─── [RESOURCE] Disk I/O Saturation event on host 'db-prod-02'                             |
|   14:04 ─── [PERFORMANCE] Response time degradation on 'Database Query: selectCart'                |
|   14:05 ─── [ERROR] Failure rate spike (84% HTTP 500) on 'Order Service' <--- Problem Opens       |
+---------------------------------------------------------------------------------------------------+
| [3. ROOT CAUSE SECTION]                                                                           |
|   • Identified Root Cause: Disk I/O Saturation on Host 'db-prod-02'                               |
|   • Contributing Change: Deployment 'v2.4.1' on Service 'Order Service'                           |
+---------------------------------------------------------------------------------------------------+

Problem Severity Escalation Rules

As an incident evolves, its overall severity dynamically mirrors the highest active event severity involved in the cascade:

1. If an incident begins with a PERFORMANCE degradation (Response time slowdown), the Problem Card is assigned PERFORMANCE severity.
2. If downstream services begin rejecting requests and throwing HTTP 500 errors, Davis merges the new ERROR event into the problem ticket, automatically escalating the problem severity to ERROR.
3. If a key process subsequently crashes or a synthetic monitor fails completely, an AVAILABILITY event is attached, immediately escalating the problem severity to AVAILABILITY.
4. When the crashed process restarts, the problem severity de-escalates back to the next highest active event until all events resolve.

Problem Closure Requirements

A Dynatrace Problem ticket never closes prematurely while contributing causal events remain active. Davis requires that:

• All active AVAILABILITY, ERROR, PERFORMANCE, and RESOURCE events have ended.
• Multi-dimensional metric baselines (response time, failure rate, traffic) have normalized within acceptable dynamic tolerance bands.

---

Maintenance Windows & Alert Suppression Strategies

During planned maintenance, rolling kernel upgrades, or disaster recovery drills, infrastructure components will reboot and services will temporarily shut down. Without proper controls, these planned actions trigger catastrophic alert storms.

Dynatrace manages planned downtime through Maintenance Windows, which can be configured with one of three distinct operational modes:

Exam Key Point: Notice that in ALL THREE maintenance window modes, Metric Collection remains fully active. OneAgent never stops collecting telemetry during a maintenance window; only problem detection and notification routing are adjusted.