8.3 Problem Notification Integrations: Webhooks, Slack, ServiceNow & ITSM
Key Takeaways
- Dynatrace problem notification integrations decouple alerting logic from delivery channels, supporting native ITSM platforms, collaboration tools, and custom webhooks.
- Notification channels follow a strict lifecycle, dispatching OPEN notifications upon initial alert threshold satisfaction, UPDATE notifications upon severity/impact changes, and RESOLVED notifications upon problem closure.
- Custom webhooks leverage dynamic placeholders (such as {ProblemID}, {PID}, {ProblemTitle}, {ProblemSeverity}, and {RootCauseEntity}) to construct arbitrary JSON or XML payloads for external event buses.
- When integrating with on-premises ITSM systems behind corporate firewalls, webhook notifications can be securely routed through internal Environment ActiveGates.
- Maintenance windows provide three granular suppression modes: detect problems and alert, detect problems but suppress alerts, or disable problem detection entirely.
8.3 Problem Notification Integrations: Webhooks, Slack, ServiceNow & ITSM
Once Davis AI detects an incident and an Alerting Profile validates that the severity, duration, and management zone criteria have been satisfied, Dynatrace dispatches actionable notifications to external IT operations systems. In enterprise environments, observability does not live in an isolated silo; it integrates directly into IT Service Management (ITSM) workflows, automated incident ticketing systems, ChatOps collaboration channels, and automated remediation engines.
Dynatrace provides out-of-the-box native integrations for industry-standard tools including ServiceNow, Jira, Slack, Microsoft Teams, PagerDuty, and Opsgenie, alongside a highly flexible Custom Webhook integration framework. Understanding the notification lifecycle, the dynamic payload templating engine, secure on-premises ActiveGate execution paths, and maintenance window suppression modes is vital for the Dynatrace Certified Associate examination.
The Three-Phase Problem Notification Lifecycle
Unlike simplistic alert senders that transmit a single unformatted email when a threshold is breached, Dynatrace maintains a stateful, bi-directional relationship with external notification endpoints throughout the entire lifetime of an incident.
A problem notification channel processes three distinct lifecycle events:
- OPEN Notification: Dispatched when a problem's elapsed duration satisfies the delay filter configured in the attached Alerting Profile. This initial payload transmits the problem ID, title, severity, affected entities, and the identified root cause, prompting the target system to create an incident ticket or page an on-call engineer.
- UPDATE Notification: Dispatched whenever Davis detects significant structural changes during an active problem. If the blast radius widens (e.g., additional upstream microservices begin throwing errors), if the overall severity escalates (e.g., a Performance problem worsens into an Availability outage), or if Davis identifies a deeper root cause, an UPDATE notification is dispatched. This ensures external tickets maintain synchronized, up-to-the-minute situational awareness.
- RESOLVED Notification: Dispatched automatically when Davis determines that all contributing metric events have returned to their normal dynamic baselines for the required sliding evaluation window. The resolution payload instructs the external ITSM tool to automatically close the incident ticket or mark the PagerDuty alert resolved.
+---------------------------------------------------------------------------------------------------+
| NOTIFICATION LIFECYCLE STATE TRANSITIONS |
+---------------------------------------------------------------------------------------------------+
| [DAVIS DETECTS ANOMALY] |
| │ |
| ▼ (Delay timer satisfies Alerting Profile threshold, e.g. 5 minutes) |
| [STATE: OPEN] ───────> Dispatches OPEN Webhook / Creates ServiceNow Incident |
| │ |
| ▼ (Cascade causes 3 new services to fail or severity upgrades) |
| [STATE: UPDATE] ─────> Dispatches UPDATE Webhook / Adds work note & updates ticket priority |
| │ |
| ▼ (Remediation successful; metrics normalize across sliding window) |
| [STATE: RESOLVED] ───> Dispatches RESOLVED Webhook / Automatically resolves ServiceNow Incident |
+---------------------------------------------------------------------------------------------------+
Native Enterprise Integrations: ServiceNow, Jira & ChatOps
Dynatrace provides deeply integrated connectors tailored to enterprise operational workflows:
ServiceNow Integration
The Dynatrace ServiceNow integration connects Davis AI with the ServiceNow ITOM (IT Operations Management) and ITSM (IT Service Management) suites:
- CMDB Synchronization: Monitored hosts, services, and applications mapped by Smartscape are synchronized directly into the ServiceNow Configuration Management Database (CMDB), establishing real-time configuration item (CI) accuracy.
- Automated Incident Creation: When a problem opens, Dynatrace creates an Incident ticket in ServiceNow. Dynatrace automatically maps the problem's severity (Availability, Error, Performance) and impact level (Application, Service, Infrastructure) to the ServiceNow Urgency and Impact fields to calculate the ticket's Priority.
- Two-Way Synchronization: When engineers enter comments in Dynatrace, they synchronize to the ServiceNow ticket work notes. When Davis automatically resolves the problem, the ServiceNow Incident state transitions to Resolved, attaching the full Davis Root Cause summary link.
Jira & Jira Service Management
- Dynatrace automatically generates Jira issues (e.g., Bug, Task, or Incident) in specified projects whenever problems occur.
- Custom field mappings allow mapping Dynatrace environment URLs, affected services, and tags into specific Jira custom fields.
- When Davis marks the problem resolved, Dynatrace automatically transitions the Jira issue along its configured workflow to
ClosedorResolved.
ChatOps & On-Call Paging (Slack, Teams, PagerDuty)
- Slack & Microsoft Teams: Problem notifications post rich, interactive Adaptive Cards into dedicated incident channels. Cards display color-coded severity banners, problem duration, affected entities, and direct deep links to the Dynatrace problem overview.
- PagerDuty & Opsgenie: Dynatrace integrates directly with paging services to trigger on-call rotation escalations for high-severity problems. When Davis resolves the problem, an automatic resolution event is sent, stopping on-call alerts without manual intervention.
Custom Webhook Integrations and Dynamic Placeholders
For proprietary ticketing platforms, event buses (e.g., Apache Kafka, RabbitMQ), or serverless automation runtimes (e.g., AWS Lambda, Azure Functions), Dynatrace provides the Custom Webhook integration. Webhooks dispatch HTTP POST or PUT requests containing arbitrary JSON, XML, or form-urlencoded payloads.
Dynamic Placeholders Reference
To populate dynamic problem attributes into custom payloads, Dynatrace provides a comprehensive set of dynamic placeholders. These placeholders are replaced with real-time incident data at dispatch time:
| Dynamic Placeholder | Output Format / Example | Description & Operational Use |
|---|---|---|
{ProblemID} or {PID} | P-2410294 | The unique, immutable internal identifier for the problem. Essential as a correlation key across OPEN, UPDATE, and RESOLVED payloads. |
{ProblemTitle} | Failure rate increase on OrderService | The concise, Davis-generated summary describing the detected anomaly and primary symptom. |
{ProblemSeverity} | AVAILABILITY, ERROR, PERFORMANCE, RESOURCE_CONTENTION, CUSTOM_ALERT | The severity category of the problem. Used by downstream event brokers to filter routing queues. |
{ProblemImpact} | APPLICATION, SERVICES, INFRASTRUCTURE | Identifies the highest level in the architecture affected by the incident. |
{State} | OPEN or RESOLVED | The current lifecycle state of the problem. Crucial for downstream systems to differentiate incident creation from resolution. |
{RootCauseEntity} | Host: linux-db-prod-01.corp | The human-readable name of the specific component pinpointed by Davis as the originating fault. |
{ImpactedEntities} | JSON array or comma-separated list of affected entity names | The list of all upstream and downstream entities suffering customer or operational impact. |
{ProblemURL} | https://env.live.dynatrace.com/#problem;pid=P-2410294 | A direct deep-link URL opening the specific Problem Card in the Dynatrace web UI. |
{ProblemDetailsText} | Plain text multi-line narrative summary | Complete textual breakdown of all symptoms, root cause evidence, and metric deviations. |
{Tags} | Environment:Production, Tier:Backend, Owner:Payments | Comma-separated list of all tags attached to the root cause and impacted entities. |
Example Custom JSON Webhook Payload
{
"incident_id": "{PID}",
"event_type": "{State}",
"severity": "{ProblemSeverity}",
"title": "{ProblemTitle}",
"impact_level": "{ProblemImpact}",
"root_cause": "{RootCauseEntity}",
"impacted_components": "{ImpactedEntities}",
"console_url": "{ProblemURL}",
"tags": "{Tags}"
}
Exam Key Point: The placeholder
{PID}(or{ProblemID}) is the only guaranteed unique, persistent identifier that remains constant across OPEN, UPDATE, and RESOLVED notifications for the same incident. Downstream systems must use{PID}to correlate updates and closures to the initial incident record.
Network Architecture: Routing Webhooks via ActiveGate
In enterprise environments, IT systems frequently reside in private, secure on-premises data centers or isolated Virtual Private Clouds (VPCs) with no direct ingress from the public internet. If a Dynatrace SaaS tenant attempts to execute an HTTP POST directly to an internal ServiceNow instance (https://servicenow.corp.internal/api), the request will fail with an HTTP network timeout.
Dynatrace resolves this network boundary challenge using the Environment ActiveGate:
+---------------------------------------------------------------------------------------------------+
| ACTIVEGATE WEBHOOK ROUTING TOPOLOGY |
+---------------------------------------------------------------------------------------------------+
| [Dynatrace SaaS Cluster] (Public Cloud) |
| │ |
| │ 1. Outbound TLS polling / message dispatch (Port 443) |
| ▼ |
| ========================= [Corporate Firewall Boundary] ======================================== |
| │ |
| ▼ |
| [Environment ActiveGate] (Internal Corporate DMZ / Subnet) |
| │ |
| │ 2. Local HTTP/HTTPS POST Execution (Internal DNS: https://servicenow.corp.internal) |
| ▼ |
| [Internal ServiceNow / On-Prem ITSM System] (No Public Internet Ingress Required) |
+---------------------------------------------------------------------------------------------------+
Configuration Mechanics
In the problem notification configuration screen, administrators can toggle the "Execute on ActiveGate" setting and select an Environment ActiveGate cluster installed within their internal network. When triggered, the Dynatrace SaaS cluster routes the notification payload internally to the selected ActiveGate over existing secure outbound communication tunnels. The ActiveGate then executes the HTTP request locally within the private subnet, eliminating the need to expose internal corporate APIs to the public internet.
Maintenance Windows and Alert Suppression Mechanics
Routine enterprise operations require periodic maintenance: operating system patching, database schema migrations, hardware firmware upgrades, and Kubernetes node recycling. During these planned windows, components intentionally restart, triggering temporary outages and latency spikes.
Without safeguards, routine maintenance would trigger hundreds of false alerts to on-call engineers and create bogus incident tickets. Dynatrace manages this through Maintenance Windows.
The Three Problem Detection and Alerting Options
When configuring a Maintenance Window, administrators must choose one of three distinct problem detection and alerting behaviors:
| Maintenance Mode Setting | Problem Detection Behavior | Notification Integration Behavior | Baseline Impact & Recommended Use |
|---|---|---|---|
| 1. Detect problems and alert | Davis actively detects anomalies and creates problem cards in the UI as usual. | External notification integrations (ServiceNow, Slack, PagerDuty) dispatch alerts normally. | Dynamic baselines continue updating. Used when monitoring critical redundant systems during maintenance where unexpected failures must still page staff. |
| 2. Detect problems but suppress alert notifications | Davis actively detects anomalies and creates problem cards in the UI. | External notification integrations are completely silenced (suppressed). | Dynamic baselines exclude anomalous maintenance periods to prevent baseline corruption. Recommended default for routine patching: engineers can inspect problem cards in the UI, but on-call staff are not paged. |
| 3. Disable problem detection | Anomaly detection is completely turned off; Davis does not generate any problem cards. | No notifications are dispatched because no problems exist. | Metric telemetry (CPU, disk, memory, request counts) is still collected and graphed, but no incidents are raised. Used during destructive testing, data center migrations, or when components are taken offline indefinitely. |
Exam Key Point: The DCA exam frequently tests the difference between "Detect problems but suppress alert notifications" and "Disable problem detection". The former still creates problem cards in the Dynatrace UI for auditing while silencing external notifications; the latter disables the Davis anomaly detection engine entirely so that zero problem cards are generated.
A financial institution integrates Dynatrace with an internal, on-premises ServiceNow instance located in a secure data center behind strict corporate firewalls. When testing a custom problem webhook from their Dynatrace SaaS environment, the webhook fails with a network connection timeout. What architectural configuration is required to establish successful webhook delivery?
An enterprise infrastructure team plans a major storage array firmware upgrade during a scheduled window from 01:00 AM to 04:00 AM. They require that Davis AI continues to detect all infrastructure anomalies and record problem cards in the Dynatrace UI for post-maintenance review, but they must prevent on-call staff from receiving PagerDuty pages or Slack notifications. Which Maintenance Window setting must be chosen?
A platform engineering team is creating a custom webhook integration to stream Dynatrace problem events into an Apache Kafka topic. Downstream event consumers must be able to correlate the initial OPEN notification with subsequent UPDATE and RESOLVED notifications for the exact same incident. Which dynamic placeholder must be included in the webhook payload to provide this persistent correlation key?