9.3 Outage Detection, Performance Baselining & Maintenance Suppression

Deploying synthetic monitors across global locations provides continuous telemetry, but turning raw test executions into actionable, high-fidelity operational signals requires sophisticated anomaly detection. In distributed environments, transient network hiccups, ISP routing glitches, and scheduled maintenance can easily trigger a deluge of false alarms, inducing alert fatigue across site reliability engineering (SRE) teams.

Dynatrace solves this challenge by combining configurable Outage Detection Thresholds, Automatic Retry Mechanics, Dynamic Performance Baselining, and automated Maintenance Window Suppression, all evaluated through the causal lens of the Davis AI engine.

Understanding how these anomaly detection systems operate and interact is critical for achieving the Dynatrace Certified Associate credential.

---

1. Synthetic Outage Detection Mechanics

Availability in Dynatrace Synthetic Monitoring is calculated as the percentage of successful test executions relative to total scheduled runs across all configured locations over a designated time window. When a test fails, Dynatrace does not blindly fire an alert; it evaluates the failure against a multi-layered detection policy.

+---------------------------------------------------------------------------------------------------+
|                            SYNTHETIC OUTAGE DETECTION LOGIC                                       |
+---------------------------------------------------------------------------------------------------+
| Initial Execution Fails at Location (e.g., London)                                                |
|   └── [Automatic Retry on Failure Enabled?]                                                       |
|         ├── Yes -> Execute immediate retry from London.                                           |
|         │           ├── Succeeded? -> Discard transient failure; Record 100% Success.             |
|         │           └── Failed?    -> Proceed to Location Consensus Evaluation.                   |
|         └── No  -> Proceed immediately to Location Consensus Evaluation.                          |
|                                                │                                                  |
|                                                ▼                                                  |
| Location Consensus Evaluation                                                                     |
|   ├── Check: Did failure occur in >= X locations simultaneously?                                  |
|   └── Check: Did failure persist for Y consecutive runs at affected locations?                    |
|         ├── No  -> Log regional metric failure; Suppress Davis Problem creation.                  |
|         └── Yes -> Determine Outage Scope:                                                        |
|                     ├── 100% of locations failed -> GLOBAL OUTAGE (Critical Severity)             |
|                     └── Subset of locations failed -> LOCAL OUTAGE (Warning / Availability Alert)  |
+---------------------------------------------------------------------------------------------------+

Global Outage vs. Local Outage

Dynatrace categorizes availability failures into two distinct operational states:

• Global Outage: Triggered when 100% of all configured locations fail the synthetic monitor simultaneously. A global outage indicates catastrophic system failure—such as complete application server crashes, DNS registrar failures, global CDN routing collapses, or unhandled data-tier deadlocks. Davis immediately generates a top-tier availability problem ticket with maximum urgency.
• Local Outage: Triggered when a defined subset of configured locations fail while other locations continue to report successful executions. Local outages highlight geographic or infrastructure-specific disruptions, such as a regional ISP fiber cut, localized CDN edge node degradation, or country-specific DNS censorship.

Location Consensus Rules

To prevent isolated regional network blips from triggering severe incident alarms, administrators configure Location Consensus Rules. In the monitor anomaly detection settings, operators define the minimum number of locations that must experience simultaneous failure before an alert is raised:

• Example: If a monitor runs across 6 global locations, the rule can be configured to alert only if at least 3 locations fail simultaneously. If only the Tokyo location encounters a temporary ISP failure while London, Frankfurt, Sydney, Newark, and São Paulo succeed, Dynatrace records the localized error in telemetry charts but suppresses problem ticket creation.

Automatic Retry on Failure

A cornerstone best practice for eliminating false-positive alerts is enabling Automatic Retry on Failure:

• When a synthetic test execution fails at any location, the synthetic engine does not immediately mark the run as failed.
• Instead, the engine initiates an immediate, automatic re-execution of the failed test from the exact same location.
• If the retry succeeds, the initial failure is treated as a transient network blip; no failure is recorded against availability, and no alert is dispatched.
• Only if the retry also fails is the execution officially classified as an outage event and passed to the consecutive failure evaluation engine.

Consecutive Failure Thresholds

Even after a confirmed retry failure, organizations can enforce Consecutive Failure Thresholds:

• Specifies that a monitor must fail for X consecutive execution cycles (e.g., 2 or 3 consecutive scheduled runs) before Davis generates a problem.
• Combining a 2-run consecutive failure rule with automatic retry means an application must fail four distinct test executions across two scheduled intervals before triggering an on-call notification, virtually eliminating transient noise while ensuring sustained outages are captured within minutes.

---

2. Dynamic Performance Baselining vs. Static Thresholds

An application does not have to be completely unavailable to harm business operations. A checkout transaction that normally takes 800 milliseconds but suddenly degrades to 15 seconds represents a critical performance degradation. Dynatrace provides two distinct methodologies for alerting on slow synthetic response times:

Static Performance Thresholds

• Mechanics: Administrators manually configure hard time boundaries (e.g., Alert if total monitor duration exceeds 5,000 ms or Alert if user action 'Click Add to Cart' exceeds 2,500 ms).
• Use Cases: Ideal for contractual Service Level Agreements (SLAs) or strict architectural budgets where performance must never cross a predetermined ceiling regardless of historical trends.

Auto-Adaptive Dynamic Baselines

• Mechanics: Davis AI continuously models expected performance by calculating historical moving baselines across a rolling 7-day evaluation window.
• Multi-Dimensional Modeling: Baselines are calculated independently for each individual user action within a clickpath, each geographic location, and each time of day (accounting for normal day-of-week and peak/off-peak variations).
• Percentile Tracking: Davis models both the median (50th percentile) and the 90th percentile response times. If actual synthetic transaction times degrade beyond the calculated statistical tolerance band, Davis automatically raises a Performance Degradation problem.

Davis Causal Root Cause Integration

When a synthetic monitor fails or experiences performance degradation, Davis AI does not simply notify that "Test X failed." Because synthetic requests inject x-dynatrace-test and W3C traceparent headers into OneAgent-monitored environments, Davis traces the synthetic client execution directly through the Smartscape topology:

1. Davis identifies that the synthetic 'Submit Order' clickpath user action degraded from 1.2s to 9.8s.
2. Following the injected trace context, Davis connects the synthetic action to the OrderProcessingService microservice running on host prod-app-04.
3. Davis analyzes the backend PurePath call tree and discovers that 92% of the transaction time was spent waiting on an unindexed SQL query or thread lock in the PostgreSQL database cluster.
4. Rather than generating multiple disconnected alerts, Davis creates a single unified problem ticket identifying the exact database query as the root cause, with the synthetic monitor listed as the affected business service.

---

3. Maintenance Windows and Alert Suppression

During scheduled infrastructure patching, cloud database migrations, or major application deployments, services are routinely taken offline or restarted. Without suppression controls, synthetic monitors will continuously fire alerts, flooding on-call incident channels and skewing monthly availability metrics.

Dynatrace addresses this through Maintenance Windows.

+---------------------------------------------------------------------------------------------------+
|                         MAINTENANCE WINDOW EXECUTION POLICIES                                     |
+-------------------------------------------------------------+-------------------------------------+
| POLICY OPTION                                               | BEHAVIOR & OPERATIONAL IMPACT       |
+-------------------------------------------------------------+-------------------------------------+
| 1. Stop Synthetic Execution                                 | • Monitors cease running entirely.  |
|                                                             | • Conserves DEM unit licensing.     |
|                                                             | • Blind to recovery progression.    |
+-------------------------------------------------------------+-------------------------------------+
| 2. Continue Execution with Problem & Alert Suppression      | • Monitors continue running 24/7.   |
|    (BEST PRACTICE FOR DEPLOYMENTS)                          | • Telemetry collected in charts.    |
|                                                             | • Davis problems & alerts silenced. |
|                                                             | • Verifies exact moment of recovery.|
+-------------------------------------------------------------+-------------------------------------+
| 3. Continue Execution with Normal Problem Detection         | • Tests execute normally.           |
|                                                             | • Problems & alerts fire normally.  |
|                                                             | • Used for high-risk supervision.   |
+-------------------------------------------------------------+-------------------------------------+

Maintenance Window Configuration

• Classification: Configured as either Planned Maintenance (scheduled ahead of time) or Unplanned Maintenance (recorded retroactively to account for emergency hardware failures).
• Recurrence: Can execute as a one-time window, daily recurrence, weekly schedule (e.g., Every Saturday from 01:00 to 03:00 AM), or monthly recurrence.
• Scope: Scoped globally to the entire environment or targeted granularly using entity tags, management zones, or specific synthetic monitors.

The Three Execution Policies

When creating a maintenance window, administrators must select one of three distinct synthetic behaviors:

1. Stop synthetic execution during maintenance: All assigned synthetic monitors are completely paused. No network requests are sent, and no browser instances are launched. This saves DEM units but leaves engineering teams blind during the maintenance period, preventing them from validating whether their deployment succeeded.
2. Continue synthetic execution with problem detection & alerting suppressed (Recommended): The synthetic engine continues to execute all HTTP and browser tests on their normal schedules. All performance metrics, waterfall charts, and screenshots are recorded. However, Davis AI suppresses problem generation and notification dispatch (no emails, Slack messages, or PagerDuty incidents are triggered). This enables the deployment team to watch real-time charts and verify the exact moment the application becomes healthy while preventing alert fatigue.
3. Continue synthetic execution with normal problem detection: Monitors execute and Davis generates problem tickets and notifications as usual. This is utilized when maintenance is being performed on unrelated peripheral systems and the primary business application must remain strictly operational.

SLA and SLO Protection

Service Level Objectives (SLOs) and contractual Service Level Agreements (SLAs) measure application uptime over monthly or quarterly intervals. Unplanned outages consume an application's allocated Error Budget.

Dynatrace natively integrates maintenance windows with the SLO engine:

• When configuring an SLO based on synthetic availability, administrators can toggle the option to exclude maintenance windows from the SLO calculation.
• When enabled, any synthetic failures or downtime that occur within a designated maintenance window are automatically filtered out of the availability denominator, ensuring that scheduled operational maintenance does not breach contractual customer SLAs or deplete engineering error budgets.