7.2 Metrics, Properties, Symptom Definitions, & Alerting Mechanics

7.2 Metrics, Properties, Symptom Definitions, & Alerting Mechanics

Exam Focus: Telemetry processing and alerting mechanics form the operational core of VCF Operations. For the VCP-VCF (2V0-17.25) exam, candidates must master the distinction between time-series Metrics and configuration Properties; understand the algorithmic calculation of Dynamic Thresholds (DT) and normal behavior bounds; evaluate the four types of Symptom Definitions (Metric, Property, Message Event, and Fault); construct Alert Definitions combining symptoms with boolean logic (ALL vs ANY); categorize Alert Impact (Health, Risk, Efficiency) and Criticality levels; and configure outbound notification plugins including SMTP, SNMP, REST webhooks, Slack, and ServiceNow integration.

---

Telemetry Fundamentals: Metrics vs. Properties

To manage an enterprise private cloud effectively, an operations platform must distinguish between dynamic, continuously fluctuating resource utilization and static structural configurations. VCF Operations structures all incoming telemetry into two foundational data types: Metrics and Properties.

Telemetry Ingestion & Processing Breakdown:

 ┌────────────────────────────────────────────────────────────────────────┐
 │                    Monitored Endpoint Telemetry Stream                 │
 └───────────────────┬────────────────────────────────┬───────────────────┘
                     │                                │
                     ▼                                ▼
     ┌───────────────────────────────┐ ┌──────────────────────────────┐
     │      Time-Series Metrics      │ │    Configuration Properties  │
     ├───────────────────────────────┤ ├──────────────────────────────┤
     │ - Numerical values over time  │ │ - Configuration attributes   │
     │ - Dynamic, periodic polling   │ │ - Static or state-change     │
     │ - e.g., CPU %, IOPS, Latency  │ │ - e.g., vCPUs, BIOS, Power   │
     │ - Stored in sharded FSDB      │ │ - Stored in PostgreSQL DB    │
     │ - Evaluated by Dynamic Thresh │ │ - Evaluated by Property Sym  │
     └───────────────────────────────┘ └──────────────────────────────┘

Time-Series Numerical Metrics

Metrics represent chronological, numerical data points collected at regular intervals (defaulting to every 5 minutes). Metrics measure consumption, throughput, rate of change, and latency across virtual and physical objects:

• Dynamic Utilization Data: Metrics fluctuate continuously based on active workloads. Examples include cpu|usage_average (%), mem|workload (%), disk:aggregate|totalReadLatency_average (ms), net:aggregate|droppedPct (%), and virtualDisk|totalIops_average.
• Data Storage & Rollups: Raw metric values are stored in the FileSystem Database (FSDB). Over time, the system aggregates raw data points into statistical rollups (hourly averages, daily averages, standard deviations, minimums, and maximums) to maintain rapid query performance across multi-year historical retention spans.
• Algorithmic Evaluation: Metrics serve as the primary input for Dynamic Threshold baseline calculations, capacity forecasting trends, and proactive anomaly detection.

Static & State Configuration Properties

Properties represent configuration parameters, hardware metadata, operational states, or software versions associated with an object. Unlike metrics, properties do not reflect continuous utilization curves:

• State & Attribute Metadata: Properties change only when an administrator alters a configuration, triggers a lifecycle task, or when an object changes its operational state. Examples include config|hardware|numCPU, config|hardware|memoryKB, runtime|powerState, summary|guestOS, hardware|biosVersion, and vCenter|version.
• Storage Engine: Stored in the PostgreSQL relational database and mirrored to Tanzu GemFire for rapid state checking.
• Operational Role: Properties are utilized for inventory filtering, grouping virtual machines into dynamic custom groups (e.g., all VMs running Linux with >= 8 vCPUs), tracking configuration drift against hardening baselines, and evaluating property-based alert symptoms.

---

Dynamic Thresholds (DT) & Historical Moving Baselines

Traditional infrastructure monitoring relies heavily on static thresholds (e.g., triggering an alert whenever CPU utilization exceeds 85%). In complex private clouds, static thresholds introduce significant operational friction:

• Alert Fatigue: A batch processing server that routinely reaches 90% CPU every Tuesday at 2:00 AM generates false-positive critical alarms under static rules.
• Undetected Anomalies: A mission-critical database that normally consumes 10% CPU might spike to 65% due to an internal deadlock. Because 65% is below an arbitrary 85% static threshold, the issue goes completely undetected.

To solve this, VCF Operations implements Dynamic Thresholds (DT) powered by machine learning algorithms that establish an individualized Normal Behavior Band for every metric on every monitored object.

Dynamic Threshold (DT) Normal Behavior Band:

 Metric Value
      ▲
      │                 Upper Normal Bound (+kσ)
   80 ┼ - - - - - ╭─────────────────╮ - - - - - - - - - - - - - - - - - - - - - 
      │          ╱                   ╲       ▲
   60 ┼─────────╭─────────────────────╮──────│──────────────▲───────────────── Anomaly Detected
      │        ╱   Historical Moving   ╲     │ Normal       │ (Metric crosses
   40 ┼───────╭        Baseline         ╮────│ Band (2σ)    │  Upper Bound)
      │      ╱                           ╲   ▼              │
   20 ┼ - - ╭─────────────────────────────╮ - - - - - - - - ┼ - - - - - - - - 
      │                                   Lower Normal Bound (-kσ)
    0 ┴─────────────────────────────────────────────────────┼────────────────▶ Time
                                                            │
                                                     Breach Timestamp

Algorithmic Mechanics of Dynamic Thresholds

1. Historical Moving Baseline: The DT engine analyzes a 30-day trailing window of historical metric data. It models recurring cyclical patterns across multiple time dimensions: hour of the day, day of the week, and weekday-versus-weekend operational cycles.
2. Standard Deviation Bands: Using statistical standard deviation ($\sigma$), the platform establishes an Upper Normal Bound and a Lower Normal Bound around the expected baseline. The width of this band represents the expected variance of the metric under healthy operating conditions.
3. Anomaly Identification: If an observed metric value breaches the Upper Normal Bound or drops below the Lower Normal Bound for a sustained period, VCF Operations flags the condition as an Anomaly. Because the bounds adjust dynamically throughout the day, the batch server reaching 90% at its expected Tuesday window remains within its normal band, while the quiet database jumping unexpectedly to 65% immediately triggers an anomaly alert.
4. Self-Tuning Adaptation: As enterprise application workloads evolve, the machine learning model continuously recalibrates its baseline, accommodating organic workload growth without requiring manual administrative recalibration.

---

Symptom Definitions: The Four Core Types

Symptom Definitions are the atomic, conditional building blocks of the alerting framework. A symptom evaluates whether a specific metric, property, event, or hardware condition has occurred on a target object. Symptoms evaluate to either active (true) or inactive (false).

VCF Operations provides four distinct types of symptom definitions:

1. Metric Symptoms

Metric symptoms evaluate numerical telemetry against either static limits or dynamic threshold bands:

• Static Evaluation: Compares a metric against a hard numerical value using comparison operators (>, >=, <, <=, ==, !=). Example: Virtual Machine | CPU | Ready Time (%) > 5%.
• Dynamic Evaluation: Evaluates whether a metric is operating outside its normal behavior band. Example: Virtual Machine | Memory | Usage (%) is above normal band.

2. Property Symptoms

Property symptoms evaluate configuration settings, state metadata, or inventory properties:

• Compares property strings or numbers against target values using equality, containment, or regular expressions.
• Example: Virtual Machine | Summary | Runtime | Power State equals poweredOff.
• Example: Host System | Configuration | Hardware | BIOS Version does not equal 2.14.0.

3. Message Event Symptoms

Message event symptoms evaluate incoming event notifications, syslog messages, or SNMP traps dispatched by external endpoints such as vCenter Server or physical top-of-rack switches:

• Triggered when a specific event ID, warning string, or error pattern is captured in the event stream.
• Example: Capturing vCenter event esx.problem.net.redundancy.lost indicating a physical network link failure on an ESXi host.

4. Fault Symptoms

Fault symptoms evaluate critical subsystem availability and fatal hardware conditions:

• Unlike metric anomalies that reflect performance degradation, faults indicate that an object or subsystem has suffered a complete operational failure or loss of access.
• Example: vSAN Disk Group Permanent Device Loss (PDL) or Host Redundant Power Supply Lost.

---

Alert Definitions Architecture: Impact, Criticality, & Logic

An Alert Definition packages one or more symptom definitions with impact categorization, criticality levels, operational recommendations, and outbound notifications. It represents an actionable operational event presented to the administrator.

Alert Definition Processing & Classification Architecture:

 ┌────────────────────────────────────────────────────────────────────────┐
 │                        Incoming Symptom Stream                         │
 │   [Metric Symptoms]    [Property Symptoms]    [Event & Fault Symptoms] │
 └───────────────────────────────────┬────────────────────────────────────┘
                                     │
                                     ▼
 ┌────────────────────────────────────────────────────────────────────────┐
 │                       Symptom Combination Logic                        │
 │        - ALL (AND): Every defined symptom must be active               │
 │        - ANY (OR):  Any single symptom triggers the alert              │
 │        - Advanced Cross-Object: Self vs. Parent vs. Children           │
 └───────────────────────────────────┬────────────────────────────────────┘
                                     │
                                     ▼
 ┌────────────────────────────────────────────────────────────────────────┐
 │                       Alert Classification Matrix                      │
 ├───────────────────────────────────┬────────────────────────────────────┤
 │          Alert Impact             │         Criticality Levels         │
 ├───────────────────────────────────┼────────────────────────────────────┤
 │ • Health: Immediate failure       │ • Critical: Catastrophic, down     │
 │ • Risk: Impending future problem  │ • Immediate: Urgent intervention   │
 │ • Efficiency: Resource waste      │ • Warning: Early non-standard state│
 │                                   │ • Info: Informational state change │
 └───────────────────────────────────┬────────────────────────────────────┘
                                     │
                                     ▼
 ┌────────────────────────────────────────────────────────────────────────┐
 │                  Outbound Integration & Automation                     │
 │   [ServiceNow ITSM]   [REST Webhooks]   [SNMP Traps]   [Auto-Actions]  │
 └────────────────────────────────────────────────────────────────────────┘

Alert Impact Categories

VCF Operations categorizes every alert under one of three operational Impacts, defining the business urgency and guiding the administrative response:

1. Health: Indicates an immediate operational failure or active degradation directly affecting the availability or performance of workloads (e.g., an ESXi host has become disconnected, a datastore is 100% full, or an NSX edge gateway is dropping packets). Health alerts demand immediate real-time triage.
2. Risk: Indicates an impending problem that does not currently disrupt service but will compromise system health if not remediated proactively (e.g., a vSAN datastore projected to run out of capacity in 14 days, a virtual machine experiencing escalating memory contention, or an expiring security certificate). Risk alerts guide planned maintenance.
3. Efficiency: Identifies resource waste, suboptimal provisioning, or optimization opportunities across the private cloud (e.g., oversized virtual machines that have consumed 8 vCPUs but never exceed 5% utilization, powered-off VMs holding stranded storage, or obsolete VM snapshots). Efficiency alerts drive cost reduction.

Criticality Levels

Every alert definition is assigned a baseline Criticality (which can be configured to inherit dynamically from the underlying symptom severity):

• Critical: Catastrophic failure causing immediate downtime or unrecoverable data loss.
• Immediate: Severe degradation requiring urgent administrative intervention to avoid an outage.
• Warning: Emerging problem or non-standard condition that should be investigated during normal shift hours.
• Info: Informational notification confirming a state transition (e.g., maintenance mode enabled).

Symptom Combination Logic

Alert definitions combine multiple symptoms using boolean logic:

• ALL (Logical AND): The alert triggers only when every configured symptom evaluates to active simultaneously. This drastically reduces false positives (e.g., trigger an alert only when VM CPU Ready % > 5% AND Guest OS CPU Workload % > 85%).
• ANY (Logical OR): The alert triggers if at least one configured symptom evaluates to active.
• Cross-Object / Parent-Child Symptom Rules: Advanced alerts evaluate symptoms across topological relationships. For example, an alert can be configured to fire on a Virtual Machine only if the VM exhibits high CPU Ready and its parent ESXi Host concurrently exhibits high host CPU contention, isolating host-wide resource starvation from isolated virtual machine misconfigurations.

---

Outbound Alert Notification Plugins & Enterprise Integrations

To integrate private cloud monitoring into enterprise operations centers, VCF Operations provides modular Outbound Notification Plugins that dispatch alert data in real time:

1. Standard Email (SMTP) Plugin:
   • Generates formatted HTML or plain-text email notifications dispatched to administrative distribution lists, system owners, or ticketing inboxes.
   • Includes critical alert metadata: object name, object type, alert impact, criticality, triggering symptoms, timestamp, and deep-link hyperlinks opening the alert directly in VCF Operations.
2. SNMP Trap Plugin:
   • Dispatches SNMP v2c or SNMP v3 traps to central Network Management Systems (NMS) such as SolarWinds, IBM Tivoli Netcool, or Micro Focus OpenView.
   • Encapsulates object FQDNs, severity codes, and alert identifiers within standardized VMware Object Identifiers (OIDs) defined in enterprise MIB files.
3. REST Webhook Plugin:
   • Streams structured JSON payloads over HTTPS to arbitrary REST endpoints.
   • Enables seamless integration with cloud automation tools, custom orchestrators, AWS Lambda/Azure Functions, or enterprise messaging buses (Kafka, RabbitMQ) for serverless remediation pipelines.
4. Collaboration Plugins (Slack & Microsoft Teams):
   • Formats alert payloads into interactive webhook cards posted directly to designated ChatOps channels.
   • Color-codes cards by criticality (red for Critical, orange for Immediate, yellow for Warning) and displays one-click buttons allowing on-call engineers to acknowledge the alert or launch the diagnostic workbench directly from chat.
5. ServiceNow ITSM Integration:
   • Delivers bi-directional incident management synchronization between VCF Operations and enterprise ServiceNow instances.
   • Automatically creates an Incident ticket when a Critical or Immediate alert triggers, populating the Configuration Item (CI) in the ServiceNow Configuration Management Database (CMDB).
   • When an administrator resolves the underlying infrastructure issue and symptoms clear in VCF Operations, the plugin automatically resolves the corresponding ServiceNow incident, maintaining complete audit fidelity.

Notification Rules & Routing

Administrators configure Notification Rules to filter and route alerts precisely:

• Rules evaluate incoming alerts against criteria including Alert Impact, Criticality, Target Custom Groups (e.g., only route alerts for Production-Databases), or specific Management Pack types.
• Alerts can be delayed by a configurable dampening period (e.g., notify only if the alert remains active for more than 10 minutes), preventing transient spikes from waking on-call personnel.

---

Alert Recommendations & Automated Remediation

An alert in VCF Operations does not merely describe a symptom; it provides prescriptive guidance and remediation tools:

• Actionable Recommendations: Every alert definition can include step-by-step troubleshooting instructions and direct hyperlinks to authoritative VMware Knowledge Base (KB) articles.
• Automated In-Context Actions: Operators can trigger remediation actions directly from the alert console, or configure policies to execute actions automatically upon alert generation:
  • Compute Actions: Add vCPU, Add Memory, Power Off VM, Restart Guest OS via VMware Tools.
  • Storage Actions: Delete Unused VM Snapshots, Resize Datastore, Migrate Virtual Machine Disks via Storage vMotion.
  • Orchestration Workflows: Invoke complex, multi-system remediation runbooks through native integration with VMware Aria Automation Orchestrator.

---

Exam Watch: Key Scenarios and Candidate Traps

[!IMPORTANT] Alert Criticality vs. Badge Impact: Do not confuse Alert Criticality (Info, Warning, Immediate, Critical) with Alert Impact (Health, Risk, Efficiency). Criticality indicates how urgently the condition requires human attention, while Impact determines which synthetic badge degradation score is affected in the VCF Operations user interface.

[!TIP] Dynamic Thresholds (DT) vs. Hard Thresholds: Static hard thresholds (e.g., CPU > 90%) trigger false positives for batch workloads that naturally peak at midnight. Dynamic Thresholds establish machine-learned normal operating bands based on cyclical trends, triggering alerts only when behavior deviates statistically from expected patterns.

[!WARNING] ALL vs. ANY Symptom Logic: When configuring complex alert definitions, selecting ANY (OR) logic across multiple symptoms can cause alert storms if non-critical properties fluctuate. Best practice for high-signal alerting is to use ALL (AND) logic combining a workload symptom with an underlying infrastructure contention symptom.