8.2 Storage, Network, and Workload Application Monitoring

8.2 Storage, Network, and Workload Application Monitoring

Exam Focus: In VMware Cloud Foundation 9.0, operational visibility extends seamlessly from physical hardware abstractions up through guest application microservices. For the VCP-VCF (2V0-17.25) exam, candidates must master full-stack performance telemetry across software-defined storage (vSAN OSA vs. ESA write buffer fullness, destaging latency, and congestion counters), traditional VMFS datastores (space, IOPS, and latency decomposition), software-defined networking (NSX Edge DPDK core utilization, Geneve overlay MTU requirements, and BGP routing states), in-guest telemetry collection via open-source Telegraf agents, and automated Service Discovery and Application Dependency Mapping (ADM).

[!NOTE] Blueprint wording map. Two official objectives use product-style names for capabilities covered here: "monitor vSAN storage using VCF Storage Operations" and "monitor networks using VCF Network Operations". The vSAN storage analytics and NSX overlay health monitoring in this section are what the first phrase refers to, delivered through VCF Operations. The second phrase refers to the separate VCF Operations for Networks product, whose flow analytics and physical-plus-virtual path visibility are covered in section 8.5 — VCF Operations alone does not provide those.

---

Full-Stack Infrastructure & Telemetry Correlation in VCF

Traditional enterprise monitoring tools typically operate in functional silos: storage engineers monitor storage arrays via proprietary element managers, network teams monitor physical switches via SNMP, virtualization administrators monitor vSphere clusters via vCenter Server, and application teams monitor services through application performance monitoring (APM) agents. When performance degradation occurs, this fragmented architecture leads to finger-pointing and protracted Mean Time to Resolution (MTTR).

VMware Cloud Foundation eliminates monitoring silos by embedding full-stack observability into VCF Operations. Through deep architectural integration with SDDC Manager, the platform correlates telemetry vertically across the entire infrastructure hierarchy:

• Physical Layer: Top-of-rack leaf-spine network switches, host hardware chassis, CPU package power/temperatures, and physical NVMe/SAS drives.
• Virtual Infrastructure Layer: vSphere ESXi hypervisors, vCenter management planes, and compute cluster resource schedulers.
• Software-Defined Storage & Network Layers: vSAN distributed object datastores and NSX software-defined network overlays.
• Workload Layer: Virtual machine compute/memory resource allocations, guest operating systems, and application middleware.

VCF Operations correlates these layers using its continuous dynamic topology engine. When an application service experiences a response-time spike, the correlation engine traces downstream dependencies across virtual machines, ESXi hosts, NSX Edge gateways, and vSAN disk components, automatically pinpointing whether the degradation stems from in-guest CPU starvation, physical switch port buffer drops, or vSAN write buffer exhaustion.

---

vSAN Storage Performance & Capacity Analytics (OSA & ESA)

In VCF 9.0, software-defined storage is powered by VMware vSAN. Depending on cluster hardware configuration, environments run either the legacy vSAN Original Storage Architecture (OSA) or the modern vSAN Express Storage Architecture (ESA). VCF Operations provides specialized telemetry models for both architectures.

Architectural Telemetry Models: OSA vs. ESA

• vSAN Original Storage Architecture (OSA): Built upon discrete disk groups consisting of one dedicated flash cache drive and multiple capacity drives (flash or spinning magnetic disk). VCF Operations tracks cache hit ratios, cache eviction rates, and destaging latency as data is written from the fast write buffer down to the capacity tier.
• vSAN Express Storage Architecture (ESA): Introduced for high-performance all-NVMe environments, ESA eliminates the concept of dedicated cache drives in favor of a unified single-tier storage pool. It utilizes the vSAN Log-structured File System (LFS) and a highly parallelized I/O path. In ESA, VCF Operations monitors unified NVMe pool throughput, hardware compression engine efficiency, adaptive RAID-5/6 write performance, and snapshot sub-system latency.

Key Performance Metrics: IOPS, Throughput, & Latency Profiles

When evaluating vSAN performance, VCF Operations bifurcates telemetry into two distinct planes:

1. Frontend Metrics: Measure the direct I/O experienced by guest virtual machines at the virtual SCSI/NVMe controller layer (Frontend IOPS, Frontend Read/Write Throughput, and Frontend Latency). This represents application perceived performance.
2. Backend Metrics: Measure internal storage traffic required to maintain distributed object resiliency (Backend IOPS, Resynchronization Throughput, and Component Rebuild Latency). In mirrored (RAID-1) or erasure-coded (RAID-5/6) storage policies, a single frontend guest write generates multiple backend write operations across cluster hosts.

Latency is dissected into granular stages: Guest Latency (in-VM queueing), Kernel Latency (vSphere storage stack processing), vSAN Subsystem Latency (network transfer and object commit), and Physical Device Latency (underlying SSD controller response time). This breakdown allows administrators to immediately isolate whether latency originates from guest-level misconfigurations or physical hardware bottlenecks.

Storage Congestion Metrics & Root-Cause Analysis

A critical concept tested on the VCP-VCF exam is vSAN Storage Congestion. When incoming guest write I/O volume outpaces vSAN's ability to destage data or transfer replica payloads across the network, vSAN's internal queues fill up. To prevent unrecoverable buffer overflows and data corruption, vSAN introduces artificial delay—known as congestion—to throttle incoming write requests.

VCF Operations tracks four distinct congestion types:

• Memory Congestion: Occurs when internal kernel memory buffers allocated for pending I/O operations are exhausted.
• SSD Buffer Congestion: Occurs in OSA when the cache-tier write buffer exceeds its high-water threshold before destaging to capacity disks can catch up.
• Slab Congestion: Indicates that the memory slab allocator is depleted, typically under extreme parallel I/O bursts.
• Log Congestion: Occurs when write-ahead log operations in the vSAN object layer experience backpressure.

[!IMPORTANT] Exam Diagnostic Rule: In a healthy vSAN cluster, storage congestion counters should remain at zero. Non-zero congestion values indicate that the storage subsystem is actively throttling guest workloads, which directly correlates with elevated guest write latency, virtual disk timeouts, and guest operating system application aborts.

Predictive Capacity Forecasting & Time to Capacity Exhaustion (TTCE)

Storage capacity management in vSAN requires more than monitoring simple free gigabytes. Because vSAN dynamically distributes object replicas across physical hosts, capacity forecasting must account for Storage Policy-Based Management (SPBM) overhead (e.g., 200% footprint for RAID-1 vs. 133% for RAID-5), deduplication and compression ratios, slack space requirements, and host rebuild reserve.

VCF Operations continuously executes predictive capacity algorithms to calculate Time to Capacity Exhaustion (TTCE):

• Usable vs. Raw Capacity: Accounts for thin provisioning, snapshot overhead, and dynamic data reduction efficiency.
• Host Rebuild Reserve: VCF Operations reserves sufficient cluster capacity to ensure that if the largest host in the cluster suffers a total hardware failure, the remaining hosts possess adequate free space to automatically rebuild all degraded object components back to full compliance.
• TTCE Calculation: Analyzes historical consumption trends and seasonal growth models to project the exact date when the cluster will breach its operational capacity threshold, triggering proactive procurement alerts months prior to an outage.

---

Traditional VMFS Datastore Space, IOPS, & Latency Metrics

While vSAN powers primary storage across VCF workload domains, enterprises frequently attach external storage arrays (Fibre Channel, iSCSI, or NVMe-oF) formatted with the Virtual Machine File System (VMFS) to support legacy workloads or massive database repositories. VCF Operations provides deep telemetry for VMFS datastores:

Datastore Space & Thin Provisioning Oversubscription

VCF Operations monitors provisioned space versus consumed space across VMFS volumes. When thin-provisioned virtual disks are deployed, administrators must monitor the oversubscription ratio. If a datastore reaches 100% physical capacity, the VMkernel pauses all virtual machines attempting to write to that datastore to prevent filesystem corruption, displaying a critical vCenter alert.

The Latency Decomposition Framework

When a virtual machine experiences elevated disk latency on a VMFS datastore, administrators decompose Total Guest Latency using the standard ESXi storage latency formula:

$\text{GAVG/cmd} \approx \text{KAVG/cmd} + \text{DAVG/cmd}$

• GAVG/cmd (Guest Average Latency): Total round-trip latency experienced by the virtual machine guest OS from I/O issuance to completion. Latency $< 2-5 \text{ ms}$ is standard for NVMe/all-flash arrays; $< 15-20 \text{ ms}$ is standard for hybrid or spinning disk tiers.
• KAVG/cmd (Kernel Average Latency): Time the I/O command spends inside the ESXi hypervisor storage virtualization stack. KAVG should consistently remain $< 1-2 \text{ ms}$. Elevated KAVG indicates host-side queue depth exhaustion, path thrashing during multipathing failover, or software adapter contention.
• DAVG/cmd (Device Average Latency): Time required for the command to traverse the physical HBA, SAN fabric, and be processed and acknowledged by the physical storage array. Elevated DAVG isolates the issue outside the ESXi host—pinpointing storage array controller saturation, disk spindle bottlenecks, or physical SAN switch congestion.
• QAVG/cmd (Queue Average Latency): Time spent waiting in the host driver queue. $QAVG > 0$ indicates that the storage adapter or LUN queue depth is fully exhausted.
• SCSI Aborts & Reservation Conflicts: VCF Operations monitors SCSI abort counters (disk:commandsAborted_count) and reservation conflicts. Frequent SCSI aborts point to severe storage fabric timeouts or array controller freezes, causing virtual machine disks to enter read-only modes.

---

NSX Network Monitoring & Overlay Health

Software-defined networking in VCF 9.0 is delivered by VMware NSX. The NSX Management Pack in VCF Operations models the complete virtual networking topology, bridging the gap between overlay virtual networks and physical underlay fabrics.

Edge Node & Edge Cluster Monitoring

NSX Edge nodes host centralized, stateful network services that cannot be distributed directly across hypervisors, including North-South Tier-0/Tier-1 routing, Network Address Translation (NAT), Edge Firewalls, IPsec/SSL VPN termination, and enterprise load balancers. Monitoring Edge health is paramount for tenant traffic delivery:

• Datapath CPU Core Utilization: Edge nodes dedicate specific CPU cores exclusively to Data Plane Development Kit (DPDK) fast-path packet processing. VCF Operations tracks DPDK core utilization; sustained saturation (>80%) indicates packet processing bottlenecks requiring Edge scale-up (adding vCPU/RAM) or scale-out (expanding the Edge cluster).
• Interface Packet Drops & Memory Pressure: Monitors physical and virtual Edge interfaces for receive (RX) and transmit (TX) ring buffer drops, which indicate packet bursts overwhelming the Edge datapath.
• Equal-Cost Multi-Path (ECMP) Balance: In Tier-0 active-active topologies, VCF Operations monitors upstream BGP routing paths to ensure ingress and egress traffic flows are evenly balanced across all active Edge nodes.

Geneve Overlay Tunnel Health & MTU Diagnostics

NSX utilizes Geneve (Generic Network Virtualization Encapsulation) as its network overlay protocol for east-west virtual machine traffic. Geneve encapsulates tenant Ethernet frames inside standard IP/UDP packets (destination port 6081) between ESXi transport nodes.

Geneve encapsulation adds a 50-byte header to every packet. To prevent severe packet fragmentation and transmission drops, VMware Cloud Foundation strictly mandates that physical top-of-rack switches, inter-switch links, and ESXi VMkernel interfaces support Jumbo Frames (standardized to MTU 9000).

VCF Operations monitors Geneve tunnel health by tracking:

• Tunnel Status: Real-time state of bidirectional Geneve tunnels connecting transport nodes (Up, Degraded, Down).
• Encapsulation Drops: Flags packets dropped due to MTU mismatches along the physical underlay path.
• Path Latency & Jitter: Measures microsecond latency between transport node tunnel endpoints (TEPs) to identify physical network congestion.

Routing Health & Distributed Firewall (DFW) Analytics

Beyond basic interface telemetry, VCF Operations monitors NSX control plane and security services:

• BGP Session Monitoring: Continuously audits the state machine of BGP neighbor peerings between Tier-0 Gateways and physical leaf/spine switches (Established vs. Idle, Connect, or Active states). It tracks prefix counts exchanged and raises alerts upon route flap events.
• Distributed Firewall (DFW) Flow Analytics: Evaluates security rule efficacy across thousands of workload interfaces. VCF Operations tracks rule hit counts (identifying obsolete or duplicate firewall rules) and visualizes dropped packet flows, providing immediate telemetry during troubleshooting of blocked application traffic.

---

Workload & In-Guest Application Monitoring via Telegraf

Traditional infrastructure monitoring stops at the hypervisor boundary, leaving operations teams blind to what occurs inside guest virtual machines. VCF Operations bridges this divide by incorporating the lightweight, open-source Telegraf agent framework for in-guest operating system and application service monitoring.

Architecture & Lifecycle Management

Rather than requiring heavy, complex third-party agent deployments, VCF Operations manages Telegraf agents natively:

• VMware Tools Integration: The Telegraf agent can be automatically deployed, bootstrapped, and lifecycle-managed directly through VMware Tools (Guest Operations). VCF Operations injects the agent binary and configuration into the guest operating system over the hypervisor-to-guest VMCI communication channel. This eliminates the requirement for direct network connectivity or open management ports (such as SSH or WinRM) between the VCF Operations cluster and production workload subnets.
• Enterprise Configuration: Alternatively, organizations can deploy and maintain Telegraf configurations via standard configuration management tools (Ansible, Salt, Puppet, or Terraform).

Operating System & Service Telemetry

Once active, Telegraf collects high-resolution operating system telemetry that hypervisors cannot see:

• Guest OS Metrics: CPU run queue depth, user vs. system CPU %, thread counts, memory page faults per second, virtual memory commit bytes, filesystem disk space, and disk queue latencies.
• Pre-Configured Service Plugins: Telegraf incorporates dozens of modular input plugins that extract deep performance metrics from standard enterprise middleware:
  • Database Engines: Microsoft SQL Server (buffer cache hit ratio, batch requests/sec, compilations, deadlocks), MySQL/PostgreSQL (active client connections, query execution times, slow query counts, lock wait times), and Oracle.
  • Web & Application Tiers: Apache HTTPD, NGINX, Microsoft IIS, Apache Tomcat, and Redis caches.

---

Service Discovery & Application Dependency Mapping (ADM)

In dynamic cloud environments, manual documentation of application architectures in CMDBs is notoriously out of date. VCF Operations addresses this through automated Service Discovery and Application Dependency Mapping (ADM).

Socket-Based Discovery Mechanics

Service Discovery operates non-intrusively without requiring network packet sniffing or invasive vulnerability scanning:

1. The service discovery engine queries the guest operating system (via VMware Tools or Telegraf) to inspect active listening network sockets (e.g., process postgres listening on TCP port 5432) and active established outbound network connections.
2. The VCF Operations analytics engine maps established connections between virtual machines across the environment. For example, if VM-App-01 maintains an active outbound TCP connection to VM-DB-01 on port 5432, the system automatically infers an application dependency relationship.
3. Known service signatures (HTTP, HTTPS, SSH, MySQL, MS-SQL, Oracle, Redis, Active Directory, DNS) are automatically cataloged.

Application Grouping & Blast Radius Analysis

Using discovered connection metadata, administrators convert dynamic topologies into formal Application Groups (e.g., "Corporate Core Banking Application").

Application Dependency Mapping delivers profound operational benefits during infrastructure maintenance and incident resolution:

• Change Impact Analysis: When planning host maintenance, ESXi upgrades, or NSX Edge router failovers, administrators inspect the application dependency map to visualize every upstream and downstream business service that depends on that component.
• Root-Cause Isolation: When users report slow response times on a web portal, ADM allows an operator to instantly observe whether the delay is originating within the web front-end, the application middleware, or a database locking issue on the backend database cluster.

---

Exam Watch: Key Scenarios and Candidate Traps

[!IMPORTANT] The MTU 9000 Mandate for NSX Geneve Overlays: When diagnosing packet drops between ESXi transport nodes across physical racks, always check the physical underlay MTU. Geneve encapsulation adds 50 bytes to standard frames. If physical switches enforce standard MTU 1500, encapsulated frames are silently dropped. Verify end-to-end MTU using vmkping -d -s 8972 <target_TEP_IP>.

[!TIP] Storage Latency Isolation (DAVG vs KAVG): If an exam question presents high guest latency where DAVG is elevated and KAVG is low (< 1 ms), the problem is external to ESXi (storage fabric or array). If KAVG is elevated, the bottleneck is inside the ESXi kernel storage stack or host adapter queue.

[!WARNING] vSAN Write Throttling: A non-zero value for any vSAN congestion counter (memory, SSD buffer, slab, log) indicates that the storage subsystem is actively throttling guest writes to protect buffer integrity, leading to elevated guest write latency.