2.1 Hypervisor Architecture & ESXi Operating Model

2.1 Hypervisor Architecture & ESXi Operating Model

Virtualization is the foundational technology that decouples operating systems and application workloads from physical computing hardware. By introducing an abstraction layer known as a hypervisor (or Virtual Machine Monitor, VMM), physical compute, memory, storage, and networking resources can be dynamically partitioned into multiple isolated, portable execution containers called virtual machines (VMs).

---

Hypervisor Classifications: Type-1 (Bare-Metal) vs. Type-2 (Hosted)

Hypervisors fall into two distinct architectural categories based on their relationship with the underlying physical hardware:

In an enterprise private cloud built on VMware vSphere Foundation (VVF), Type-1 virtualization is non-negotiable. Because the hypervisor possesses direct, unmediated access to physical memory controllers, PCI Express busses, and processor instruction sets, it delivers near-native performance while providing resource-isolation controls between competing tenant workloads.

---

VMware ESXi Architecture & The VMkernel Operating Model

VMware ESXi is a purpose-built, bare-metal operating system specifically engineered to run virtual machines. Unlike legacy virtualization platforms that wrapped hypervisor functions inside a modified Linux distribution, modern ESXi utilizes a modular microkernel architecture centered around the VMkernel.

1. The VMkernel

The VMkernel is the core engine of ESXi. It controls all hardware access, schedules processes, manages memory allocation, and provides core platform services:

• Process Scheduling: Dispatches virtual CPU (vCPU) threads across physical CPU cores (pCPUs) while enforcing resource allocations, reservations, and limits.
• Memory Management: Virtualizes physical RAM, maintaining page mapping tables and orchestrating reclamation mechanisms when physical capacity is constrained.
• Storage Stack: Implements the Pluggable Storage Architecture (PSA) and Native Multipathing Plugin (NMP) to manage supported block-storage paths; NVMe paths can use the High Performance Plug-in (HPP), while NFS uses its separate client stack.
• Networking Stack: Manages virtual standard switches (vSS) and virtual distributed switches (vDS), packet encapsulation, VLAN tagging, and traffic filtering.

2. UserWorld

Outside the VMkernel microkernel sits UserWorld, a POSIX-compliant user-space execution environment. UserWorld runs essential management daemons and administrative utilities without exposing the core kernel to user-space failures:

• hostd (Host Daemon): The primary host management service. It provides the programmatic interface for direct management via the VMware Host Client and DCUI, tracking local VM states and storage configurations.
• vpxa (vCenter Server Agent): The communication bridge between the ESXi host and vCenter Server. When a host is added to vCenter, vpxa is installed and connects to the vCenter vpxd service, relaying cluster directives, DRS migrations, and alarm states.
• syslogd: Centralized logging daemon responsible for buffering and dispatching system messages to local storage and remote collectors such as VCF Operations for Logs.

3. Native Driver Model

Modern ESXi utilizes a 64-bit native device driver model (VMKAPI). Hardware vendors write device drivers compiled directly against the VMkernel API, eliminating legacy Linux emulation layers (vmklinux). This architectural shift ensures superior I/O throughput, lower CPU utilization during packet processing, and hardware diagnostic reporting.

---

CPU Virtualization, Scheduling & Overcommitment

Hardware-Assisted CPU Virtualization

Modern x86-64 processors include hardware-assisted virtualization extensions: Intel VT-x and AMD-V. Prior to hardware assistance, hypervisors relied on complex binary translation to trap and virtualize privileged Ring 0 instructions.

Hardware extensions introduce two operating modes:

• VMX Root Operation: The execution mode utilized by the VMkernel to manage physical resources.
• VMX Non-Root Operation: The execution mode where guest operating system code and applications execute natively. When a guest OS executes a privileged instruction requiring hypervisor intervention, hardware initiates a VM-Exit, returning execution control to the VMkernel. Once handled, a VM-Entry resumes guest execution.

vCPU to pCPU Mapping & Proportional Share Scheduling

A virtual CPU (vCPU) represents an execution context presented to the guest operating system. The VMkernel scheduler maps each vCPU thread to a logical processor core (a physical core or an enabled simultaneous multithreading / Hyper-Threading logical thread).

Administrators may provision more vCPUs than physical cores. CPU overcommitment is viable when workloads do not demand all provisioned CPU simultaneously, but no universal vCPU-to-core ratio guarantees performance. Measure application latency, CPU demand, ready time and co-stop, including busy periods and a host-out-of-service scenario. Hyper-Threading logical processors are not equivalent to additional physical cores. Reduce unnecessary vCPUs before treating a low utilization percentage as a reason to add more.

Relaxed Co-Scheduling & Performance Metrics

Early hypervisors employed strict co-scheduling, requiring every vCPU of a symmetric multiprocessing (SMP) virtual machine to be scheduled simultaneously across physical cores. If an 8-vCPU VM needed execution time, all 8 physical cores had to be idle simultaneously, leading to severe CPU starvation.

ESXi utilizes relaxed co-scheduling. The VMkernel allows vCPUs within the same VM to advance at slightly different rates. If a specific vCPU outpaces its sibling vCPUs beyond a defined threshold, the scheduler artificially throttles that thread. Administrators track this via two primary metrics in esxtop:

• %RDY (Ready Time): The percentage of time a vCPU was ready to execute instructions but had to wait in the VMkernel run queue for an available physical processor core. Interpret ready time with the vCPU denominator, sample interval and workload baseline. Sustained elevated values justify investigation; a single fixed threshold is not proof of one root cause.
• %CSTP (Co-Stop Time): The percentage of time a vCPU was paused by the scheduler to allow lagging sibling vCPUs to catch up. A high %CSTP indicates that the VM has been configured with more vCPUs than its workload requires (oversizing), wasting scheduling overhead.

---

NUMA and Virtual NUMA (vNUMA) Topologies

Modern multi-socket enterprise servers utilize Non-Uniform Memory Access (NUMA) architectures. In a NUMA system, physical memory is partitioned into distinct nodes, each directly wired to an on-die memory controller of a specific CPU socket:

• Local Memory Access: When a CPU core accesses memory modules attached to its own socket's memory controller, access latency is minimal and throughput is maximized.
• Remote Memory Access: When a CPU core must read or write memory attached to a different processor socket, traffic must traverse inter-socket point-to-point buses (such as Intel Ultra Path Interconnect - UPI, or AMD Infinity Fabric). Remote access may incur additional latency; the amount depends on the hardware and workload.

The ESXi NUMA Scheduler

By default, the VMkernel NUMA scheduler monitors VM placement:

1. Home Node Assignment: When a VM powers on, the hypervisor assigns it a primary NUMA "home node" where all its vCPU threads and memory allocations reside.
2. Dynamic Rebalancing: If host workloads become unbalanced across sockets, the scheduler migrates entire VMs or moves memory pages between nodes to equalize socket utilization.

Virtual NUMA (vNUMA)

Virtual NUMA (vNUMA) exposes a virtual NUMA topology to a supported guest. Its topology depends on the VM compatibility level, CPU topology and configuration; inspect the topology generated for the deployed version rather than memorizing a nine-vCPU threshold for every release.

vNUMA constructs a virtualized NUMA topology inside the guest OS. Advanced workloads (such as Microsoft Windows Server, Linux NUMA schedulers, and database engines like Microsoft SQL Server and Oracle) detect these virtual NUMA nodes. The guest OS then intelligently places internal application threads alongside their respective memory allocations, ensuring that physical memory calls remain local to the corresponding physical CPU socket.

---

Memory Virtualization & Overcommitment Reclamation Hierarchy

In physical computing, an operating system translates Virtual Memory Addresses (VA) to Physical Memory Addresses (PA). In a virtualized architecture, ESXi introduces a third layer:

• Guest Virtual Address (GVA): The virtual address space mapped by applications inside the guest OS.
• Guest Physical Address (GPA): What the guest OS perceives as contiguous physical RAM.
• Host Physical Address (HPA): The actual physical DRAM chips installed on the server motherboard.

Modern hardware manages this two-dimensional translation via hardware-assisted nested paging: Extended Page Tables (EPT) on Intel, and Nested Page Tables (NPT) on AMD, caching translations in the processor's Translation Lookaside Buffer (TLB).

Reclamation Mechanisms and Their Trade-offs

ESX can reclaim memory with page sharing, ballooning, compression and host swapping. This is a useful order of increasing potential cost, not a strict sequence guaranteed for every VM. Memory state, large pages, available reclaimable memory, Tools availability and reservations affect which mechanisms are used. Do not apply old fixed free-memory percentages to every modern host. Watch the actual counters and investigate the source of pressure.

1. Transparent Page Sharing (TPS)

• Mechanism: The VMkernel scans guest physical memory pages and generates hash signatures for each 4 KB page. If identical hash values match, it performs a bit-by-bit comparison. If identical, the hypervisor updates the memory table so both VMs reference a single shared physical page, marking it read-only with Copy-on-Write (COW) protection. If either VM writes to the page, ESXi transparently instantiates a private copy.
• Salting & Security: Modern vSphere versions restrict inter-VM page sharing by default to prevent speculative execution and side-channel information leaks. Inter-VM TPS is only allowed between virtual machines sharing identical cryptographic salts (Mem.ShareForceSalting). Intra-VM TPS (sharing identical pages within the same VM) remains active.
• Impact: Sharing can save memory, but scanning and copy-on-write still have costs.

2. Memory Ballooning (vmmemctl)

• Trigger: Guest-assisted reclamation is requested in response to host memory pressure.
• Mechanism: The VMkernel instructs the balloon driver (vmmemctl, installed via VMware Tools) inside the guest OS to inflate. The balloon driver allocates guest memory from the guest OS kernel. Because the guest OS believes physical memory is depleting, its internal memory manager pages out idle, cache, or non-critical process data to its own local guest swap file on disk. The balloon driver pins these physical pages and informs the VMkernel, which safely reclaims the underlying Host Physical Addresses (HPAs) for other constrained VMs.
• Impact: Low to moderate. The guest OS chooses which pages to swap out using its own internal Least Recently Used (LRU) algorithms, allowing the guest to choose pages; heavy ballooning can still harm application performance.

3. Memory Compression

• Trigger: Pages eligible for swapping may be compressed first when that is effective.
• Mechanism: When a page must be reclaimed from a virtual machine, the VMkernel checks if the page can be compressed. If the page can be compressed by 50% or more (reducing a 4 KB page to 2 KB or smaller), the compressed page is stored in a designated per-VM compression cache residing in host physical RAM. When the guest VM later reads that page, ESXi decompresses it in microseconds.
• Impact: Incurs modest CPU utilization to perform compression and decompression algorithms, but avoids the devastating latency of physical disk storage I/O.

4. Hypervisor Swapping (.vswp)

• Trigger: Host swapping may be used when reclaiming physical memory requires it.
• Mechanism: The VMkernel forcefully intercepts guest memory pages and writes them directly to the virtual machine's swap file (<vmname>.vswp), located on the VM's datastore. By default, the .vswp file is sized equal to the unreserved memory of the VM (Configured RAM minus Memory Reservation).
• Impact: Catastrophic performance degradation. Because the VMkernel possesses no insight into which guest pages are vital application buffers versus idle processes, it may swap active database execution threads to disk, causing guest applications to freeze due to disk read latency.

---

Diagnosing Memory Pressure

Compare demand with physical capacity after a host is removed for maintenance. A memory reservation protects the reserved amount but reduces placement flexibility. Additional RAM, workload relocation or rightsizing may address the cause; raising shares alone does not create physical memory.

Official Reading

Use the official 2V0-16.25 objective guide for scope and Broadcom technical documentation for the installed 9.0 component. This independent lesson was reviewed October 18, 2026; actual deployment support depends on the applicable version and configuration.

Review Takeaways

• ESX is a bare-metal hypervisor whose VMkernel schedules CPU, memory and I/O.
• Right-size vCPU allocations using workload demand and contention evidence rather than a universal consolidation ratio.
• vNUMA exposes a virtual topology to supported guests; inspect the actual generated topology.
• Page sharing, ballooning, compression and swapping depend on current memory conditions.