6.1 Hypervisor Architectures and Resource Allocation

Hypervisor Architectures and Resource Allocation

Core Virtualization Principle: Server virtualization decouples the operating system and its applications from the underlying physical server hardware. By establishing a thin, highly optimized software or firmware abstraction layer—the hypervisor—multiple isolated virtual machines (VMs) share physical compute, memory, storage, and network resources concurrently with deterministic performance and fault boundaries.

Modern enterprise data center operations depend on virtualization to maximize hardware utilization, reduce physical footprint, consolidate thermal and electrical loads, and enable rapid workload provisioning. Server administrators must understand the architectural mechanics of hypervisor kernels, resource scheduling constraints, memory reclamation hierarchies, and storage provisioning models to prevent severe performance bottlenecks in production environments.

---

Hypervisor Classifications: Type 1 (Bare-Metal) vs. Type 2 (Hosted)

Hypervisors, also designated as Virtual Machine Monitors (VMM), fall into two distinct structural classifications based on where the abstraction layer sits in relation to the physical hardware.

+------------------------------------+      +------------------------------------+
|      TYPE 1 (BARE-METAL)           |      |         TYPE 2 (HOSTED)            |
+------------------------------------+      +------------------------------------+
|  +------------+    +------------+  |      |  +------------+    +------------+  |
|  | Guest VM 1 |    | Guest VM 2 |  |      |  | Guest VM 1 |    | Guest VM 2 |  |
|  +------------+    +------------+  |      |  +------------+    +------------+  |
|  |  Guest OS  |    |  Guest OS  |  |      |  |  Guest OS  |    |  Guest OS  |  |
|  +------------+    +------------+  |      |  +------------+    +------------+  |
|  | Virtual HW |    | Virtual HW |  |      |  | Virtual HW |    | Virtual HW |  |
|  +------------+----+------------+  |      |  +------------+----+------------+  |
|  |      HYPERVISOR KERNEL       |  |      |  |     TYPE 2 HYPERVISOR        |  |
|  | (VMkernel / Root Partition)  |  |      |  | (Application / User Space)   |  |
|  +------------------------------+  |      |  +------------------------------+  |
|  |      PHYSICAL HARDWARE       |  |      |  |    HOST OPERATING SYSTEM     |  |
|  | (Bare-Metal CPU, RAM, NIC)   |  |      |  +------------------------------+  |
|  +------------------------------+  |      |  |      PHYSICAL HARDWARE       |  |
+------------------------------------+      +------------------------------------+

Type 1: Bare-Metal Hypervisors

Type 1 hypervisors install directly onto the bare-metal physical server hardware, operating at the highest processor privilege level (Ring 0 / VMX root operation). There is no underlying host operating system. The hypervisor contains its own optimized scheduling kernel, physical device drivers, memory management units, and hardware abstraction layers.

Enterprise Type 1 hypervisors employ two main architectural models:

1. Monolithic Architecture (e.g., VMware ESXi):
   • The hypervisor kernel (VMkernel) contains all hardware drivers (NIC, HBA, RAID, chipset) directly within its codebase.
   • Virtual machines interact directly with the VMkernel scheduler and drivers.
   • Advantage: Extremely low overhead, sub-millisecond hypervisor latency, and minimal attack surface because extraneous OS packages, graphical shells, and generic services are omitted.
   • Constraint: Hardware compatibility is strictly restricted to certified components documented on the vendor's Hardware Compatibility List (HCL).
2. Microkernel / Parent Partition Architecture (e.g., Microsoft Hyper-V, KVM, Nutanix AHV):
   • Microsoft Hyper-V: The hypervisor layer runs directly on the hardware, but physical hardware drivers reside within a specialized virtual machine called the Management Operating System or Parent Partition (running Windows Server). The parent partition creates and manages Child Partitions (guest VMs) using the Virtual Machine Bus (VMBus) high-speed inter-partition channel.
   • Kernel-based Virtual Machine (KVM): Integrates directly into the monolithic Linux kernel by loading the kvm.ko kernel module along with processor-specific modules (kvm-intel.ko or kvm-amd.ko). The Linux kernel itself is transformed into a bare-metal Type 1 hypervisor, where each virtual machine is scheduled as a standard Linux process managed via QEMU user-space tools.
   • Nutanix AHV (Acropolis Hypervisor): An enterprise-hardened hypervisor built upon the open-source KVM architecture, orchestrated by a dedicated Controller Virtual Machine (CVM) running on each node to deliver unified hyper-converged storage and compute.

Type 2: Hosted Hypervisors

Type 2 hypervisors execute as standard user-space software applications on top of a conventional host operating system (e.g., Windows 11, macOS, Linux). Examples include VMware Workstation, VMware Fusion, and Oracle VM VirtualBox.

• I/O and Context Switching Overhead: When a guest VM inside a Type 2 hypervisor requests an I/O operation, the request must traverse multiple translation layers: Guest OS $\rightarrow$ Virtual Hardware $\rightarrow$ Type 2 Hypervisor Application $\rightarrow$ Host OS System Call $\rightarrow$ Host OS Kernel Drivers $\rightarrow$ Physical Hardware. This double-layer operating system overhead creates non-deterministic scheduling latency, substantial CPU overhead, and severe I/O throughput penalties.
• Use Cases: Type 2 hypervisors are strictly intended for local software development, sandbox testing, malware analysis, and educational environments. They are never deployed for production enterprise server workloads due to lack of high availability, lack of direct hardware interrupt routing, and dependency on the stability of the client host OS.

Architectural Comparison

---

Compute Resource Allocation: Physical Cores, SMT Threads, and vCPUs

Virtual machine compute capacity is provisioned in units of Virtual CPUs (vCPUs). Sizing and scheduling these virtual processing units requires understanding how the hypervisor maps virtual execution pipelines onto physical silicon.

Physical Cores vs. Simultaneous Multithreading (SMT)

A physical CPU package contains discrete hardware execution units called Physical Cores. When Simultaneous Multithreading (SMT)—termed Intel Hyper-Threading (HT) or AMD SMT—is enabled in UEFI firmware, each physical core exposes two Logical Processors (Threads) to the hypervisor scheduler:

$\text{Total Logical Processors (Host Compute Units)} = \text{Physical Sockets} \times \text{Cores per Socket} \times \text{SMT Threads per Core}$

For example, a dual-socket server equipped with two 16-core AMD EPYC processors with SMT enabled presents:

$2 \times 16 \times 2 = 64 \text{ Logical Processors}$

[!IMPORTANT] An SMT logical thread does not deliver the compute performance of an independent physical core. SMT duplicates only register state while sharing execution pipelines (ALUs, FPUs) and L1/L2 caches. In virtualized environments, an SMT thread typically delivers approximately 20% to 30% of the throughput of a dedicated physical core. Sizing mission-critical VMs assuming a 1:1 performance equivalence between SMT threads and physical cores will lead to severe CPU contention.

vCPU Overcommitment Ratios

CPU Overcommitment is the practice of assigning more aggregate vCPUs across all running virtual machines than the total number of physical cores or logical processors available on the host:

$\text{vCPU-to-pCPU Ratio} = \frac{\sum \text{vCPUs Assigned across all Active VMs}}{\text{Total Host Physical Cores (or Logical Processors)}}$

Recommended overcommitment guidelines vary based on workload predictability:

• 1:1 Ratio (No Overcommitment): Mandatory for latency-sensitive, high-frequency transactional workloads, real-time audio/video encoding, and heavy database engines (SAP HANA, Microsoft SQL Server Enterprise, Oracle Database).
• 2:1 to 3:1 Ratio: Conservative, standard enterprise ratio for general application servers, web servers, and internal line-of-business software.
• 4:1 to 5:1 Ratio: Aggressive ratio suitable for development/test clusters, staging environments, and batch workloads with asynchronous processing schedules.
• 6:1 to 10:1+ Ratio: Deployed strictly in Virtual Desktop Infrastructure (VDI) environments where end users exhibit low average CPU utilization with staggered peak demand.

CPU Scheduling Metrics: Ready Time (%RDY) and Co-Stop Time (%CSTP)

When administrators overcommit CPU resources, hypervisor scheduling bottlenecks emerge. In VMware environments, administrators inspect these bottlenecks using the real-time interactive utility esxtop (or equivalent performance counters in Hyper-V and KVM):

   ID    GID NAME             NWLD   %USED    %RUN    %SYS   %WAIT   %VMWAIT    %RDY   %CSTP  %MLMTD
10421  10421 sql-prod-01        16  312.45  310.12    1.20  1285.4      0.00    8.42    4.15    0.00
10422  10422 web-frontend-01     4   45.20   44.10    0.40   355.2      0.00    0.85    0.02    0.00

1. CPU Ready Time (%RDY)

• Definition: The percentage of time a virtual machine is in a runnable state (ready to execute instructions) but must wait in the hypervisor scheduling queue because all physical cores and logical processors are fully saturated by other tasks.
• Evaluation Thresholds:
  • < 2% per vCPU: Optimal performance. No noticeable scheduling delay.
  • 2% to 5% per vCPU: Borderline contention. Requires monitoring during peak utilization windows.
  • > 5% per vCPU: Severe contention. Guest operating systems and users experience application latency, dropped network packets, and transaction timeouts.
  • > 10% per vCPU: Catastrophic bottleneck. The host is critically overcommitted, and vCPUs must be reduced or workloads migrated.

2. CPU Co-Stop Time (%CSTP)

• Definition: The percentage of time a multi-vCPU virtual machine is paused by the hypervisor scheduler while waiting for its companion vCPUs to catch up. Modern hypervisors use relaxed co-scheduling. When a VM has 8 or 16 vCPUs, the guest operating system expects all vCPUs to advance in lockstep. If physical core contention causes vCPU 0 to advance 15 milliseconds ahead of vCPU 7, the hypervisor halts vCPU 0 (registering %CSTP) until vCPU 7 catches up, preventing guest OS kernel clock drift and thread deadlocks.
• Evaluation Threshold: Any %CSTP value exceeding 3% indicates that the VM has been allocated more vCPUs than the host scheduler can efficiently dispatch concurrently.

[!TIP] The "Monster VM" Anti-Pattern: A common administrative error is allocating 16 or 32 vCPUs to a VM under the assumption that more vCPUs always yield faster execution. If the host cannot find 16 idle physical threads simultaneously, the VM spends substantial time in %CSTP and %RDY states. Downgrading an over-provisioned VM from 16 vCPUs to 4 or 8 vCPUs frequently increases its real-world performance because the hypervisor can schedule smaller thread sets with zero co-stop delay.

---

Memory Allocation and Overcommitment Techniques

Unlike CPU resources, which can be dynamically timeshared in nanosecond slices across physical cores, physical Random Access Memory (RAM) cannot be timeshared. A physical memory address holding active data must remain resident in silicon until paged or swapped. To maximize RAM efficiency, enterprise hypervisors implement sophisticated memory virtualization and overcommitment architectures.

[Host Physical Memory Saturation Warning]
                    |
                    v
[Stage 1: Transparent Page Sharing (TPS)] ----> Eliminates identical RAM pages (Zero Impact)
                    |
            (Memory Still Depleted)
                    v
[Stage 2: Memory Ballooning (vmmemctl)]    ----> Forces Guest OS to page idle RAM to guest disk
                    |
            (Memory Still Depleted)
                    v
[Stage 3: Hypervisor Memory Compression]  ----> Compresses pages into in-RAM compression cache
                    |
            (Host Hard Memory Wall Reached)
                    v
[Stage 4: Hypervisor Swapping (.vswp)]    ----> Directly writes active RAM pages to host disk (Catastrophic Latency)

The Memory Abstraction Tiers

Hypervisors manage three levels of memory addressing:

1. Guest Virtual Memory (GVM): The virtual address space presented to user-space applications running inside the guest OS.
2. Guest Physical Memory (GPM): The contiguous address space the guest OS believes is physical hardware RAM.
3. Host Physical Memory (HPM): The actual, physical DDR4 or DDR5 DRAM chips installed in the motherboard memory channels.

Hardware extensions—Second-Level Address Translation (SLAT), known as Intel Extended Page Tables (EPT) and AMD Rapid Virtualization Indexing (RVI)—perform hardware-accelerated mapping between GPM and HPM, eliminating software shadow page tables.

The Four-Stage Memory Reclamation Hierarchy

When the total aggregate memory allocated to running VMs exceeds physical host RAM, the hypervisor initiates a strictly ordered four-tier reclamation sequence:

Stage 1: Transparent Page Sharing (TPS)

• Mechanism: The hypervisor scans physical memory pages (typically 4 KB blocks) and computes a cryptographic hash of their contents. When identical pages are identified (such as duplicate OS kernel code across twenty identical Windows Server VMs), the hypervisor updates the page tables so that all VMs reference a single shared physical memory page marked as Copy-on-Write (COW). The redundant physical pages are freed.
• Write Handling: If a VM attempts to write to the shared page, the MMU triggers a page fault. The hypervisor intercepts the fault, creates an independent private copy of the page for that VM, and commits the write with zero data corruption.
• Security Context: Due to side-channel timing attacks (e.g., Spectre, Meltdown), modern hypervisors disable inter-VM TPS across different virtual machines by default (using TPS salting), restricting TPS deduplication to memory pages shared within the same individual VM.

Stage 2: Memory Ballooning

• Mechanism: When physical host memory utilization climbs past warning thresholds (e.g., ESXi "Soft" state at 6% free memory), the hypervisor commands a specialized device driver installed inside the guest OS—the balloon driver (such as vmmemctl in VMware Tools or VirtIO balloon driver in KVM)—to expand (inflate).
• Execution: The balloon driver requests memory allocations from the guest operating system's internal memory manager. Because the guest OS treats the balloon driver as a standard application, it allocates its least-recently-used, idle, or cached memory pages to the balloon. If free guest RAM is exhausted, the guest OS uses its own internal paging algorithms to write idle pages to the guest swapfile/pagefile.
• Reclamation: The balloon driver captures these physical memory addresses and notifies the hypervisor via hypercalls. The hypervisor unmaps the backing Host Physical Memory pages and reclaims them for other constrained VMs.
• Performance Impact: Moderate. Because the guest operating system's internal memory manager chooses which pages to page out, active application working sets are preserved. Memory ballooning requires the vendor integration tools to be installed and active.

Stage 3: Hypervisor Memory Compression

• Mechanism: If host memory contention escalates and ballooning is insufficient or slow to react (e.g., ESXi "Hard" state at 4% free memory), the hypervisor intercepts memory pages scheduled to be written to disk. Instead of executing slow disk I/O, the hypervisor runs a fast in-memory compression algorithm.
• Storage: If the 4 KB page can be compressed to 2 KB or smaller, it is stored in a dedicated memory compression cache carved out of the host's physical RAM. If the VM subsequently requests that page, it is rapidly decompressed back into the VM's working set.
• Advantage: Memory decompression in RAM requires microseconds, whereas reading an uncompressed page from disk requires milliseconds—making compression orders of magnitude faster than swapping.

Stage 4: Hypervisor Swapping (.vswp)

• Mechanism: The final, emergency defense against complete host kernel panics (e.g., ESXi "Critical" state at 1% free memory). When all physical RAM, ballooning, and compression limits are breached, the hypervisor forcibly extracts memory pages from the VM and writes them to a dedicated host-level swap file (e.g., the .vswp file in VMware ESXi or .bin/.vsv files in Hyper-V) located on the underlying datastore.
• Catastrophic Performance Impact: The hypervisor possesses no semantic awareness of guest OS processes. It may swap out active kernel data structures, transaction locks, or database index trees. The guest VM experiences massive I/O waits, application hangs, and severe operational degradation. Hypervisor swapping is a temporary safety mechanism to prevent host crashes, not a viable operating state.

Memory Reclamation Comparison

---

Virtual Disk Formats and Storage Provisioning Architectures

Virtual machines persist operating systems, configuration state, and application data inside encapsulated virtual disk files stored on hypervisor datastores.

Virtual Disk Formats

Enterprise hypervisors implement distinct virtual disk container specifications:

1. VMDK (Virtual Machine Disk):
   • Developed by VMware for ESXi and Workstation. On enterprise VMFS (Virtual Machine File System) or NFS datastores, a VMDK consists of a small plain-text descriptor file (<vmname>.vmdk) and a raw data extents file (<vmname>-flat.vmdk).
   • Supports maximum virtual disk sizes up to 62 TB on VMFS-5 and VMFS-6.
2. VHD and VHDX (Virtual Hard Disk - Microsoft):
   • VHD (Legacy): Max capacity limited to 2 TB. Does not automatically protect against data corruption caused by power interruptions during metadata updates.
   • VHDX (Modern Enterprise Standard): Introduces a 64 TB capacity limit. Integrates an internal log to track metadata changes, providing automatic resilience against structural corruption during unexpected power outages. Supports 4 KB logical sector alignments for modern Advanced Format enterprise drives.
3. QCOW2 (QEMU Copy-On-Write 2 - KVM/OpenStack):
   • The primary format for KVM, Proxmox VE, and OpenStack environments.
   • Employs sophisticated copy-on-write capabilities, small initial file size footprints, multi-tier snapshot trees, optional zlib/zstd in-line disk compression, and transparent AES-256 cryptographic encryption.

Storage Provisioning: Thin vs. Thick Provisioning

When allocating virtual disks on enterprise storage arrays or local datastores, administrators must balance capacity efficiency against sustained performance and storage stability.

+-------------------------------------------------------------+
|        100 GB VIRTUAL DISK PROVISIONING BEHAVIOR            |
+-------------------------------------------------------------+
| THIN PROVISIONED:                                           |
| [Data: 10 GB] [Unallocated Storage Space: 90 GB]            |
| Actual Datastore Space Consumed: 10 GB                      |
+-------------------------------------------------------------+
| THICK PROVISIONED (LAZY ZEROED):                            |
| [Allocated Blocks (Dirty / Unzeroed): 100 GB]               |
| Actual Datastore Space Consumed: 100 GB                     |
| *Blocks are zeroed on first write by guest OS               |
+-------------------------------------------------------------+
| THICK PROVISIONED (EAGER ZEROED):                           |
| [Allocated & Pre-Zeroed Blocks (Clean 0s): 100 GB]          |
| Actual Datastore Space Consumed: 100 GB                     |
| *Blocks are pre-cleared at creation; no first-write penalty |
+-------------------------------------------------------------+

1. Thin Provisioning

• Allocation Behavior: Virtual disk capacity is allocated and consumed dynamically on-demand as data is written by the guest operating system. A 500 GB thin-provisioned disk containing 20 GB of OS files consumes only 20 GB of real physical space on the underlying datastore.
• Advantages: Maximizes storage utilization; prevents stranded capacity; enables substantial storage overcommitment.
• Operational Risk: Storage overcommitment introduces the risk of datastore exhaustion. If multiple thin-provisioned VMs expand simultaneously and consume 100% of the physical datastore capacity, the hypervisor can no longer service write requests. To prevent filesystem corruption, the hypervisor freezes (stuns) all running virtual machines on that datastore until physical storage is expanded. Thin provisioning requires strict automated monitoring and capacity threshold alerting.
• Write Latency: When the guest OS writes to an unallocated block, the hypervisor must pause the write, allocate a new block from the physical datastore, zero the block, and then commit the guest write. This creates a minor "first-write latency penalty."

2. Thick Provisioning (Lazy Zeroed vs. Eager Zeroed)

• Thick Provision Lazy Zeroed:
  • The full physical capacity of the virtual disk is reserved on the datastore at the time of creation. A 500 GB virtual disk immediately consumes 500 GB of datastore space.
  • Mechanism: The hypervisor does not clear residual physical blocks during creation. When the guest OS executes its first write to a block, the hypervisor zeroes out the physical block immediately before committing the guest data.
  • Profile: Fast disk creation time, guarantees datastore capacity cannot be depleted by other VMs, but retains a minor first-write latency penalty.
• Thick Provision Eager Zeroed:
  • The full physical capacity is reserved, and every single sector is completely wiped and written with binary zeroes across the physical storage media during disk creation.
  • Advantages: Delivers the highest possible sustained read/write performance because blocks are pre-allocated and pre-zeroed (eliminating the first-write penalty). Completely scrubs previous data remnants from physical platters or flash blocks, satisfying stringent enterprise data sanitization standards.
  • Requirement: Eager Zeroed disks are mandatory for multi-writer shared disk clustering, such as Microsoft Windows Server Failover Clustering (WSFC) and Oracle Real Application Clusters (RAC) running on VMware vSphere, and for FT (Fault Tolerance) VMs.
  • Constraint: Creation time is significantly longer because the entire allocated capacity must be sequentially written to the storage array during provisioning.

Provisioning Models Comparison