9.2 Azure VM Host Caching, Disk Bursting, Ephemeral OS Disks & Disk Operations

Azure VM Host Caching, Disk Bursting, Ephemeral OS Disks & Disk Operations

1. Host Disk Caching Strategies for Windows Workloads

Azure VM host caching utilizes the physical hypervisor host's high-speed RAM and local NVMe storage to accelerate disk reads and writes. Misconfiguring host caching can cause severe database corruption during ungraceful host failovers.

+-----------------------------------------------------------------------------+
|                   AZURE VM HOST CACHING ARCHITECTURE                        |
|                                                                             |
|   +---------------------------------------------------------------------+   |
|   |  AZURE HYPERVISOR HOST                                              |   |
|   |                                                                     |   |
|   |  +--------------------+             +----------------------------+  |   |
|   |  |  Host RAM / NVMe   |             | Windows Guest OS           |  |   |
|   |  |  Read Cache        | <---------- | - OS Disk (C:)             |  |   |
|   |  +--------------------+             |   (Cache: Read/Write)      |  |   |
|   |                                     |                            |  |   |
|   |  +--------------------+             | - SQL Data Disk (.mdf)     |  |   |
|   |  |  Host RAM / NVMe   | <---------- |   (Cache: ReadOnly)        |  |   |
|   |  |  Write Buffer      |             |                            |  |   |
|   |  +--------------------+             | - SQL Log Disk (.ldf) &    |  |   |
|   +------------------------------------ |   AD NTDS.dit              |  |   |
|                     |                   |   (Cache: NONE)            |  |   |
|                     v                   +----------------------------+  |   |
|   +---------------------------------------------------------------------+   |
|   |  REMOTE AZURE MANAGED DISK STORAGE SUBSYSTEM                        |   |
|   +---------------------------------------------------------------------+   |
+-----------------------------------------------------------------------------+

Caching Modes and Workload Mapping:

1. Read/Write Caching (Default on OS Disks):

   • Mechanism: Reads are served from host memory; writes are acknowledged once written to the host cache before being flushed asynchronously to remote Azure Storage.
   • Usage: Standard for OS Disks (C:) to accelerate operating system boot times, software installation, and temporary registry operations.
   • Restriction: NEVER use Read/Write caching for database transaction log files.

2. ReadOnly Caching:

   • Mechanism: Reads are cached in high-speed host RAM/local SSD. Writes bypass the cache and commit synchronously to remote persistent storage.
   • Usage: Highly recommended for SQL Server data files (.mdf), reporting databases, and read-heavy application data where repeated queries benefit from zero-latency memory cache hits.

3. None Caching (Host Caching Disabled):

   • Mechanism: All reads and writes bypass the host cache and commit directly and synchronously to the remote managed disk storage subsystem.
   • Usage: MANDATORY for SQL Server transaction log files (.ldf), Active Directory database and log files (NTDS.dit, edb.log), and transactional queue systems.
   • Integrity Rule: If an Azure physical host experiences a power failure while uncommitted writes reside in host RAM cache, uncommitted database transactions are lost, causing Update Sequence Number (USN) rollbacks in AD DS and transaction chain corruption in SQL Server.

[!CAUTION] The Volatile Temporary Disk (D:) Rule: Most Azure VM sizes include a local temporary scratch disk assigned as drive letter D:. This drive is physically located on the hypervisor host and is completely erased during VM resizing, planned hardware maintenance, or VM deallocation. Never store user data, AD DS directory partitions, or SQL database data/log files on D:. It is only suitable for the Windows pagefile (pagefile.sys) or SQL Server tempdb (when configured to recreate directories upon startup).

---

2. Disk Bursting: Credit-Based vs On-Demand Bursting

Disk bursting allows managed disks to exceed their baseline provisioned IOPS and throughput limits to handle sudden, unexpected traffic spikes or batch jobs.

+-----------------------------------------------------------------------------+
|                     DISK BURSTING MODES COMPARISON                          |
|                                                                             |
|   FEATURE          CREDIT-BASED BURSTING        ON-DEMAND BURSTING          |
|   ---------------+----------------------------+-----------------------------|
|   Applicability  | Premium SSD <= 512 GiB     | Premium SSD > 512 GiB       |
|                  | (Tiers P1 to P20)          | (Tiers P30+ and all sizes)  |
|   Mechanics      | Accumulates credits when   | Bursts dynamically whenever |
|                  | I/O is below baseline      | workload exceeds baseline   |
|   Burst Max      | Up to 3,500 IOPS / 170 MB/s| Up to 30,000 IOPS / 1,000 MB|
|   Duration       | Up to 30 minutes continuous| Unlimited (uncapped)        |
|   Cost           | Free (included in tier)    | Hourly enablement fee +     |
|                  |                            | per-burst transaction fee   |
+-----------------------------------------------------------------------------+

• Credit-Based Bursting: When a small disk (such as a P10 OS disk) operates below its baseline IOPS and throughput, it accumulates burst credits in a bucket. When demand surges (e.g., during OS boot, patch installation, or initial application load), the disk consumes credits to burst up to 3,500 IOPS and 170 MB/s for a maximum of 30 minutes at no additional cost.
• On-Demand Bursting: Designed for larger production workloads where burst durations cannot be constrained by credit buckets. When enabled on Premium SSD disks (such as P30 or P40), the disk bursts whenever demanded by the OS, up to 30,000 IOPS and 1,000 MB/s, with charges applied per burst transaction.

---

3. Ephemeral OS Disks

Traditional Azure VMs store the operating system disk on remote Azure Storage, requiring network hops across the datacenter fabric for OS reads and writes. Ephemeral OS Disks provision the operating system virtual hard disk directly on the local VM host hypervisor storage (using either the VM's local temporary NVMe disk or the VM's OS cache allocation).

+-----------------------------------------------------------------------------+
|                  PERSISTENT MANAGED DISK VS EPHEMERAL OS DISK               |
|                                                                             |
|   PERSISTENT MANAGED OS DISK            EPHEMERAL OS DISK                   |
|   +--------------------------+          +--------------------------+        |
|   | Compute Host Hypervisor  |          | Compute Host Hypervisor  |        |
|   | (VM Core + RAM)          |          | +----------------------+ |        |
|   +--------------------------+          | | VM Core + RAM        | |        |
|                |                        | +----------------------+ |        |
|       (Network Fabric)                  | | Local Temp NVMe /    | |        |
|                v                        | | Cache: OS Disk (C:)  | |        |
|   +--------------------------+          | +----------------------+ |        |
|   | Remote Azure Storage     |          +--------------------------+        |
|   | - Persistent OS Disk     |          (Zero Remote Storage Hop)           |
|   +--------------------------+          (Instant Re-imaging & Fast Reset)   |
+-----------------------------------------------------------------------------+

Key Architectural Features of Ephemeral OS Disks:

1. Zero Storage Cost: Storage capacity and transactional I/O operations for ephemeral OS disks are free; billing is tied strictly to the underlying compute VM size.
2. Extreme Performance: Delivers sub-millisecond read/write latency and maximum IOPS matching the physical NVMe drive on the host.
3. Rapid Re-imaging & Provisioning: VM re-imaging takes seconds rather than minutes because no remote disk copying or storage fabric registration is involved.
4. Workload Suitability: Perfect for stateless applications, Virtual Machine Scale Sets (VMSS) running web farms, batch processing workers, and Azure Kubernetes Service (AKS) agent nodes.
5. Key Constraints:
   • The VM size's local cache or temp disk must be equal to or larger than the OS image size (typically 127 GiB for Windows Server).
   • Data on the OS disk is lost immediately if the VM is stopped/deallocated or moved to another host due to maintenance.
   • Does not support Azure Backup VM-level snapshots, disk capturing, or customer-managed disk encryption sets.

---

4. Attaching, Resizing & Aggregating Managed Disks with PowerShell

Expanding Managed Disks Online

Azure supports online expansion of managed data disks without stopping the VM on modern VM sizes. Managed disks can only be expanded; they cannot be shrunk.

# 1. Update the managed disk size in Azure to 512 GiB
$VirtualMachine = Get-AzVM -ResourceGroupName 'RG-Production' -Name 'AZ-APP01'
$Disk = Get-AzDisk -ResourceGroupName 'RG-Production' -DiskName 'AZ-APP01-Data01'
$Disk.DiskSizeGB = 512
Update-AzDisk -ResourceGroupName 'RG-Production' -Disk -DiskName 'AZ-APP01-Data01'

# 2. Inside the Windows Server Guest OS (via PowerShell Remoting or Run Command):
# Scan for disk changes and extend the volume to maximum available size
Update-HostStorageCache
$Size = (Get-PartitionSupportedSize -DiskNumber 2 -PartitionNumber 2)
Resize-Partition -DiskNumber 2 -PartitionNumber 2 -Size $Size.SizeMax

Aggregating Disk IOPS with Storage Spaces Striping

When an application's I/O requirements exceed the maximum limits of a single Premium SSD v1 disk (20,000 IOPS / 900 MB/s), administrators attach multiple managed disks to the VM and configure a striped Storage Space (RAID-0 / Simple Resiliency) inside the Windows guest OS.

# Inside Windows Server: Initialize disks and create a striped Storage Pool
$PhysicalDisks = Get-PhysicalDisk -CanPool $true

# Create the Storage Pool
New-StoragePool `
    -FriendlyName 'DataStoragePool' `
    -StorageSubsystemFriendlyName 'Windows Storage*' `
    -PhysicalDisks $PhysicalDisks

# Create a Striped (Simple) Virtual Disk across all pooled managed disks
New-VirtualDisk `
    -StoragePoolFriendlyName 'DataStoragePool' `
    -FriendlyName 'StripedDataVolume' `
    -ResiliencySettingName 'Simple' `
    -NumberOfColumns $PhysicalDisks.Count `
    -UseMaximumSize

# Initialize and format the new striped volume with 64KB allocation unit for SQL
Initialize-Disk -VirtualDisk (Get-VirtualDisk -FriendlyName 'StripedDataVolume') -PartitionStyle GPT
New-Partition -DiskNumber (Get-Disk -FriendlyName '*StripedDataVolume*').Number -UseMaximumSize -DriveLetter F
Format-Volume -DriveLetter F -FileSystem NTFS -AllocationUnitSize 65536 -NewFileSystemLabel 'SQLData'

[!TIP] Why Simple Resiliency in Azure? Because Azure Managed Disks are already protected by underlying three-way locally redundant storage (LRS), configuring Mirroring or Parity inside Windows Server Storage Spaces introduces redundant software overhead without additional availability benefits. Simple (Striped) resiliency aggregates IOPS and throughput linearly across all attached disks up to the VM's compute-level I/O capping limits.