2.3 vMotion & Enhanced vMotion Compatibility

vMotion & Enhanced vMotion Compatibility

VMware vSphere vMotion enables the live migration of active, powered-on virtual machines from one physical ESXi host to another with zero application downtime. When combined with Enhanced vMotion Compatibility (EVC), vMotion allows workloads to move freely across disparate server hardware generations within and between enterprise clusters.

---

vMotion Architectural Workflow & Migration Lifecycle

The vMotion engine transfers a virtual machine's active execution state—specifically its in-memory working set, device states, and virtual CPU thread contexts—over the network. The migration can be understood as seven phases:

+--------------------------------------------------------------------------+
|                         vMotion Execution Phases                         |
+--------------------------------------------------------------------------+
|  Phase 1: Initialization & Pre-Flight Compatibility Verification         |
|  Phase 2: Target Host Shadow VM Instantiation                            |
|  Phase 3: Initial Memory Pre-Copy (Base Memory Transfer)                 |
|  Phase 4: Iterative Dirty Page Tracking (Bitmap Memory Pre-Copy)         |
|  Phase 5: Final Switchover & Sub-Second VM Stun (State Transfer)         |
|  Phase 6: Shadow VM Activation & Physical Network RARP Broadcast         |
|  Phase 7: Source VM Teardown & Resource Release                          |
+--------------------------------------------------------------------------+

Detailed Phase Breakdown

1. Initialization & Compatibility Check: vCenter Server verifies that the source and destination hosts share compatible networking (port groups or VLANs), datastore access, and CPU instruction sets.
2. Shadow VM Creation: The destination ESXi host instantiates an empty "shadow" virtual machine shell, reserving the necessary compute and memory footprint.
3. Initial Memory Pre-Copy: The source host begins transmitting the virtual machine's physical memory pages across the vMotion network to the target host. During this phase, the guest OS continues running and executing I/O operations normally.
4. Iterative Memory Pre-Copy & Dirty Page Tracking: As memory pages are transferred, the guest OS writes new data to RAM. ESXi uses an internal memory tracking mechanism (a dirty page bitmap) to track every modified page. In successive iterations, vMotion transfers only the dirty pages. If the rate of guest memory modification ("dirtying rate") exceeds the network transmission speed, ESXi uses Stun During Page Send (SDPS), introduced in vSphere 5.0, to inject microsecond-scale pauses into the guest's vCPUs until the pre-copy converges.
5. Final Switchover & VM Stun: When the volume of remaining dirty pages can be transferred across the network in less than the target stun time (VMware targets a switchover under 1 second), the source host briefly stuns (quiesces) the VM. It transfers the final dirty memory pages, CPU register states, and device contexts to the target host.
6. Resume on Destination & Reverse ARP: The target host un-stuns and resumes the virtual machine. Immediately upon activation, the target ESXi host generates a broadcast Reverse Address Resolution Protocol (RARP) frame on behalf of the VM's MAC address. Physical top-of-rack switches receive this RARP and immediately update their MAC address tables (CAM tables) to point to the new physical switch port, ensuring inbound network packets route to the new host without dropped connections.
7. Cleanup: The source host releases the VM's memory allocations and deletes the source VM instance.

---

vMotion Network Requirements & Maximums

Dedicated vMotion TCP/IP Stack

vSphere provides a dedicated vMotion TCP/IP stack at the VMkernel layer. Using this stack isolates vMotion traffic from standard management networking. It features its own routing table, default gateway, and ARP table, preventing vMotion traffic from traversing management gateways or causing routing conflicts across layer-3 subnets.

---

Multi-NIC vMotion Architecture

A single vMotion migration stream cannot easily saturate multiple physical 10 GbE or 25 GbE uplinks due to single-stream TCP limitations. Multi-NIC vMotion solves this by establishing multiple concurrent TCP connections across separate physical network adapters.

Configuration Rules for Multi-NIC vMotion

To configure Multi-NIC vMotion correctly on a vSphere Standard Switch (VSS) or vSphere Distributed Switch (VDS):

1. Create two or more VMkernel interfaces (e.g., vmk1 and vmk2) and configure both with the vMotion service enabled.
2. Assign unique IP addresses within the same subnet (or across routed subnets if using the vMotion TCP/IP stack).
3. Configure explicit failover ordering on the virtual switch:
   • For vmk1: Set vmnic1 as Active and vmnic2 as Standby (or Unused).
   • For vmk2: Set vmnic2 as Active and vmnic1 as Standby (or Unused).

When a migration starts, vMotion opens parallel streams across all vMotion-enabled VMkernel adapters, so each additional adapter-and-uplink pair adds bandwidth.

---

Enhanced vMotion Compatibility (EVC)

The CPU Instruction Incompatibility Problem

When a virtual machine powers on, the guest operating system queries the physical processor's CPUID feature flags and utilizes available hardware instruction extensions (such as AVX, AVX2, AVX-512, BMI2, or AES-NI). If an administrator attempts to migrate this running VM to an older ESXi host lacking those specific CPU instruction sets, the guest OS would encounter illegal instruction faults and trigger an immediate kernel panic or crash.

Cluster-Wide EVC

Enhanced vMotion Compatibility (EVC) eliminates this barrier. Configured at the cluster level, EVC presents a standardized baseline CPU feature set to all virtual machines in the cluster:

• ESXi intercepts CPUID queries from the guest OS and masks out advanced instruction sets supported by newer physical hosts, ensuring all VMs run at the cluster baseline.
• A cluster can encompass multiple generations of processors from the same manufacturer (e.g., Intel Broadwell, Skylake, Cascade Lake, and Ice Lake).
• Limitation: EVC cannot bridge between different processor vendors. Intel and AMD CPUs cannot be combined in the same EVC cluster.

Per-VM EVC

Introduced in vSphere 6.7, Per-VM EVC decouples the CPU compatibility baseline from the cluster object (the VM needs hardware version 14 or later):

• The EVC baseline is configured directly on an individual virtual machine while powered off.
• The EVC mode is saved in the VM's configuration, so it travels with the VM.
• When the VM migrates to another cluster, another vCenter Server, or a VMware cloud, it keeps its CPU baseline, so the destination cluster does not need the same cluster-wide EVC mode. Each destination host's CPU must still support the VM's EVC mode.

---

Advanced Cross vCenter vMotion (X-vMotion)

Cross vCenter vMotion (X-vMotion) enables live migration of virtual machines across independent vCenter Server instances without requiring them to share a common Single Sign-On (SSO) domain or Enhanced Linked Mode.

Operational Prerequisites

1. Workflow: Advanced Cross vCenter vMotion, introduced in vSphere 7.0 Update 1c, adds Import and Export workflows to the vSphere Client. You can migrate between vCenter instances in different SSO domains without Enhanced Linked Mode.
2. Network Latency: Maximum supported Round-Trip Time (RTT) between source and destination ESXi hosts is 150 ms.
3. Network Provisioning: Target port groups must be mapped, or layer-2 network stretch must exist to avoid IP address disconnects.
4. Disk Provisioning: If storage is not shared, X-vMotion performs a simultaneous Storage vMotion and compute vMotion over the network.