11.3 Node Maintenance Mode, Host Shutdown, and Cluster Shutdown

11.3 Node Maintenance Mode, Host Shutdown, and Cluster Shutdown

"Place a node into or out of maintenance mode" is the first knowledge statement under Objective 3.2, and it is the single most common planned operational task an associate-level administrator performs. Anything that requires a host to stop running workloads — replacing a failed DIMM, reseating a NIC, a manual firmware update, physically moving a node — begins here.

What Maintenance Mode Does

Maintenance mode tells the cluster: stop scheduling work on this node, and move what is already there. On AHV, entering maintenance mode through Prism produces the following sequence:

1. The AHV host begins entering maintenance mode.
2. Guest VMs are evacuated. Highly available VMs are live migrated to other hosts in the cluster. VMs that cannot be migrated are powered off — specifically pinned VMs (those bound to this host by an affinity rule) and RF1 VMs (whose single data copy lives on this node).
3. The AHV host enters maintenance mode.
4. The CVM enters maintenance mode, so the cluster stops directing storage work to it and its services are gracefully taken out of the data path.

[!IMPORTANT] Not everything live migrates. Pinned VMs and RF1 VMs are powered off, not moved. If an exam scenario says an administrator entered maintenance mode and a specific VM went down while its neighbours moved cleanly, the reason is almost always an affinity rule pinning it to that host, or RF1 placement.

Prism vs. the Command Line

Nutanix recommends using the Prism web console to place nodes into maintenance mode, and that is the answer to give unless a question specifically asks about CLI capability.

The distinction worth knowing:

That CLI-only capability exists because the two roles are genuinely separable: the host runs guest VMs, while the CVM contributes storage services. Work on the hypervisor need not always remove a node's storage contribution.

The Host Is Not Shut Down Automatically

A detail that catches people out regardless of which method they used: placing a node into maintenance mode does not power off the AHV host. Maintenance mode evacuates and isolates the node logically. If the physical work requires the server to be off, shutting it down is a separate, deliberate action taken after maintenance mode is established.

The correct order for physical work on a node is therefore:

1. Enter maintenance mode and confirm the VMs have evacuated.
2. Shut down the CVM and then the host, if the task requires power off.
3. Perform the physical work.
4. Power the host back on and let the CVM start.
5. Exit maintenance mode.
6. Confirm cluster health and data resiliency have returned to normal before touching another node.

One Node at a Time

On an RF2 cluster, a node in maintenance mode means the cluster is already operating without one copy of some data. Taking a second node out at the same time risks data unavailability.

The discipline:

• Work on one node at a time.
• Confirm the cluster has returned to full resiliency — the Prism data resiliency status — before starting the next node.
• This is exactly what LCM does automatically during a rolling upgrade, which is why LCM is the right tool for software and firmware updates rather than a manual node-by-node procedure.

Shutting Down an Entire Cluster

Occasionally a whole cluster must go down — a planned power event, a datacenter move. Order matters, because the cluster must stop serving storage after its consumers have stopped using it:

1. Gracefully shut down all guest VMs. Anything still running is still writing.
2. Stop the cluster, which stops the AOS services across the CVMs.
3. Shut down the CVMs.
4. Shut down the hypervisor hosts.

Bringing it back is the reverse: power on the hosts, let the CVMs boot, start the cluster, confirm services are up and health is green, then power on guest VMs.

[!WARNING] Do not stop the cluster while guest VMs are still running. Removing storage from beneath live workloads is how filesystems get corrupted. Shut the workloads down first, every time.

Exiting Maintenance Mode

Exiting is as important as entering, and easier to forget. A node left in maintenance mode contributes nothing: it hosts no VMs and its CVM is out of the data path, so the cluster is quietly running with less capacity and less resiliency than it appears to have. After exiting, the AHV host is once again a scheduling target — and because VMs are not automatically migrated back, the cluster may be unbalanced until Acropolis Dynamic Scheduling redistributes load.