8.2 Nutanix Cluster Check (NCC) Architecture and Execution

Nutanix Cluster Check (NCC) Architecture and Execution

Quick Summary: Nutanix Cluster Check (NCC) is a comprehensive diagnostic framework that audits hardware, storage, networking, hypervisor configurations, and distributed services across every node. Initiated via the CVM command line (ncc health_checks run_all) or the Prism Health dashboard, NCC produces standardized results (PASS, INFO, WARN, FAIL) directly linked to Nutanix Knowledge Base articles. Crucially, NCC releases are decoupled from AOS, allowing non-disruptive updates via Life Cycle Manager (LCM) or standalone tarball installers.

---

1. NCC Diagnostic Framework & Distributed Architecture

In complex hyperconverged environments, silent configuration drift, firmware incompatibilities, or asymmetric network configurations can introduce subtle operational risks. Nutanix Cluster Check (NCC) is the premier diagnostic suite engineered by Nutanix to uncover hardware anomalies, software misconfigurations, and performance bottlenecks before they escalate into production outages.

+-------------------------------------------------------------------------+
|                        NCC Distributed Execution                        |
+-------------------------------------------------------------------------+
|  [ Initiator: CVM 1 ]                                                   |
|  Command: ncc health_checks run_all                                     |
|  - Coordinates check execution                                          |
|  - Splits plugins into parallel tasks                                   |
+--------------------+-------------------+--------------------------------+
                     |                   |                                 
       (Internal RPC | & Passwordless    | SSH)                            
                     v                   v                                 
           +-------------------+ +-------------------+                     
           |    Peer CVM 2     | |    Peer CVM 3     |                     
           |  - Hardware Check | |  - Hardware Check |                     
           |  - Storage Check  | |  - Storage Check  |                     
           +---------+---------+ +---------+---------+                     
                     |                     |                               
                     v                     v                               
           +-------------------+ +-------------------+                     
           |  AHV/ESXi Host 2  | |  AHV/ESXi Host 3  |                     
           |  - OVS MTU Check  | |  - OVS MTU Check  |                     
           +-------------------+ +-------------------+                     
                     |                     |                               
                     +----------+----------+                               
                                |                                          
                                v                                          
                     [ Aggregated Report ]                                 
                     Status: PASS | INFO | WARN | FAIL                     
                     Output: Direct KB Article Links                       

Master-Worker Execution Model

NCC is built as a distributed, multi-node Python framework. When an administrator initiates NCC from any Controller VM (CVM):

1. Coordinator Role: The local CVM acts as the master coordinator. It queries the cluster database to enumerate all active nodes, hypervisors, and storage containers.
2. Parallel Task Dispatch: The coordinator splits the diagnostic suite into parallel test routines, dispatching execution instructions to worker daemons on all peer CVMs and underlying hypervisor hosts using passwordless intra-cluster SSH and internal remote procedure calls (RPCs).
3. Local Introspection: Each CVM inspects its own local daemons (querying Cassandra tables, Stargate sockets, and Zookeeper nodes), interrogates physical server hardware via IPMI/BMC interfaces, and queries hypervisor kernel modules.
4. Result Aggregation: Worker nodes return serialized check results to the coordinator CVM, which consolidates the findings into an interactive report displayed in the terminal or web GUI.
5. Non-Disruptive Execution: NCC operates entirely with read-only inspection primitives. It does not place locks on user data, restart system daemons, or interrupt storage I/O paths.

---

2. Command-Line Interface (CLI) Execution and Targeted Checks

While Prism provides an intuitive graphical interface, the CVM command-line interface provides systems engineers and automation scripts with fine-grained control over NCC execution.

Running All Checks

To execute the complete diagnostic suite across every node, container, and hypervisor in the cluster, log into any CVM via SSH as user nutanix and execute:

ncc health_checks run_all

A full run executes over 400 specialized plugins and typically requires 10 to 20 minutes depending on cluster node count and disk capacity.

Executing Targeted Check Modules

When troubleshooting an isolated incident—such as resolving an upstream switch flap or checking disk replacement status—running the entire suite introduces unnecessary delay. Administrators can target specific functional modules:

Targeting Specific Individual Plugins

To test a single parameter, administrators use the --plugin_list flag with the exact check plugin name:

# Check CVM memory allocation against minimum sizing standards
ncc health_checks run_all --plugin_list=cvm_memory_usage_check

# Validate inter-CVM network latency
ncc health_checks run_all --plugin_list=cvm_latency_check

# Check NTP clock synchronization across all nodes
ncc health_checks run_all --plugin_list=ntp_server_check

Output Modifiers

• --send_email=1: Dispatches an HTML summary of the execution results to the email addresses configured in Prism Alert settings.
• --log_file=<path>: Directs verbose diagnostic logging to a custom file path (default output is stored in /home/nutanix/data/logs/ncc-output-latest.log).

---

3. Prism Element Health Dashboard GUI Execution

For administrators who prefer graphical operations, Prism Element embeds full NCC execution capabilities:

1. Log into Prism Element and navigate to Health.
2. In the right-hand action bar, click the Actions drop-down and select Run NCC Checks.
3. The Run Checks modal dialog appears, presenting two operational scopes:
   • All Checks: Executes the entire diagnostic catalog.
   • Specific Checks: Exposes checkboxes enabling the operator to select discrete categories (Hardware, Storage, Virtualization, Network, System).
4. Optionally enable Send email report to notify operational stakeholders upon completion.
5. Click Run.

+-------------------------------------------------------------------------+
| Run NCC Checks                                                      [X] |
+-------------------------------------------------------------------------+
| Scope:                                                                  |
|   ( ) All Checks                                                        |
|   (*) Specific Checks                                                   |
|                                                                         |
| Categories:                                                             |
|   [X] Hardware Checks          [X] Network Checks                       |
|   [ ] Storage Checks           [ ] Virtualization Checks                |
|   [ ] System Checks            [ ] Data Protection Checks               |
|                                                                         |
| Email Notification:                                                     |
|   [X] Send email report to: ops-team@enterprise.local                   |
|                                                                         |
|                                          [ Cancel ]  [ Run Checks ]     |
+-------------------------------------------------------------------------+

Prism dispatches the request to the cluster master, registers an asynchronous task in the Tasks drawer, and visualizes real-time progress percentages. Once finished, a consolidated execution summary is appended to the Health page, and detailed reports can be exported as text or PDF documents.

---

4. Interpreting NCC Diagnostic Results and KB Resolution Workflow

Every check evaluated by NCC yields one of four standardized states. Understanding the operational severity of these states is vital for cluster administrators and exam candidates.

The Knowledge Base (KB) Integration Pattern

A distinguished design feature of NCC is that it does not merely report that a test failed; it provides the exact diagnostic context and remediation recipe. Every WARN or FAIL output includes a direct URL and numerical identifier to a Nutanix Knowledge Base (KB) article.

Terminal Execution Output Example:

Detailed information for cvm_memory_usage_check:
Node 10.10.10.11:
FAIL: CVM 10.10.10.11 memory usage is at 94% (exceeds threshold of 90%).
Refer to KB 2473 (http://portal.nutanix.com/kb/2473) for details on cvm_memory_usage_check or 
remedy to resolve this failure.

When administrators observe this output, they access portal.nutanix.com, search for KB 2473, and immediately obtain:

1. The underlying mathematical conditions that trigger the check.
2. Commands to isolate which CVM daemon is consuming excessive RAM.
3. Prescriptive steps to adjust CVM memory allocation or resolve memory leaks.

---

5. Independent NCC Release Cycle and Upgrade Methods

In traditional enterprise architectures, diagnostic tools are tied to monolithic operating system releases. If a vendor identifies a new bug signature or supports a new third-party drive model, customers must upgrade the entire operating system to obtain the updated diagnostic checks.

Nutanix decouples NCC from AOS. While every AOS release bundles the latest NCC build available at shipping time, the NCC release cycle operates on an independent, rapid cadence—often delivering updates every two to four weeks.

+-------------------------------------------------------------------------+
| Monolithic Architecture vs. Decoupled NCC Architecture                  |
+-------------------------------------------------------------------------+
| Traditional: [ Monolithic OS (Kernel + Drivers + Diagnostics) ]          |
|   - Updating diagnostic rules requires full OS reboot and downtime      |
|                                                                         |
| Nutanix:     [ Acropolis OS (AOS) Base Platform (Long-Term Stability) ] |
|              [ Nutanix Cluster Check (NCC) Suite (Rapid Decoupled) ]    |
|   - Updated independently in 2 minutes via LCM or tarball               |
|   - Zero CVM reboots | Zero host maintenance | Zero VM impact           |
+-------------------------------------------------------------------------+

Why Decoupling Matters

• Proactive Detection: When Nutanix Worldwide Support discovers a rare bug signature in the field, engineering codifies that signature into a new NCC plugin. Customers can update NCC immediately, scanning their clusters to detect the defect before it impacts production.
• New Hardware Validation: As hardware vendors release new SSD, NVMe, and NIC firmware revisions, NCC plugins are updated to validate compatibility without requiring an AOS upgrade.
• Pre-Upgrade Qualification: Prior to executing major AOS or hypervisor upgrades, administrators upgrade NCC to the newest version to run pre-upgrade validation checks with the latest test logic.

Upgrade Delivery Methods

1. 1-Click via Life Cycle Manager (LCM): In internet-connected environments, running an LCM inventory automatically discovers the latest NCC release. The administrator clicks Update, and LCM non-disruptively installs NCC across all nodes in under three minutes.
2. Prism Element Software Upgrade: Navigate to Settings > Upgrade Software > NCC. Prism downloads the package directly from the Nutanix Support Portal and applies it cluster-wide.
3. Dark Site Manual Installation (Air-Gapped): In secure environments lacking outbound internet connectivity, administrators download the standalone installer tarball (nutanix-ncc-el7.x86_64-<version>.tar.gz) from portal.nutanix.com, SCP the archive to /home/nutanix on any CVM, and execute:

   tar -xzf nutanix-ncc-el7.x86_64-<version>.tar.gz
   ./ncc/bin/install.sh
   

   The installer automatically distributes and updates the NCC binaries across all peer CVMs in the cluster. CVM services remain fully active, hypervisor hosts do not enter maintenance mode, and virtual machine execution is 100% unaffected.