8.3 Log Analysis & Diagnostic Data Collection

8.3 Log Analysis & Diagnostic Data Collection

When performance metrics and interactive dashboards fail to isolate the root cause of an infrastructure fault, log analysis serves as the primary investigative method. VMware ESXi and vCenter Server generate granular, timestamped operational telemetry across multiple architectural layers. Mastering the location, purpose, and structure of these logs—as well as deciphering low-level kernel warnings and SCSI sense codes—is an indispensable skill for enterprise operations and the VCP-DCV certification.

ESXi Host Log Directory Hierarchy (/var/log/)

On an ESXi 8.0 host, operational system logs reside in the directory /var/log/. In modern vSphere architectures, /var/log is a symlink pointing to /scratch/log/, which resides directly on the high-endurance ESX-OSDATA partition formatted with VMFS-L. This ensures that log entries persist across host reboots.

+-----------------------------------------------------------------------------------------+
| ESXi 8.0 /var/log/ Directory Structure                                                  |
|                                                                                         |
| /var/log/ -> /scratch/log/ (Persisted on OSDATA Partition)                              |
|   |                                                                                     |
|   +-- vmkernel.log    : Core hypervisor events, device drivers, SCSI commands, NICs     |
|   +-- vmkwarning.log  : Filtered warnings and errors extracted from vmkernel.log        |
|   +-- hostd.log       : Local host management daemon (VM power, local tasks, UI)        |
|   +-- vpxa.log        : vCenter Server agent daemon (vpxd communication & heartbeats)   |
|   +-- syslog.log      : System initialization, management scripts, NTP daemon           |
|   +-- auth.log        : Authentication, SSH sessions, PAM, DCUI logins                  |
|   +-- shell.log       : Command-line audit log of commands typed in ESXi Shell          |
|   +-- vobd.log        : VMkernel Observation Daemon (broadcasts VOB events to vCenter)  |
+-----------------------------------------------------------------------------------------+

Core ESXi Log Files and Diagnostic Functions

1. vmkernel.log: The foundational hypervisor log file. It records low-level operations from the core VMkernel operating system, including:
   • Physical hardware device detection, PCI initialization, and device driver loads.
   • Network uplink link-state changes (link down/link up, speed negotiation, duplex mismatches).
   • Storage commands, LUN discovery, path transitions, SCSI aborts, and SCSI sense codes.
   • Memory allocation alerts and NUMA node placement messages.
2. vmkwarning.log: A filtered, high-signal subset of vmkernel.log. It captures exclusively messages tagged with WARNING or ALERT severities. When beginning a hardware or storage triage session, reviewing vmkwarning.log allows administrators to bypass informational kernel noise.
3. hostd.log: The log for the local host management daemon (hostd). It records:
   • Local virtual machine registration, power transitions, snapshot workflows, and configuration changes.
   • ESXi Host Client (web UI) session activities.
   • Internal host tasks, scheduled maintenance, and direct CIM provider interactions.
4. vpxa.log: The log for the vCenter Server Agent (vpxa). It records:
   • Communication exchanges between vCenter's vpxd daemon and the host.
   • Task instructions issued from vCenter (e.g., vMotion migrations, distributed switch port reconfigurations).
   • UDP heartbeat transmissions (port 902) confirming host liveness.
5. auth.log: Security and authentication log recording:
   • User login attempts via SSH, DCUI, and ESXi Host Client.
   • Pluggable Authentication Module (PAM) successes and failures.
   • Privilege escalation and invalid credential lockouts.
6. shell.log: Audit logging for administrative compliance. Records every command entered into the ESXi Shell or SSH sessions, including timestamps and the originating user account.
7. vobd.log: The VMkernel Observation Daemon log. VOB captures critical system events (e.g., storage path failures, network link drops, PSOD crash events) and broadcasts formatted event notifications to vCenter Server to trigger alarms.

Virtual Machine & vCenter Server Logging

Diagnostic telemetry extends beyond the ESXi hypervisor to individual virtual machines and the vCenter Server Appliance (vCSA).

Virtual Machine Directory Logs (vmware.log)

Every virtual machine maintains its own operational log named vmware.log. Crucially, this log does not reside in /var/log/ on the ESXi host. Instead, it is located inside the virtual machine's dedicated folder on the datastore, directly alongside the .vmx configuration file and .vmdk disk descriptors.

• Historical Archiving: When a virtual machine undergoes a power cycle (power off and power on), the active vmware.log is closed and renamed sequentially to vmware-1.log, vmware-2.log, etc., and a fresh vmware.log is initialized. By default, ESXi preserves up to 6 historical log files.
• Diagnostic Content: vmware.log records:
  • Virtual hardware initialization (virtual CPU count, virtual NVMe/LSI Logic controllers, E1000e/VMXNET3 adapters).
  • Guest operating system BSOD (Windows Blue Screen) or kernel panic stack traces.
  • Virtual Machine monitor (VMM) panics and crash dumps.
  • Snapshot creation, delta disk opening, and disk consolidation timings.
  • vMotion migration handoffs and stun times (the brief sub-second freeze during final memory copy).

vCenter Server Appliance (vCSA) Log Hierarchy

Within the vCenter Server Appliance (based on VMware Photon OS), log files reside under /var/log/vmware/ organized by microservice:

SCSI Sense Code Diagnostics & Support Bundles

When storage errors occur, the VMkernel logs raw SCSI sense codes in vmkernel.log returned from the storage array. Deciphering these codes allows administrators to distinguish between transient fabric timeouts and catastrophic hardware failures.

Decoding SCSI Sense Codes

SCSI error lines in vmkernel.log typically appear in the following standard notation: ScsiDeviceIO: 4056: Cmd(0x43a600) 0x28, CmdSN 0x80 from world 1024 to dev "naa.6005..." failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x00

• Host Status (H:): Status from the local HBA driver. 0x0 indicates Host OK; 0x1 indicates No Connect; 0x5 indicates Command Aborted.
• Device Status (D:): Status returned by the target array. 0x0 indicates Good; 0x2 indicates Check Condition (sense data follows).
• Sense Key / ASC / ASCQ (0xS 0xASC 0xASCQ): The three critical hex values defining the exact failure reason:
  1. Sense Key (0xS): General category of error (e.g., 0x2 = Not Ready, 0x5 = Illegal Request, 0x6 = Unit Attention).
  2. Additional Sense Code (0xASC): Specific error description.
  3. Additional Sense Code Qualifier (0xASCQ): Granular sub-code detailing the error condition.

Critical SCSI Sense Code Diagnostic Matrix

Diagnostic Data Collection: Support Bundles

When escalating issues to VMware Support or conducting deep root-cause audits, administrators must generate comprehensive diagnostic bundles containing logs, configuration files, and state dumps:

1. ESXi Host Diagnostic Bundle (vm-support):
   • Executed via ESXi Shell / SSH: vm-support
   • Best Practice Execution: Direct output to a shared datastore to avoid filling host ramdisk memory:

     vm-support -w /vmfs/volumes/SAN-Datastore-01/support_bundles/
     

   • Performance Snapshot Mode: Captures high-frequency performance counters over time:

     vm-support -p -d 30 -i 5
     

     (Captures performance data every 5 seconds for 30 seconds duration).
2. vCenter Server Diagnostic Bundle (vc-support):
   • Executed from the appliance Bash shell: vc-support
   • Packages /var/log/vmware/, PostgreSQL configuration dumps, and SSO identity states into a compressed .tar.gz archive.
3. vSphere Client GUI Export:
   • In the vSphere Client, right-click any ESXi host, cluster, or vCenter object -> select Export System Logs -> select specific log components -> download bundle directly through the web browser.

Centralized Syslog Forwarding & VMware Skyline

Because local host logs are subject to rotation and can be lost if system storage fails, enterprise vSphere deployments enforce centralized remote logging conforming to RFC 5424:

• Configuring Remote Syslog via esxcli:

  # Configure remote syslog target (IP/FQDN, protocol, and port)
  esxcli system syslog config set --loghost='tcp://syslog.corp.local:514'
  
  # Reload the syslog daemon to apply configuration
  esxcli system syslog reload
  
  # Open the outgoing syslog port through the ESXi host firewall
  esxcli network firewall ruleset set --ruleset-id=syslog --enabled=true
  

• VMware Skyline (Objective 5.11): Skyline used a Skyline Collector virtual appliance in the customer environment to send product usage data to VMware. The Skyline Advisor portal turned that data into proactive findings (known issues, VMSA security advisories, best-practice gaps) with remediation guidance, and Log Assist could upload support bundles to a support request. Status: Broadcom ended the Skyline Advisor service on October 4, 2024. Its findings are moving into VMware Cloud Foundation operations tooling (for example, Aria Operations 8.18 and VCF 5.2 surface critical findings), while vSphere and vSAN keep their in-product Skyline Health checks in the vSphere Client.