12.3 Storage Network, HBA, and SAN Triage

12.3 Storage Network, HBA, and SAN Triage

Quick Answer: Storage Area Network (SAN) troubleshooting demands a layered methodology spanning physical optics, fabric protocols, and host operating system initiators. In Fibre Channel (FC) SANs, physical layer faults stem from contaminated optical connectors and fiber bends, diagnosed using Digital Optical Monitoring (DOM) on SFP+ transceivers (show interface transceiver). Fabric-level triage resolves Fabric Login (FLOGI) failures, soft WWPN zoning mismatches, uncommitted zonesets, and Registered State Change Notification (RSCN) broadcast storms caused by multi-initiator zoning. On the storage array, missing LUN masking records prevent healthy hosts from discovering presented disks. In iSCSI environments, scenarios where ICMP ping succeeds but target discovery fails indicate TCP port 3260 firewall blocks, IQN syntax typos, CHAP authentication mismatches, or Jumbo Frame MTU mismatches (MTU 9000 vs 1500) where intermediate switches silently drop oversized storage packets. Finally, Multipath I/O (MPIO) triage distinguishes between transient All Paths Down (APD) and permanent Permanent Device Loss (PDL) states using Linux multipath -ll and Windows mpclaim.

Enterprise datacenters rely on block-level storage networks to deliver high-throughput, low-latency shared volumes to virtualization clusters, transaction databases, and bare-metal application nodes. Isolating SAN failures requires systems administrators to systematically analyze physical transceivers, fabric switch zoning, target access lists, and multipathing load-balancing engines.

---

Fibre Channel (FC) Physical Layer and Optical Diagnostics

Fibre Channel SANs utilize high-speed optical transceivers (SFP+, SFP28, SFP56) operating across multi-mode fiber (MMF - OM3/OM4 with 850 nm Short Wave lasers) or single-mode fiber (SMF - OS2 with 1310 nm Long Wave lasers). Physical layer degradation accounts for the overwhelming majority of SAN link flaps, frame discards, and Cyclic Redundancy Check (CRC) errors.

+-------------------------------------------------------------------------+
|                   Fibre Channel Optical Layer Triage                    |
|                                                                         |
|   [ Transmit SFP+ ] ===(Laser: TX Power)====> [ Patch Cable ]           |
|                                                     |                   |
|   * Common Optical Faults:                          |                   |
|     - Dirty LC Connector (Dust / Skin Oil) <--------+                   |
|     - Macrobend (Exceeding minimum bend radius)     |                   |
|     - Microbend (Shear stress on fiber core)        |                   |
|                                                     v                   |
|   [ Receive SFP+ ] <====(Attenuated Light)==========+                   |
|         |                                                               |
|         v                                                               |
|   [ Digital Optical Monitoring - DOM ]                                  |
|     - Reads RX Power (dBm / uW)                                         |
|     - If RX Power < -15 dBm: Bit Error Rate spikes, CRC errors, link loss|
+-------------------------------------------------------------------------+

Contaminated Optical Endfaces

The physical core of multi-mode optical fiber is only 50 microns in diameter (comparable to a strand of human hair). Microscopic dust particles, lint, or skin oils deposited on the polished ceramic ferrule of an LC connector scatter and attenuate laser light:

• Symptoms: Optical attenuation (loss of signal strength), high Bit Error Rates (BER), and incrementing CRC error counters on the FC switch port.
• Remediation: Inspect fiber endfaces using an optical inspection microscope. Clean connectors using dedicated dry mechanical click-cleaners or optical lint-free wipes saturated with 99% electronic-grade isopropyl alcohol. Never insert an uninspected fiber patch cable into a production SFP+ transceiver.

Microbends and Macrobends

• Macrobend: Occurs when an optical patch cord is bent beyond its specified minimum bend radius (typically 30 mm, or approximately 10 times the outer cable diameter). When bent severely around a rack post or pinched in a server cable management arm, laser light breaches the critical angle of reflection within the fiber core and leaks into the cladding material, causing catastrophic signal loss.
• Microbend: Microscopic geometric deformations along the fiber core caused by localized mechanical pressure (such as zip-ties pulled too tightly around cable bundles). Microbends introduce subtle signal attenuation that manifests intermittently under thermal expansion.

Digital Optical Monitoring (DOM) Diagnostics

Modern SFP+ transceivers incorporate Digital Optical Monitoring (DOM)—also known as Digital Diagnostics Monitoring (DDM). Internal sensors monitor real-time operating metrics, readable via the switch CLI (e.g., Brocade Fabric OS sfpshow or Cisco MDS NX-OS show interface transceiver details):

FC-Switch-01# show interface fc1/4 transceiver details
-----------------------------------------------------------------------------
Metric                  Current Value   High Alarm   Low Alarm   Status
-----------------------------------------------------------------------------
Temperature             38.4 C          75.0 C       -5.0 C      OK
Voltage                 3.28 V          3.63 V       2.97 V      OK
Current                 7.42 mA         12.0 mA      1.0 mA      OK
Tx Power                -2.8 dBm        -1.0 dBm     -8.0 dBm    OK
Rx Power               -17.2 dBm        -1.0 dBm    -14.0 dBm    LOW ALARM
-----------------------------------------------------------------------------

• Optical Power Metrics: Transmit (TX) and Receive (RX) optical power are measured in decibels relative to one milliwatt (dBm) or microwatts ($\mu\text{W}$).
• Evaluating Power Thresholds: Normal enterprise short-wave transceivers operate with RX power between -3 dBm and -11 dBm. If the RX power drops below the Low Alarm threshold (e.g., -15 dBm to -18 dBm), the receiver photodiode cannot reliably distinguish optical pulses, resulting in frame corruption, link flapping, and dropped buffer credits.

---

FC Fabric Protocols, Zoning Architecture, and RSCN Storms

Once physical optical links achieve signal synchronization, the Host Bus Adapter (HBA) initiates a three-step protocol login sequence with the Fibre Channel fabric:

+-------------------------------------------------------------------------+
|                     Fibre Channel Login Sequence                        |
|                                                                         |
|   [ Host HBA ]                                     [ Fabric Switch ]    |
|        |                                                    |           |
|        | -------- 1. FLOGI (Fabric Login: 0xFFFFFE) ------> |           |
|        | <------- FCID Assigned + BB_Credits Granted ------ |           |
|        |                                                    |           |
|        | -------- 2. PLOGI (Port Login to Target WWPN) ---> |           |
|        | <------- Service Parameters Negotiated ----------- |           |
|        |                                                    |           |
|        | -------- 3. PRLI (Process Login: FCP / SCSI-3) --> |           |
|        | <------- SCSI Capabilities Confirmed ------------- |           |
|        v                                                    v           |
|   * Result: Initiator can issue SCSI Read/Write commands to Target LUNs |
+-------------------------------------------------------------------------+

1. Fabric Login (FLOGI): The HBA port contacts the well-known Fabric Controller address (0xFFFFFE). The switch assigns the host a dynamic 24-bit Fibre Channel ID (FCID) (consisting of Domain, Area, and Port IDs) and exchanges Buffer-to-Buffer Credits (BB_Credits) for hardware-level flow control.
2. Port Login (PLOGI): The host initiates end-to-end communication with the target storage array port, exchanging operating parameters and buffer capabilities.
3. Process Login (PRLI): The host and storage array establish the higher-layer protocol mapping—specifically initializing the Fibre Channel Protocol (FCP) to encapsulate SCSI-3 commands across the fabric.

Soft Zoning vs. Hard Zoning

Fibre Channel switches utilize Zoning to partition the fabric into isolated communication groups, preventing unauthorized nodes from communicating:

• Soft Zoning (WWPN-Based Zoning): Nodes are identified strictly by their 64-bit World Wide Port Name (WWPN) (e.g., 20:00:00:25:B5:11:22:33). Soft zoning is the enterprise industry standard because it is independent of physical cabling; if an administrator moves an HBA cable to a different switch port, the soft zone remains intact.
• Hard Zoning (Port-Based Zoning): Nodes are partitioned based on their physical switch hardware ports (e.g., Switch Domain 1, Port 4). While immune to WWPN spoofing, hard zoning is fragile; moving a cable during maintenance immediately severs SAN connectivity.

Common Fabric Configuration Errors

• Typographical Errors in WWPNs: Mistyping a single hexadecimal character when defining a host WWPN alias prevents the switch from matching the node during name server lookups.
• Uncommitted / Inactive Zonesets: Creating a zone without adding it to the Active Zoneset (or failing to commit the configuration via zoneset activate or cfgenable) leaves the zone dormant in the switch's staging database.

Registered State Change Notification (RSCN) Storms

When a switch port state changes (e.g., an HBA reboots or an optical link flaps), the switch fabric controller broadcasts a Registered State Change Notification (RSCN) frame to inform nodes to flush their name server tables:

[!CAUTION] The Peril of Multi-Initiator Zoning: If multiple host HBAs are placed into the same zone (multi-initiator zoning), an optical link flap on one server causes the switch to flood RSCN frames to all other servers in that zone. The receiving hosts halt I/O pipelines to re-query the fabric name server. In large fabrics, this triggers an RSCN storm, causing massive SAN-wide latency spikes, aborted SCSI commands, and virtual machine disconnects.

The Golden Rule: Always implement Single-Initiator Zoning (one host initiator WWPN zoned together with one or more storage target WWPNs). Single-initiator zoning isolates link state churn strictly to the affected host.

---

Storage Controller LUN Masking and Target Presentation Failures

Even when physical fiber links are healthy and fabric zoning is fully active, a server operating system may report zero accessible storage disks. In such cases, the failure almost invariably resides in the storage array's LUN Masking engine.

+-------------------------------------------------------------------------+
|                    LUN Masking Architecture on Array                    |
|                                                                         |
|   [ Storage Array Controller ]                                          |
|   +-----------------------------------------------------------------+   |
|   | Storage Pool: Carves Logical Volumes (LUN 10, LUN 20, LUN 30)   |   |
|   |                                                                 |   |
|   | LUN MASKING TABLE:                                              |   |
|   | - LUN 10 ===> Mapped to Host A WWPN: 20:00:00:25:B5:11:22:33    |   |
|   | - LUN 20 ===> Mapped to Host B WWPN: 20:00:00:25:B5:44:55:66    |   |
|   | - LUN 30 ===> Mapped to Host C IQN : iqn.1998-01...host-c       |   |
|   +-----------------------------------------------------------------+   |
|                                                                         |
|   * If Host A's WWPN is missing from the masking table, the array       |
|     masks (hides) LUN 10, returning SCSI "Logical Unit Not Supported"!  |
+-------------------------------------------------------------------------+

How LUN Masking Operates

Logical Unit Number (LUN) Masking is an access-control security mechanism configured directly within the storage array software. While zoning controls which physical ports can negotiate handshakes across the fabric, LUN masking determines which specific logical volumes an authorized host is permitted to see and mount:

1. The storage administrator creates an Initiator Group (Host Group) containing the host HBA's WWPNs (in FC) or IQNs (in iSCSI).
2. The administrator maps specific LUN numbers to that Initiator Group.
3. When the host issues a SCSI REPORT LUNS command, the storage controller evaluates its internal masking table. If the host's WWPN is not explicitly mapped, the controller conceals the volumes, returning an empty inventory.

LUN Masking Diagnostic Checklist

• Missing WWPN in Storage Group: Verify whether the HBA WWPN was properly added to the array's host group after an HBA replacement card was installed.
• LUN ID Zero (LUN 0) Conflicts: Many legacy operating systems and hypervisors fail to complete SCSI bus enumeration if LUN 0 is not presented in the volume mapping table. If LUNs are numbered beginning at LUN 1 without a LUN 0 anchor, the OS may fail to discover any disks.
• Target Portal Presentation: In dual-controller arrays, verify that the LUN is assigned to an active Target Portal Group reachable across the host's zoned fabric.

---

iSCSI Protocol Triage: Discovery, CHAP Authentication, and Firewalling

iSCSI encapsulates SCSI-3 storage command blocks inside standard TCP/IP packets across Ethernet networks (typically dedicated 10GbE, 25GbE, or 100GbE storage VLANs).

+-------------------------------------------------------------------------+
|                        iSCSI Triage Decision Tree                       |
|                                                                         |
|   Can Host PING Storage IP?                                             |
|        |                                                                |
|        +--- NO  ---> Check IP/Subnet, VLAN tagging, cabling, switch port|
|        |                                                                |
|        +--- YES ---> Can Host connect to TCP Port 3260?                 |
|                           |                                             |
|                           +--- NO  ---> Firewall blocking Port 3260 or  |
|                           |             iscsid daemon stopped on host   |
|                           |                                             |
|                           +--- YES ---> Does Target Discovery succeed?  |
|                                              |                          |
|        +-------------------------------------+                          |
|        |                                                                |
|        +--- NO  ---> Check IQN syntax typo or Target Portal IP          |
|        |                                                                |
|        +--- YES ---> Does Target Login / Mount succeed?                 |
|                           |                                             |
|                           +--- NO  ---> CHAP secret mismatch OR         |
|                           |             Mutual CHAP configured in reverse
|                           |                                             |
|                           +--- YES ---> Do Large I/O Reads/Writes stall?|
|                                              |                          |
|                                              +--- YES -> MTU MISMATCH!  |
+-------------------------------------------------------------------------+

The "Ping Works but iSCSI Fails" Paradox

A classic CompTIA Server+ troubleshooting scenario involves an administrator who can successfully ping the storage array's IP address using standard ICMP, but the host cannot discover or connect to iSCSI target portals. ICMP operates at the network layer (Layer 3), whereas iSCSI operates at the transport/session layer (Layer 4/5). Common root causes include:

• TCP Port 3260 Blocked: iSCSI discovery and session management communicate exclusively over TCP port 3260. If an intermediate hardware firewall, switch ACL, or host software firewall (Windows Defender Firewall, Linux iptables / nftables) blocks port 3260, ICMP pings succeed while all iSCSI commands fail. Verify port accessibility using PowerShell or Linux:

  Test-NetConnection -ComputerName 192.168.50.100 -Port 3260
  

  nc -zvw3 192.168.50.100 3260
  

• iSCSI Initiator Daemon Stopped: In Linux, verify the status of the iscsid background service:

  systemctl status iscsid
  

  If inactive, discovery commands (iscsiadm -m discovery -t st -p <IP>) fail immediately.
• iSCSI Qualified Name (IQN) Syntax Typos: An IQN follows a strict regulatory format: iqn.yyyy-mm.naming-authority:unique-name (e.g., iqn.1998-01.com.vmware:esx-host-01-a1b2c3d4). If the storage array's target ACL expects this string, a single mistyped character in the host's initiator IQN prevents authentication.

Challenge-Handshake Authentication Protocol (CHAP) Failures

iSCSI utilizes CHAP to verify node identity before granting access to storage targets:

• One-Way (Unidirectional) CHAP: The target storage array authenticates the host initiator. The host must supply the exact username and secret configured in the target's database.
• Mutual (Bi-Directional) CHAP: The target authenticates the initiator, and the initiator simultaneously authenticates the target array. Mutual CHAP requires two distinct secret keys.
• Common CHAP Pitfalls:
  • Secret Length Constraints: RFC 3720 mandates that CHAP secrets must contain between 12 and 16 characters. Passwords that are too short or exceed 16 characters cause silent authentication rejects.
  • Configuration Direction Mismatch: If the storage target mandates Mutual CHAP but the host initiator is configured for One-Way CHAP, target login aborts with an authentication failure error.

---

The Jumbo Frame MTU Mismatch: The Silent Drop Phenomenon

Standard Ethernet transmits data with a Maximum Transmission Unit (MTU) of 1,500 bytes. In high-speed iSCSI storage networks, packaging large 64 KB storage blocks into 1,500-byte packets generates massive packet fragmentation, driving host CPU interrupt utilization to extreme levels. Enterprise environments deploy Jumbo Frames (MTU 9000 bytes) to reduce CPU overhead and maximize wire throughput.

+-------------------------------------------------------------------------+
|                    The Jumbo Frame MTU Mismatch Drop                    |
|                                                                         |
|   [ Host Initiator NIC ]  --> Configured for MTU 9000                   |
|             |                                                           |
|             v                                                           |
|   (Standard Ping: 64 bytes) ====> Traverses Path Cleanly (PING SUCCEEDS)|
|             |                                                           |
|   (iSCSI Data Frame: 9000 bytes with Don't Fragment flag set)           |
|             |                                                           |
|             v                                                           |
|   [ Intermediate Switch Port ] --> MISCONFIGURED FOR STANDARD MTU 1500! |
|             |                                                           |
|             +---> Frame > 1500 bytes AND DF=1                           |
|             v                                                           |
|   [ PACKET IS SILENTLY DROPPED! ] =====> HOST EXPERIENCES I/O TIMEOUT!  |
+-------------------------------------------------------------------------+

[!IMPORTANT] The End-to-End MTU Rule: For Jumbo Frames to function, every single interface along the complete physical and logical pathway must be configured for MTU 9000:

1. Host Physical NIC / Storage VMkernel / vSwitch.
2. Switch Ingress Port.
3. Switch Fabric Core / Storage VLAN.
4. Switch Egress Port.
5. Storage Array Target Port.

The Silent Drop Symptom

If an administrator enables MTU 9000 on the host NIC and the storage array target, but forgets to enable Jumbo Frames on an intermediate switch port (leaving it at MTU 1500):

• Small packets (such as standard ICMP pings of 64 bytes or TCP SYN handshakes on port 3260) pass through the switch effortlessly.
• Target discovery succeeds, and the LUN mounts cleanly.
• However, the microsecond the application attempts to execute actual disk I/O (sending 9,000-byte data frames with the IP Don't Fragment (DF) bit set), the intermediate switch port silently drops the oversized packets.
• The host experiences immediate I/O timeouts, disk queue locks, and application crashes.

Testing for MTU Mismatches via CLI

Administrators detect MTU mismatches by sending unfragmented ping packets sized to fill the maximum 9,000-byte frame. Because 28 bytes are consumed by the IP header (20 bytes) and ICMP header (8 bytes), the maximum payload size is 8,972 bytes:

• Windows Ping Test (DF bit set, 8972 payload):

  ping 192.168.50.100 -f -l 8972
  

• Linux Ping Test (DF bit set, 8972 payload):

  ping -M do -s 8972 192.168.50.100
  

• Interpreting Results: If a standard ping (ping <IP>) returns replies, but the 8,972-byte unfragmented ping returns Packet needs to be fragmented but DF set or 100% packet loss, an intermediate switch port is capped at MTU 1500 and must be adjusted to MTU 9000.

---

Multipath I/O (MPIO) Diagnostics, Path Failure Modes, and Load Balancing

In high-availability SAN fabrics, servers connect to storage via redundant Host Bus Adapters, redundant network fabrics (Fabric A and Fabric B), and dual storage controllers. Multipath I/O (MPIO) software coalesces these redundant physical paths into a single logical device node presented to the operating system.

+-------------------------------------------------------------------------+
|                        MPIO Dual-Fabric Topology                        |
|                                                                         |
|   [ Host Server Node ]                                                  |
|     [HBA Port 1]                          [HBA Port 2]                  |
|          |                                     |                        |
|          v                                     v                        |
|   [ SAN Fabric A Switch ]               [ SAN Fabric B Switch ]         |
|          |                                     |                        |
|          v                                     v                        |
|   [Storage Controller A]                [Storage Controller B]          |
|          +------------------+------------------+                        |
|                             v                                           |
|                     [ Enterprise LUN ]                                  |
|                                                                         |
|   * Normal State: Both paths Active/Ready (Round Robin balancing)       |
|   * Single Path Failure: Fabric A drops; MPIO routes 100% to Fabric B   |
|   * APD State: Both fabrics drop; host queues I/O waiting for timeout   |
|   * PDL State: Array unmaps LUN; host immediately terminates I/O        |
+-------------------------------------------------------------------------+

Single Path Failure vs. Path Flapping

• Single Path Failure: If a transceiver fails or a cable is disconnected on Fabric A, the MPIO driver transparently reroutes active I/O queues to the surviving path on Fabric B within milliseconds. The host operating system experiences zero data interruption, generating only a warning in system logs.
• Path Flapping: Occurs when an optical transceiver degrades or a cable is loose, causing the link to oscillate between active and dead multiple times per second. Path flapping severely destabilizes host MPIO drivers, causing continuous path re-negotiation, queue freezing, and kernel lockups. Systems administrators must immediately administratively disable (fence) the flapping switch port to force clean failover to the surviving link.

All Paths Down (APD) vs. Permanent Device Loss (PDL)

In enterprise virtualization (such as VMware vSphere ESXi and Microsoft Hyper-V), administrators must distinguish between two critical storage outage states:

• All Paths Down (APD): Occurs when the host loses all communication pathways to the storage array (e.g., simultaneous switch reboots or severed cable trunks), but the hypervisor cannot determine if the outage is temporary or permanent. The host holds all pending I/O in internal queues, pausing virtual machines while an APD Timeout counter runs (typically 140 seconds). If paths restore before timeout expiry, queued I/O flushes and applications resume without crashing.
• Permanent Device Loss (PDL): Occurs when the storage array explicitly transmits a SCSI sense code to the host (such as ASC 0x25 / ASCQ 0x00 - Logical Unit Not Supported), confirming that the presented LUN has been permanently unmapped, deleted, or destroyed. The hypervisor immediately fails all pending I/O without waiting for a timeout, terminating affected virtual machine processes to prevent data corruption.

MPIO Load-Balancing Policies

• Round Robin (RR): The MPIO driver distributes incoming I/O requests sequentially across all active, optimized paths (Path 1 $\rightarrow$ Path 2 $\rightarrow$ Path 1). Maximizes aggregate throughput across symmetric fabrics.
• Least Queue Depth (LQD): The driver dynamically evaluates the number of outstanding I/O requests queued on each physical path, directing the next command to the link with the smallest queue. Ideal for fabrics with asymmetric latency or mixed link speeds.
• Failover Only (Active/Passive): All traffic traverses a single primary path while the secondary path sits completely idle. The secondary path activates only when the primary path fails entirely.

Command-Line MPIO Diagnostics: Linux and Windows

• Linux Device-Mapper Multipath (multipath -ll):

  multipath -ll
  

  Output Analysis:

  mpatha (36005076380818123400000000000010a) dm-2 IBM,2145
  size=2.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
  `-+- selector='round-robin 0' priority=50 status=active
    |- 1:0:0:1 sdb 8:16 active ready running
    `- 2:0:0:1 sdc 8:32 active ready running
  

  Verify that path status reports active ready running. If a path reports failed faulty, inspect physical cabling and switch zoning for that specific HBA channel.
• Windows Server CLI (mpclaim):
  • Query configured MPIO disks:

    mpclaim -s -d
    

  • Inspect detailed path status for a specific disk number:

    mpclaim -s -d 1
    

    Output Analysis: Confirms whether the load balance policy is Round Robin or Failover Only, and identifies the operational state (Active/Optimized vs Standby or Failed) of each underlying DSM hardware path.