3.3 Storage Architectures and SAN Protocols

3.3 Storage Architectures and SAN Protocols

Quick Answer: Enterprise enterprise storage models divide into Direct-Attached Storage (DAS), Network-Attached Storage (NAS), and Storage Area Networks (SAN). While NAS operates at the file level using NFS (Linux/UNIX) or SMB (Windows) over general LANs, a SAN provides block-level storage over dedicated high-speed fabrics. Prominent SAN protocols include iSCSI (mapping SCSI blocks into TCP/IP over port 3260, identified via IQNs and secured with CHAP), Fibre Channel (FC) (operating at 16/32/64 Gbps over dedicated optical fabrics using WWPNs and a three-step FLOGI/PLOGI/PRLI login), and Fibre Channel over Ethernet (FCoE). SAN security requires Fabric Zoning on switches and LUN Masking on array controllers. Multipath I/O (MPIO) ensures high availability via active/active or active/passive path failover.

Understanding how server operating systems interact with external storage fabrics is fundamental for the CompTIA Server+ (SK0-005) certification. Systems engineers must master network storage encapsulation, identify initiator and target naming conventions, implement access control boundaries, and configure multipathing load balancing.

---

Enterprise Storage Topologies: DAS vs. NAS vs. SAN

+-------------------------------------------------------------------------+
|                   Enterprise Storage Architecture Spectrum              |
|                                                                         |
|   DAS (Direct-Attached Storage):                                        |
|   [Server Node] ===(Internal SAS / SFF-8644 Cables)===> [JBOD Shelf]   |
|   * Access Level: Block-level | Scope: Isolated to single server host   |
|                                                                         |
|   NAS (Network-Attached Storage):                                       |
|   [Clients / Servers] ---(Standard TCP/IP LAN: SMB/NFS)---> [NAS Head]  |
|   * Access Level: File-level (Files & Folders) | Shared across LAN      |
|                                                                         |
|   SAN (Storage Area Network):                                           |
|   [Servers (Initiators)] ===(Fibre Channel / iSCSI Fabric)===> [SAN]    |
|   * Access Level: Block-level (Raw LUNs presented as local disks)       |
+-------------------------------------------------------------------------+

Direct-Attached Storage (DAS)

• Architecture: Storage drives reside either physically inside the server chassis or in an adjacent external drive enclosure (such as a 2U or 4U SAS JBOD shelf) wired directly to the server's Host Bus Adapter using external mini-SAS HD cables (e.g., SFF-8644).
• Access Level: Block-level storage. The server operating system formats the volume with a native local file system (e.g., NTFS, ext4, XFS).
• Trade-offs: Delivers high throughput, low latency, and low cost. However, storage capacity is isolated in a dedicated hardware silo; it cannot be dynamically pooled, carved, or shared across multiple independent physical server clusters.

Network-Attached Storage (NAS)

• Architecture: A dedicated, specialized file-serving appliance connected to the standard enterprise Local Area Network (LAN). The NAS appliance manages its own internal disks, RAID sets, and local file systems.
• Access Level: File-level storage. The client or server operating system mounts a shared directory across the network. File access is abstracted: client applications issue file-oriented requests (open(), read(), write(), close()) rather than raw disk sector block addresses.
• Protocols:
  • Network File System (NFS): The native network file sharing protocol for Linux, UNIX, and VMware ESXi environments. NFSv3 operates over TCP/UDP port 2049 (using RPC services); NFSv4 establishes stateful, secure TCP sessions on port 2049 with integrated ACLs and Kerberos authentication.
  • Server Message Block (SMB): The standard network file sharing protocol in Windows Server environments (historically evolved from CIFS - Common Internet File System). SMB 3.0/3.1.1 supports transparent failover, end-to-end AES encryption, and SMB Direct over Remote Direct Memory Access (RDMA).

Storage Area Network (SAN)

• Architecture: A specialized, high-performance, dedicated data network engineered specifically to interconnect multiple server compute nodes (termed Initiators) to centralized enterprise storage arrays (termed Targets).
• Access Level: Block-level storage. To the host operating system, a volume presented across a SAN appears as an unformatted, raw physical SCSI hard drive. The host operating system creates its own local file system or clustered file system (e.g., VMware VMFS, Windows CSV) directly onto the storage blocks.

---

Internet Small Computer Systems Interface (iSCSI)

iSCSI maps standard SCSI-3 block commands, data, and status messages into standard TCP/IP packets, enabling enterprise block storage networks to run over commodity Ethernet switches and copper/fiber cabling.

+-------------------------------------------------------------------------+
|                        iSCSI Architectural Stack                        |
|                                                                         |
|   +-----------------------------------------------------------------+   |
|   | Operating System SCSI Block Layer (Local Disk Abstraction)      |   |
|   +-----------------------------------------------------------------+   |
|   | iSCSI Layer: Encapsulates SCSI-3 CDBs into iSCSI PDUs           |   |
|   +-----------------------------------------------------------------+   |
|   | TCP Layer: Reliable stream transport (TCP Port 3260)            |   |
|   +-----------------------------------------------------------------+   |
|   | IP Layer: Packet routing & addressing across storage subnets    |   |
|   +-----------------------------------------------------------------+   |
|   | Ethernet Data Link: Jumbo Frames (MTU 9000) for zero packet loss|   |
|   +-----------------------------------------------------------------+   |
+-------------------------------------------------------------------------+

Operational Mechanics and Port Architecture

• Port Allocation: iSCSI targets listen for incoming connection requests on standard registered TCP port 3260.
• Addressing via IQN: Endpoints in an iSCSI network are uniquely identified via an iSCSI Qualified Name (IQN). An IQN follows a strict standardized naming syntax defined in RFC 3720: $\text{iqn}.\text{yyyy-mm}.\text{naming-authority}:\text{unique-string}$
  • iqn. : Mandatory type prefix.
  • 2026-03. : The year and month when the naming authority registered its domain name.
  • com.storagecorp: : The reversed domain name of the organization.
  • san-target-cluster-pool01 : Unique local device identifier assigned by the storage administrator.
  • Example: iqn.1998-01.com.vmware:esx-prod-node04-7b89f012

Software Initiators vs. Hardware iSCSI HBAs (TOE)

• Software Initiator: The host operating system kernel incorporates a native iSCSI driver stack (e.g., Microsoft iSCSI Initiator, Linux open-iscsi). The server's standard CPU handles all TCP/IP protocol processing, packet sequencing, checksums, and SCSI encapsulation over standard Network Interface Cards (NICs). While inexpensive, heavy I/O consumes measurable host CPU cycles.
• Hardware iSCSI HBA: A dedicated adapter card featuring a specialized TCP Offload Engine (TOE) ASIC and hardware iSCSI processing silicon. The adapter processes the entire network stack and presents a raw SCSI device directly to the server bus, completely offloading the host CPU and lowering latency.

iSCSI Security and Network Optimization

• Challenge-Handshake Authentication Protocol (CHAP): Used to authenticate iSCSI sessions:
  • One-Way (Target) CHAP: The storage target authenticates the host initiator before granting access to storage volumes.
  • Mutual (Bi-Directional) CHAP: The target authenticates the initiator, and the initiator simultaneously authenticates the target, preventing rogue storage arrays or Man-in-the-Middle (MitM) spoofing on the IP network.
• Jumbo Frames (MTU 9000): Standard Ethernet frames cap payload sizes at 1,500 bytes. In block storage, small packets generate massive CPU interrupt overhead. Implementing Jumbo Frames (MTU 9000) end-to-end across server NICs, Ethernet switches, and storage target ports allows 4 KB and 8 KB disk blocks to be transmitted in single frames, slashing packet fragmentation and packet-processing overhead.
• VLAN Isolation: iSCSI storage traffic must always be segregated onto dedicated, non-routable Storage VLANs isolated from general user LAN traffic to avoid latency jitter and broadcast storms.

---

Fibre Channel (FC) and Fibre Channel over Ethernet (FCoE)

Fibre Channel is a purpose-built, high-speed network technology engineered specifically for lossless, deterministic, low-latency block storage delivery in enterprise datacenters.

Physical Fabric and World Wide Names (WWN)

• Signaling Speeds: Modern FC operates at 16 Gbps (Gen 5), 32 Gbps (Gen 6), and 64 Gbps (Gen 7) using optical transceivers (SFP+ / SFP28 / SFP56) and Multi-Mode Fiber (MMF) cabling with LC connectors.
• Buffer-to-Buffer Credits (BB_Credits): Fibre Channel implements physical flow control at the hardware level. A sending port can transmit frames only if it holds positive buffer credits granted by the receiving port. When the receiving buffer fills, credits reach zero, pausing transmission. This guarantees zero frame loss due to network congestion.
• World Wide Names (WWN): Every FC device and physical port is permanently assigned an IEEE 64-bit hexadecimal identifier:
  • World Wide Node Name (WWNN): Identifies the overall physical server chassis or storage array controller node (shared across ports on the device).
  • World Wide Port Name (WWPN): Uniquely identifies an individual physical optical port on a Host Bus Adapter (HBA) or FC switch (e.g., 21:00:00:24:ff:5a:3b:01). Storage administrators bind all zoning and masking rules strictly to the WWPN.

+-------------------------------------------------------------------------+
|                        Fibre Channel Login Phases                       |
|                                                                         |
|   [ Host Server HBA ]                                [ FC Switch Fabric]|
|           |                                                  |          |
|           | ----- (1) FLOGI: Fabric Login -----------------> |          |
|           | <---- Returns 24-bit FCID & Buffer Credits ----- |          |
|           |                                                  |          |
|           |                                      [ Storage Target Port ]|
|           |                                                  |          |
|           | ----- (2) PLOGI: Port Login -------------------> |          |
|           | <---- Exchanges WWPNs & Service Parameters ----- |          |
|           |                                                  |          |
|           | ----- (3) PRLI: Process Login -----------------> |          |
|           | <---- Initializes SCSI-3 Upper Layer Protocol -- |          |
|           |                                                  |          |
|           | ===== Ready for SCSI-3 Block Data Transfers ==== |          |
+-------------------------------------------------------------------------+

The Three Fabric Login Phases

When a server HBA powers on and establishes link light with a Fibre Channel switch, it must successfully complete three sequential login handshakes before I/O can traverse the fabric:

1. Fabric Login (FLOGI): The HBA port connects to the FC switch fabric (at well-known address 0xFFFFFE). The switch assigns the HBA a dynamic 24-bit Fibre Channel ID (FCID) (used for routing frames within the fabric) and negotiates buffer-to-buffer credits.
2. Port Login (PLOGI): The HBA initiates point-to-point communication with the target storage array port. The two endpoints exchange their 64-bit WWPNs and service parameters.
3. Process Login (PRLI): The HBA and storage target port establish the Upper Layer Protocol (ULP) communication parameters—specifically initializing the FCP (Fibre Channel Protocol / SCSI-3) command environment, enabling logical unit discovery.

Fibre Channel over Ethernet (FCoE)

FCoE encapsulates native Fibre Channel frames directly inside standard Ethernet frames (using dedicated EtherType 0x8906), eliminating the need for dedicated optical FC switches and host HBAs. FCoE converges LAN and SAN traffic over shared 10GbE / 25GbE cabling using Converged Network Adapters (CNAs):

• Lossless Ethernet via DCB: Standard Ethernet is inherently lossy (dropping packets when buffers overflow). FCoE mandates Data Center Bridging (DCB) extensions—specifically Priority-based Flow Control (PFC - IEEE 802.1Qbb), which divides an Ethernet link into 8 virtual lanes and pauses only the storage traffic lane during congestion, guaranteeing lossless transport without impacting standard TCP/IP traffic.

---

SAN Access Control: LUN Masking vs. Fabric Zoning

To prevent data corruption in multi-tenant environments, storage fabrics implement a mandatory two-tier security model:

+-------------------------------------------------------------------------+
|                  Two-Tier SAN Access Control Architecture               |
|                                                                         |
|   TIER 1: SAN FABRIC ZONING (Configured on FC Switches)                 |
|   +-----------------------------------------------------------------+   |
|   | FC Switch Fabric enforces hardware isolation:                   |   |
|   | [Zone A]: Initiator HBA WWPN <----> Storage Target Port 1 WWPN  |   |
|   | (Prevents unauthorized hosts from seeing or communicating with  |   |
|   | target ports across the optical fabric).                        |   |
|   +-----------------------------------------------------------------+   |
|                                 |                                       |
|                                 v                                       |
|   TIER 2: LUN MASKING (Configured on Storage Array Controller)          |
|   +-----------------------------------------------------------------+   |
|   | Storage Array Controller restricts individual volume exposure:  |   |
|   | [LUN 0 (OS Boot)] ----> Bound strictly to Initiator WWPN A      |   |
|   | [LUN 1 (Database)] ---> Bound strictly to Initiator WWPN B      |   |
|   | (Prevents Host A from mounting, overwriting, or corrupting      |   |
|   | Host B's file systems).                                         |   |
|   +-----------------------------------------------------------------+   |
+-------------------------------------------------------------------------+

SAN Fabric Zoning (Switch Level)

Zoning is configured on the Fibre Channel switches. It partitions the physical fabric into isolated logical communication groups. Devices in different zones cannot discover each other, route frames, or exchange state-change notifications:

• Soft Zoning (WWPN Zoning): Zones are defined by specifying the World Wide Port Names of member HBAs and storage ports. Soft zoning is the industry best practice because it is flexible: an administrator can unplug a fiber patch cable and move it to a different port on the switch without breaking the zone.
• Hard Zoning (Port Zoning): Zones are bound strictly to physical switch port numbers (e.g., Switch 1, Port 4 and Port 12). While secure against WWPN spoofing, hard zoning breaks immediately if a technician moves a physical cable during datacenter maintenance.
• Single-Initiator Zoning: The universal architectural standard. Every zone must contain exactly one initiator WWPN along with one or more target WWPNs. Multi-initiator zoning causes cross-talk, unnecessary login chatter, and Registered State Change Notification (RSCN) storms that can destabilize host operating systems.

LUN Creation and LUN Masking (Storage Array Level)

• Logical Unit Number (LUN): A logical virtualization slice carved from a large storage pool on the array (e.g., carving a 2 TB LUN from a 50 TB RAID 6 pool) that is presented to initiators as an addressable SCSI disk.
• LUN Masking: Configured directly inside the storage array controller software. LUN masking grants access to specific LUNs based strictly on the initiator's WWPN (in FC) or IQN (in iSCSI). If Host Server 1 and Host Server 2 are zoned to the identical storage controller, LUN masking ensures Host 1 can view only LUN 10, while Host 2 can view only LUN 20. Without LUN masking, both hosts would attempt to initialize and format the same volume simultaneously, causing catastrophic file system corruption.

---

Multipath I/O (MPIO) and High Availability

In enterprise SAN environments, a single cable disconnect, SFP failure, or switch reboot must never sever storage access. Multipath I/O (MPIO) creates redundant physical pathways between server compute nodes and storage arrays.

+-------------------------------------------------------------------------+
|                         MPIO Dual-Fabric Topology                       |
|                                                                         |
|   [ Host Server Node ]                                                  |
|     [HBA Port 1]                          [HBA Port 2]                  |
|          |                                     |                        |
|          v                                     v                        |
|   [ SAN Fabric A Switch ]               [ SAN Fabric B Switch ]         |
|          |                                     |                        |
|          v                                     v                        |
|   [Storage Controller A]                [Storage Controller B]          |
|          +------------------+------------------+                        |
|                             v                                           |
|                     [ Enterprise LUN ]                                  |
|                                                                         |
|   * MPIO Driver coalesces Path A and Path B into a single virtual disk. |
|   * Automatic path failover occurs in milliseconds if Fabric A drops.   |
+-------------------------------------------------------------------------+

Path Architectures and Operational Modes

• Active / Passive (ALUA - Asymmetric Logical Unit Access): The storage array designates one controller path as "Optimal/Active" and the secondary controller path as "Non-Optimal/Passive/Standby". All I/O traverses the active path. If the primary path drops, the MPIO driver executes failover, waking the standby path to service I/O within milliseconds.
• Active / Active: Both physical pathways simultaneously transmit I/O across redundant fabrics, doubling aggregate throughput and eliminating failover delay.

MPIO Load-Balancing Policies

• Round Robin: The MPIO driver rotates incoming I/O requests sequentially across all available active paths (e.g., Request 1 to Path A, Request 2 to Path B, Request 3 to Path A). Ideal for balanced, homogeneous network fabrics.
• Least Queue Depth: The MPIO driver monitors the current workload on each path and dynamically routes the next I/O request to the path with the fewest currently pending transactions. Highly effective when paths possess asymmetric bandwidth or mixed latencies.
• Weighted Paths: The administrator assigns manual priority numbers to specific links, ensuring high-speed optical links absorb primary traffic while lower-bandwidth links serve strictly as backup failover routes.