14.2 IP Routing and Core Infrastructure Service Failures

14.2 IP Routing and Core Infrastructure Service Failures

Network Layer Triage Directive: Once physical link carrier and Layer 2 data link integrity are verified, communication failures migrate to Layer 3 (Internet Protocol) routing parameters and mission-critical infrastructure services. An erroneous subnet mask bit, a rogue DHCP server, or a clock drift of mere minutes between an application node and a domain controller can completely halt enterprise computing operations. Structured troubleshooting requires isolating local subnet connectivity from remote routing, verifying service daemon states, and interrogating operational protocol handshakes.

Enterprise servers rarely operate in isolated silos. They depend continuously on hierarchical routing fabrics and core foundational services—specifically the Domain Name System (DNS), Dynamic Host Configuration Protocol (DHCP), and Network Time Protocol (NTP). A failure in any of these mechanisms produces symptoms that superficially resemble hardware or software crashes.

+-----------------------------------------------------------------------------+
|              Enterprise IP Routing & Core Service Triage Flow               |
|                                                                             |
|   [ 1. Local Stack & Addressing ]                                           |
|         │  (Loopback 127.0.0.1, local IP assignment, Gratuitous ARP check)  |
|         ▼                                                                   |
|   [ 2. Local Subnet & Gateway Reachability ]                                |
|         │  (Subnet mask boundary, ARP table, default gateway ping)          |
|         ▼                                                                   |
|   [ 3. Host Routing Table & Multi-Homed Paths ]                             |
|         │  (route print / ip route show, prevent asymmetric firewall drops) |
|         ▼                                                                   |
|   [ 4. Core Name Resolution (DNS) ]                                         |
|         │  (nslookup, dig, /etc/resolv.conf, flush resolver cache)          |
|         ▼                                                                   |
|   [ 5. Dynamic Addressing & Timing (DHCP / NTP) ]                           |
|            (APIPA 169.254.x.x fallback, DHCP relay, Kerberos clock skew)    |
+-----------------------------------------------------------------------------+

---

IP Addressing and Subnet Mask Configuration Failures

Misconfigurations in static IP assignments or automated lease configurations directly prevent servers from communicating within their assigned subnets or finding paths to remote networks.

Duplicate IP Address Conflicts and Gratuitous ARP Detection

When two active network adapters on the same broadcast domain share an identical IPv4 address, both hosts experience intermittent or complete loss of network connectivity:

• Gratuitous ARP (GARP) Detection Mechanism: Operating systems adhere to RFC 5227 (IPv4 Address Conflict Detection). When a server interface initializes with a static IP or acquires a lease, it broadcasts an ARP Probe and a Gratuitous ARP frame (an ARP request where the sender IP and target IP are identical, sent to the broadcast MAC ff:ff:ff:ff:ff:ff). If another host on the network already claims that IP, it responds with an ARP reply containing its own MAC address.
• Operating System Defensive Actions: Upon receiving a conflicting ARP reply, the initializing host immediately de-configures or unplumbs the IP address to prevent corrupting network traffic. In Windows Server, the interface is disabled, an alert prompt appears, and the System Event Log records Event ID 4199 (The system detected an address conflict for IP address 192.168.10.50 with the system having network hardware address 00:1A:4B:9C:2D:11). In Linux, the kernel logs warnings to /var/log/messages or journalctl -k (kernel: eth0: duplicate address detected!), and tools like arping -D -I eth0 <IP> return positive conflict signals.
• Network Fabric Impact: Before the conflict is cleared, upstream switches experience severe MAC address table thrashing (CAM flapping). As packets with the same IP originate from two different switch ports, the switch constantly updates its forwarding table, causing erratic packet delivery and dropped TCP sessions for both hosts.

Subnet Mask Miscalculation and Boundary Errors

The subnet mask (e.g., 255.255.255.0 or /24 in CIDR notation) defines the boundary separating the Network ID from the Host ID. A mathematical error in configuring this mask disrupts local vs. remote decision-making:

• Subnet Mask Configured Too Narrow (e.g., /24 instead of /23):
  • Assume a corporate subnet spans 172.16.10.0/23 (valid host range: 172.16.10.1 through 172.16.11.254).
  • If a technician configures a server with 172.16.10.50 using a /24 mask (255.255.255.0), the server believes its local subnet only extends from 172.16.10.1 to 172.16.10.254.
  • When this server attempts to communicate with a database host on 172.16.11.20, the server erroneously concludes that the target resides on an external, remote network. Instead of sending a direct ARP request onto the local wire, the server forwards the packet to its default gateway router. If the router does not permit routing between subnets on the same physical interface or hairpins the traffic, communication fails.
• Subnet Mask Configured Too Broad (e.g., /23 instead of /24):
  • Assume a server sits on 192.168.1.0/24, but is accidentally configured with a /23 mask (255.255.254.0), making the server believe the subnet covers 192.168.0.0 through 192.168.1.255.
  • When attempting to contact 192.168.0.15 (which is actually a completely different physical subnet located behind a router), the server assumes the host is local and repeatedly broadcasts ARP requests on the local segment. Because no local host responds to the ARP requests, the server reports Destination host unreachable, never attempting to forward packets to its default gateway.

---

Default Gateway Connectivity and Routing Anomalies

The Default Gateway is the local router interface that forwards packets destined for IP addresses located outside the host's local subnet. Troubleshooting gateway connectivity follows a strict step-ladder methodology:

+-----------------------------------------------------------------------------+
|                 Layer 3 Diagnostic Step-Ladder Methodology                  |
|                                                                             |
|   [ Step 1 ]  ping 127.0.0.1          ──> Validates local TCP/IP stack      |
|   [ Step 2 ]  ping <Local_Host_IP>    ──> Validates NIC driver & binding    |
|   [ Step 3 ]  ping <Local_Neighbor>   ──> Validates local Layer 2 broadcast |
|   [ Step 4 ]  ping <Default_Gateway>  ──> Validates Layer 3 router link     |
|   [ Step 5 ]  ping <Remote_IP_Target> ──> Validates end-to-end IP routing  |
|   [ Step 6 ]  ping <Remote_FQDN>      ──> Validates DNS name resolution     |
+-----------------------------------------------------------------------------+

Primary Default Gateway Failure Modes

• Missing Default Gateway: The server can communicate seamlessly with all systems within its local Layer 2 broadcast domain, but cannot reach any external subnets, corporate branches, or cloud infrastructure. Running ipconfig (Windows) or ip route show (Linux) reveals that the Default Gateway field is blank.
• Incorrect Gateway IP Address: The gateway address is configured with a typo (e.g., 10.0.1.254 instead of 10.0.1.1) or an IP belonging to an entirely different subnet. Because the server cannot ARP for the invalid gateway or sends packets to an unresponsive IP, all routed traffic is dropped.
• Default Gateway Offline / VRRP-HSRP Failure: In high-availability environments, redundant routers share a virtual IP (VIP) using First Hop Redundancy Protocols (FHRP) like HSRP (Hot Standby Router Protocol) or VRRP (Virtual Router Redundancy Protocol). If the active router fails and split-brain occurs, or if gratuitous ARPs fail to update the switch CAM tables during failover, the gateway VIP becomes unreachable.

---

Host Routing Table and Asymmetric Routing on Multi-Homed Servers

Enterprise servers frequently integrate multiple physical or virtual network adapters. A multi-homed server might possess one NIC dedicated to client application traffic and a second NIC connected to an isolated storage, backup, or out-of-band management network.

Inspecting Host Routing Tables

Administrators inspect routing tables using standard CLI utilities:

• Windows Server: route print or PowerShell Get-NetRoute.
• Linux: ip route show or netstat -rn.

# Windows Server Routing Table Inspection
C:\\> route print -4
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0       10.10.1.1      10.10.1.50      25
          0.0.0.0          0.0.0.0      172.16.1.1     172.16.1.50      25  <-- CONFLICT!
       10.10.1.0    255.255.255.0         On-link       10.10.1.50     281
      172.16.1.0    255.255.255.0         On-link      172.16.1.50     281
===========================================================================

The Multi-Default-Gateway and Asymmetric Routing Trap

A critical, widespread administrative error on multi-homed servers is configuring a default gateway on more than one network adapter:

+-----------------------------------------------------------------------------+
|                        Asymmetric Routing Drop Loop                         |
|                                                                             |
|   [ Remote Corporate Client ] (192.168.100.15)                              |
|         │                                                                   |
|         │ 1. TCP SYN sent to Server Management IP (10.10.1.50)              |
|         ▼                                                                   |
|   [ Perimeter Firewall A ] (Records State: SYN seen on Path A)              |
|         │                                                                   |
|         ▼                                                                   |
|   [ Server NIC 1 ] (10.10.1.50)                                             |
|         │                                                                   |
|         │ Server processes packet; replies with TCP SYN-ACK.                |
|         │ Problem: Host routing table picks Default Gateway on NIC 2!       |
|         ▼                                                                   |
|   [ Server NIC 2 ] (172.16.1.50) ──> Exits via Gateway 172.16.1.1           |
|         │                                                                   |
|         ▼                                                                   |
|   [ Perimeter Firewall B ] (Path B)                                         |
|         │                                                                   |
|         ▼                                                                   |
|   * Firewall B Drops Packet: Stateful inspection sees SYN-ACK without       |
|     prior SYN on this path. Connection hangs and times out!                 |
+-----------------------------------------------------------------------------+

1. Ingress: A remote client on 192.168.100.15 initiates an SSH connection to the server's management interface on NIC 1 (10.10.1.50). The packet traverses the corporate firewall, which creates an active connection state entry expecting a matching reply on that same interface.
2. Egress Misdirection: The server receives the TCP SYN packet on NIC 1. However, when the TCP/IP stack generates the TCP SYN-ACK response, it consults its internal routing table. If NIC 2 possesses a default gateway with an equal or lower routing metric, the server forwards the return packet out of NIC 2 (172.16.1.50).
3. Stateful Firewall Drop: The return packet arrives at a different firewall interface or an entirely different transit firewall. The stateful firewall inspects the packet, detects an out-of-sequence TCP SYN-ACK with no preceding SYN in its state table, and immediately discards the packet as an invalid or spoofed connection.
4. Remediation: A server must have exactly ONE default gateway. All secondary network adapters must have their default gateway fields left completely blank. To reach specific remote networks (such as backup servers or management consoles) via secondary NICs, administrators must configure explicit static persistent routes:
   • Windows: route -p add 192.168.100.0 mask 255.255.255.0 10.10.1.1
   • Linux: ip route add 192.168.100.0/24 via 10.10.1.1 dev eth0 (and persist in Netplan or network-scripts).

---

Core Infrastructure Service Resolution: Domain Name System (DNS)

The Domain Name System (DNS) translates human-readable Fully Qualified Domain Names (FQDNs) into routable IP addresses. When DNS resolution fails, users and applications report that the server is "completely down," even when underlying IP routing is flawless.

Diagnosing DNS Resolution vs. Transport Failures

The classic signature of a DNS failure is unambiguous: Pinging the destination IP address succeeds, but pinging the destination FQDN fails.

$ ping 192.168.1.50
64 bytes from 192.168.1.50: icmp_seq=1 ttl=64 time=0.452 ms  <-- ROUTING WORKS

$ ping db.domain.local
ping: cannot resolve db.domain.local: Unknown host          <-- DNS FAILURE

Essential Diagnostic Utilities: nslookup, dig, and Resolve-DnsName

Administrators must never rely on ping alone to test DNS because ping evaluates multiple name resolution mechanisms (including local /etc/hosts files, NetBIOS, and LLMNR). Use dedicated DNS interrogation tools:

• nslookup (Cross-Platform): Queries DNS servers interactively or directly.
  • Command: nslookup db.domain.local 10.0.0.1 (queries the specific DNS server 10.0.0.1 directly, bypassing OS caching).
• dig (Domain Information Groper - Linux/UNIX): Provides deep protocol diagnostics, query timings, and explicit DNS header flags.
  • Command: dig @10.0.0.1 db.domain.local A
  • Status Codes: NOERROR (successful query); NXDOMAIN (non-existent domain; record does not exist); SERVFAIL (DNS server encountered an internal zone database or upstream forwarding failure); REFUSED (DNS server policy refused query processing).
  • Command: dig +trace db.domain.local (executes an iterative trace starting from the root name servers down to the authoritative zone).
• PowerShell Resolve-DnsName (Windows Server): Advanced cmdlet returning typed record objects.
  • Command: Resolve-DnsName -Name db.domain.local -Server 10.0.0.1 -Type A

Client-Side DNS Configuration Triage

When DNS queries fail, inspect the host resolver configuration:

• Linux Resolver Configuration: Inspect /etc/resolv.conf. Ensure valid nameserver IP directives exist. In modern Linux distributions running systemd-resolved, inspect status via resolvectl status or systemd-resolve --status. Verify /etc/nsswitch.conf contains hosts: files dns to ensure proper query prioritization.
• Primary vs. Secondary DNS Resolver Failures: Client operating systems query the primary configured DNS server first. If the primary DNS server is powered off or unreachable, the operating system waits for a timeout (typically 2 to 5 seconds per query) before failing over to the secondary DNS server. While name resolution ultimately succeeds, this introduces massive latency spikes across applications. If users report that "everything takes 5 seconds to load," check whether the primary DNS server is offline.
• Stale DNS Resolver Caches: When a server's IP address is modified in DNS, client machines continue attempting connections to the old IP until the record's Time-To-Live (TTL) expires in the local resolver cache.
  • Remediation (Windows): Execute ipconfig /flushdns (inspect current cache with ipconfig /displaydns).
  • Remediation (Linux): Execute resolvectl flush-caches (or systemd-resolve --flush-caches).

---

Dynamic Host Configuration Protocol (DHCP) Failures

DHCP automates the assignment of IPv4/IPv6 addresses, subnet masks, default gateways, and DNS servers using the four-step DORA sequence: Discover, Offer, Request, Acknowledge.

DHCP Scope Exhaustion and APIPA Fallback

• Scope Exhaustion (100% Full Pool): When the number of active clients on a subnet exceeds the available IP addresses configured in the DHCP server's scope pool, incoming DHCPDISCOVER broadcast packets are ignored because no free leases remain.
• Automatic Private IP Addressing (APIPA / Link-Local): When a client server configured for DHCP transmits four DHCPDISCOVER broadcasts without receiving a DHCPOFFER, the operating system gives up and automatically assigns itself an APIPA address from the reserved block 169.254.0.1 to 169.254.255.254 with a subnet mask of 255.255.0.0 (/16) (RFC 3927).
• Operational Diagnostic: If a server displays an IP address starting with 169.254.x.x, it has completely failed to communicate with a DHCP server. While the server can communicate with other APIPA hosts sharing the exact same physical wire, it cannot communicate with its default gateway, enterprise DNS, or any remote subnets.

Rogue DHCP Servers

A rogue DHCP server is an unauthorized device (such as an improperly configured test server, lab router, or attacker) connected to a production broadcast domain that responds to client DHCPDISCOVER broadcasts faster than the legitimate enterprise DHCP server.

• Consequences: The rogue server distributes invalid IP parameters, such as a nonexistent default gateway (causing a Denial of Service) or its own IP as the gateway and DNS server (orchestrating a Man-in-the-Middle - MitM attack).
• Mitigation: Enable DHCP Snooping on enterprise managed switches. DHCP Snooping classifies switch ports as trusted (connected to legitimate DHCP servers and upstream trunks) or untrusted (connected to end hosts). The switch drops any DHCPOFFER or DHCPACK packets arriving on untrusted ports.

DHCP Relay Agent (ip helper-address) Failures

Because DHCP discovery relies on Layer 2 broadcast frames (255.255.255.255), routers block these packets by default at the subnet boundary. In enterprise data centers where a centralized DHCP server services multiple subnets, the local router or Layer 3 switch must act as a DHCP Relay Agent:

• The relay agent intercepts the local broadcast DHCPDISCOVER packet, encapsulates it into a unicast IP packet, and forwards it directly to the centralized DHCP server's IP address.
• Failure Mode: If a network engineer provisions a new VLAN on a Layer 3 switch but omits the relay directive (e.g., the Cisco IOS command ip helper-address <DHCP_Server_IP>), every host on that new VLAN will fail to obtain an IP lease and will drop into APIPA (169.254.x.x).

---

Network Time Protocol (NTP) Synchronization and Kerberos Skew

Network Time Protocol (NTP, UDP port 123) synchronizes computer clocks across distributed networks using a hierarchy of time sources (Stratum 0 atomic clocks, Stratum 1 primary time servers, down to Stratum 2/3 enterprise servers).

The Kerberos 5-Minute Time Skew Rule (KRB_AP_ERR_SKEW)

In enterprise networks utilizing Microsoft Active Directory, FreeIPA, or MIT Kerberos, time synchronization is not merely a logging convenience—it is a critical cryptographic dependency:

+-----------------------------------------------------------------------------+
|                   Kerberos Time Skew Failure Condition                      |
|                                                                             |
|   [ Domain Controller / KDC ]                   [ Enterprise Member Server ]|
|   - Current UTC Time: 14:00:00                  - Current UTC Time: 14:06:00|
|          │                                                  │               |
|          │ Client presents Kerberos ticket timestamped 14:06│               |
|          │<─────────────────────────────────────────────────┤               |
|          │                                                                  |
|          ▼                                                                  |
|   * KDC Evaluates Skew: |14:06:00 - 14:00:00| = 6 minutes (360 seconds).     |
|     Threshold is 5 minutes (300 seconds).                                   |
|     Result: KDC Rejects Authentication! Returns: KRB_AP_ERR_SKEW            |
+-----------------------------------------------------------------------------+

• To prevent replay attacks (where an eavesdropper captures an encrypted Kerberos ticket off the network and retransmits it later to gain unauthorized access), Kerberos tickets contain an explicit generation timestamp.
• The Kerberos authentication protocol specification enforces a default maximum allowable clock drift of 5 minutes (300 seconds) between the client, the application server, and the Key Distribution Center (KDC / Domain Controller).
• The Failure: If an application server's clock drifts more than 5 minutes ahead or behind the Domain Controller, all Kerberos authentication attempts fail instantaneously. Users cannot authenticate, scheduled tasks fail, and accessing SMB network shares returns The system detected an authentication error. Clock skew too great.
• Event Logs: In Windows Server, the System Event Log records Kerberos Event ID 4 returning error code 0x25 (KRB_AP_ERR_SKEW).

Verifying and Restoring NTP Synchronization

Administrators must verify time synchronization status via the command line:

• Windows Server (w32time service):
  • Query Status: w32tm /query /status (displays reference source, stratum, precision, and last synchronization time).
  • Query Peer Configuration: w32tm /query /peers
  • Force Instant Resynchronization: w32tm /resync /force
• Linux (chrony / NTP daemons):
  • Chrony Status: chronyc tracking (displays reference ID, stratum, system time offset, and root delay).
  • Chrony Sources: chronyc sources -v (displays operational state of configured upstream NTP servers; an asterisk * denotes the current synchronized source).
  • Legacy NTP: ntpq -p (inspects peer jitter, offset, and reachability).

The Blueprint Command Set for Network Triage

SK0-005 names a specific list of tools under network connectivity troubleshooting. Knowing which layer each one interrogates is what makes a scenario answerable.

Sequencing: Work the Layers in Order

The blueprint's tools-and-techniques bullet begins with physical checks for a reason. The efficient sequence is:

1. Check link lights and confirm power on the NIC and switch port — no link means nothing above Layer 1 matters.
2. Verify cable integrity and appropriate cable selection — test with a certified cable tester where available, confirm the correct category and type for the speed (Cat 6/6A for 10GBASE-T, correct single-mode vs. multimode fiber and matching transceiver wavelength), confirm the run is within distance limits, and swap in a known-good cable as the definitive test. Bad cables and wrong cable selection are named causes precisely because they masquerade as NIC or switch faults.
3. Confirm local IP configuration (ipconfig /all, ip addr) — this is where improper IP configuration, a wrong subnet mask, and incorrect DHCP information surface. An APIPA address means the DHCP server, relay, or the path to it failed.
4. Ping the loopback, then the local interface, then the gateway, then an off-subnet address, then a name — the first step that fails localizes the fault.
5. Trace the path (tracert/traceroute) when off-subnet destinations fail but the gateway responds.
6. Separate name resolution from transport — if pinging an IP works and pinging the name does not, the fault is DNS, not connectivity.

IPv4 vs. IPv6 Misconfiguration

Dual-stack hosts prefer IPv6 over IPv4 by default. If IPv6 is enabled on the server but the network path, firewall rules, or destination service is IPv4-only, the client attempts IPv6 first and stalls until it falls back — presenting as intermittent, slow, or name-dependent connectivity that "works sometimes." Conversely, Microsoft does not recommend disabling IPv6 on Windows Server — it is a mandatory part of the OS and several components, including parts of failover clustering and DirectAccess, depend on it. The correct remediation is to fix the IPv6 path or adjust prefix policy precedence so IPv4 is preferred, not to unbind the protocol stack.