12.1 Network Connectivity, Routing & Name Resolution Diagnostics
Key Takeaways
Check the interface and expected carrier and negotiation. With bonding, inspect member and aggregator state as well as overall link state.
Use the actual route, neighbor and address family for the failed destination. DHCP, duplicate IPs, a bad prefix, and an incorrect gateway can produce different evidence.
An IPv4 DF Echo probe can help test packet size along a path. A successful small probe gives a passing size, not an exact maximum.
Listening sockets and established connections require different ss selections. Persistent CLOSE_WAIT indicates that the peer started closing and the local endpoint has yet to finish.
Compare glibc NSS and direct DNS queries when they differ, and test IPv4 and IPv6 separately when investigating a dual-stack incident.
12.1 Network Connectivity, Routing & Name Resolution Diagnostics
Start with the failed operation and compare it with something that works. A working console narrows the problem differently from a successful connection to a numeric address. Change one condition at a time and verify that the same client and service can complete the operation afterward.
The Layered Network Troubleshooting Model
| What to check | First evidence |
|---|---|
| Link and negotiation | Interface flags, carrier and speed. Compare ip link, ethtool and switch or virtual port status. |
| Addresses and routes | Compare address, prefix, gateway, route, neighbor information and address family. |
| Transport and firewall | Check listener address and port with ss. Compare the permitted traffic and relevant firewall rule or log under the host's configured policy. |
| DNS and application | Compare application-style name resolution with a direct DNS query. Then inspect TLS or protocol behavior as needed. |
Start from the symptom. If a cable was recently replaced, carrier and negotiation are efficient first checks. If numeric-address tests work but the application reports name resolution failure, compare DNS and local resolver behavior before changing routing.
Layer 1 & Layer 2: Physical & Data Link Layer Diagnostics
ip link show dev eth0 can report UP in the interface flags and a separate operational state. The UP flag shows administrative enablement. LOWER_UP means the driver reports lower-layer carrier. Neither alone proves that higher layers can carry desired traffic. Check physical connections, switch membership or virtual network attachment when appropriate.
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP
link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
ethtool eth0 on a supported interface can show advertised modes, negotiated speed, duplex and link detection. Use ethtool -S eth0 or ip -s link show dev eth0 to compare changes in available counters during the problem. Typical symptoms of an older Ethernet duplex mismatch include asymmetric errors and poor performance under load. The meaning and availability of counters vary by driver. Correct the agreed settings on both ends, and avoid assuming failed negotiation always falls back to half duplex.
Bonding, MAC addresses, and misconfiguration
Inspect a Linux bond with cat /proc/net/bonding/bond0. A bond in active-backup mode selects one active member and can switch when its health policy detects failure. Mode 4 (802.3ad) aggregates links with compatible switch configuration and LACP. For example, if the aggregate shows one healthy member and a second member reports carrier down, inspect the second link and switch port before changing the hashing policy. If both have carrier but fail to join the same aggregator, compare actor/partner information and switch configuration. miimon is a carrier monitoring option. Check the configured mode and validation method before concluding that no failure detection is available.
If a switch reports an unexpected MAC address or movement between ports, compare the interface's current address, the intended profile, and switch or virtual switch records. A bond failover, VM move or intentionally assigned address can explain a legitimate change. An unauthorized spoofed address or conflicting configuration requires validating and restoring the intended assignment through the platform's controls.
Layer 3: Addressing, DHCP, Neighbors & Routes
ip -br address show gives a quick inventory. Confirm the address and prefix on the affected interface. If a host meant to use 192.168.1.50/24 instead uses /16, it may treat remote addresses such as 192.168.22.30 as on-link and try neighbor discovery instead of using its gateway. IP scope global in Linux can include private addresses and is not proof of public reachability.
If the interface should receive DHCP configuration, check the active profile and client logs for an offer, acknowledgment and leased parameters. Inspect address, route and resolver settings together. A client can have a lease and still be unreachable because it is on the wrong network or received an incorrect default route or DNS server. If there is no offer, investigate the local link or VLAN, DHCP relay and address pool, using approved infrastructure evidence. Check the lease acquisition before turning a dynamic deployment into a static one.
Duplicate IPv4 addresses
ip neigh show reports the neighbor cache. A stale entry is not automatically a fault. Compare recent address changes, DHCP leases, static assignments and observed responders. On a controlled network, iputils arping -D -I eth0 -c 3 192.168.1.50 can probe for a duplicate IPv4 address on the same link before assigning it. With iputils, exit status zero in duplicate detection mode means that no reply was received during the probe. A reply gives evidence to investigate, while silence does not rule out an intermittent conflict. Verify the installed implementation and interface. IPv6 uses Neighbor Discovery and has its own duplicate address detection.
Routing and gateway checks
ip route show
ip route get 192.0.2.20
One possible result is 192.0.2.20 via 192.168.1.1 dev eth0 src 192.168.1.50. Check the selected next hop and source before editing. If there is no default route, a specific route can still reach an off-link network. Prefix selection, policy rules and metrics influence the result. Where usable routes in the same table and prefix differ only by metric, a lower metric is preferred. In a multihomed case compare forward and reverse paths and any stateful filtering. Strict Linux reverse-path filtering can drop an arriving IPv4 packet if the best return path to its source uses a different interface. Inspect the applicable all and per-interface settings and the intended routing policy before changing them.
A server that does not answer an echo probe may still accept its application traffic. Compare a known destination and allowed port. When the service is listening but clients cannot reach it, review the relevant interfaces, zones or chains, rule counters and firewall logs. Determine which direction and flow is affected before adjusting a rule. A successful port connection still leaves application and authentication checks.
Path MTU Mismatches & Packet Fragmentation
Path maximum transmission unit (PMTU) is the size limit along a path. Encapsulation can reduce the usable packet size. An IPv4 router unable to forward a packet with the Don't Fragment (DF) bit set due to its outgoing MTU can send an ICMP Destination Unreachable, Fragmentation Needed response. If that feedback is filtered, small packets can work while larger transfers stall. For IPv6, Path MTU Discovery uses ICMPv6 Packet Too Big messages.
For an approved remote peer at 192.0.2.20 in a lab, compare two IPv4 Echo payloads:
ping -4 -c 2 -M do -s 1472 192.0.2.20
ping -4 -c 2 -M do -s 1392 192.0.2.20
The minimum IPv4 header and ICMP Echo header add 28 bytes, so these requests are 1500 and 1420 bytes in this example. A successful smaller probe shows that this size passed. It does not establish the largest supported size. An explicit 1420-byte limit from a fragmentation-needed message is stronger evidence. Use tracepath, interface configuration and permitted diagnostics to check whether the issue is local, on the path, or due to filtering. Fix the relevant tunnel, interface or policy in coordination with its owner and retest the original application operation.
Layer 4: Transport Layer & Socket Diagnostics
| Command | Evidence |
|---|---|
ss -tuln | Listening TCP and UDP sockets with numeric addresses and ports. |
ss -tulpn | The same selection with process information where permissions allow. |
ss -tan | TCP sockets across states, including listeners and established connections. |
ss -t state close-wait | TCP sockets where the peer has sent a FIN and the local endpoint has not finished closing. |
An application bound to 127.0.0.1:8080 can work on the server and be unreachable on its external address. Check the service's intended binding rather than opening additional ports first. In CLOSE_WAIT, the peer initiated closure and the local endpoint has yet to finish closing. A persistent, growing count for the same application suggests that it is retaining sockets. Check process ownership and application logs before concluding that this alone caused an outage. TIME_WAIT is a normal part of TCP shutdown that helps handle delayed segments.
Application & Name Resolution Diagnostics
As in the configuration section, /etc/nsswitch.conf selects and orders sources for NSS-aware applications, while tools such as dig ask DNS. On a system with hosts: files dns and ordinary status actions, a successful hosts-file result can differ from the DNS answer. Check the relevant lookup method and address family.
getent ahostsv4 db01.corp.example
dig @10.0.0.2 db01.corp.example A
resolvectl status
resolvectl query db01.corp.example
Use the named DNS server only when it is the approved resolver for the example domain. The resolvectl commands apply when systemd-resolved is running. On a host using its stub, /etc/resolv.conf may point to a file listing 127.0.0.53. Another host may use a file with upstream nameservers directly. Compare the active resolver, link-specific routing domains and source configuration. A direct query that works can help isolate a missing per-link DNS route without proving every other part of the client path is functioning. For a public delegation, dig +trace can walk DNS delegation when the network allows direct server queries. It does not reproduce an internal split-DNS path.
Use curl -v https://api.corp.example on a service in your care to inspect connection and TLS evidence. If the service uses a private CA, verify the expected CA and supply it under the site's trust policy. Keep certificate verification enabled.
Dual-Stack IPv4 / IPv6 Troubleshooting
Test the families separately when one path may be broken. Check IPv4 and IPv6 addresses, routes and neighbor information. In IPv6, Router Advertisements can supply default-router information and prefixes for stateless address autoconfiguration. A link-local address is scoped to a link. For example, ping -6 -c 2 fe80::1%eth0 identifies the interface when testing that scope. Link-local reachability alone does not show that a routable IPv6 address or a default route exists.
Compare A and AAAA results and try the same authorized service using curl -4 and curl -6. If the service works only on IPv4, inspect the IPv6 route, firewall and listener, and check whether the advertised AAAA address is current. Applications can differ in how they race or fall back between families, so verify the original client after the repair.
On an approved IPv4 lab path to 192.0.2.20, small pings and interactive SSH work but bulk transfers stall. Which test directly compares IPv4 Echo probes sized for 1500 and 1420 bytes without fragmentation?
Compare local CRC counters with ethtool
Send two ARP requests for the local gateway
Compare ping -4 -M do with -s 1472 and -s 1392 to the same remote peer
List local listening TCP ports with ss
A host runs systemd-resolved and uses its 127.0.0.53 stub. Querying the corporate DNS server directly succeeds, but an internal name fails through the host resolver. Which command shows the resolver’s per-link DNS configuration?
resolvectl status
ip neigh show
ss -tuln
journalctl -k -b
A monitored application has a persistent, growing number of TCP sockets in CLOSE_WAIT. What does the state show for those sockets?
The local endpoint has finished closing and is waiting for delayed segments
The remote peer sent a FIN and the local endpoint has yet to finish closing
The Ethernet link negotiated half duplex
The firewall dropped a Path MTU message
Sections you finish are checked off in the contents.