13.1 Troubleshooting Cloud Network Connectivity & Routing

Troubleshooting Cloud Network Connectivity & Routing

Network connectivity and routing issues represent the most common and critical operational disruptions in cloud environments. Because cloud software-defined networking (SDN) abstracts physical switches, routers, and firewalls into virtual constructs, engineers must master logical packet flow analysis, security perimeter evaluation, and distributed routing diagnostics.

For the CompTIA Cloud+ (CV0-004) examination, candidates must be able to isolate network failures rapidly across virtual private clouds (VPCs), hybrid interconnects, domain name systems, and application load balancers.

---

1. Security Groups vs. Network ACLs Root Cause Triage

In public cloud architectures (such as AWS VPC, Azure Virtual Network NSGs, and Google Cloud VPC firewall rules), network perimeter security is enforced across two distinct abstraction layers: Elastic Network Interface (ENI) firewalls and Subnet boundary filters.

+---------------------------------------------------------------------------------------------------+
|                         SECURITY GROUPS VS. NETWORK ACLS (NACLS)                                  |
|                                                                                                   |
|  Attribute               Security Group (SG)                 Network ACL (NACL)                   |
|  +---------------------+-----------------------------------+------------------------------------+ |
|  | Operating Layer     | Virtual NIC (ENI) level           | Subnet boundary level              |
|  | Statefulness        | Stateful (return traffic allowed) | Stateless (inbound/outbound distinct) |
|  | Rule Types          | Allow rules only (implicit deny)  | Allow AND Deny rules               |
|  | Rule Processing     | All rules evaluated collectively  | Numbered order (lowest number first)| |
|  | Ephemeral Ports     | Handled automatically by state    | Must explicitly permit 1024-65535  |
|  | Default Posture     | Inbound: Deny all; Outbound: Allow| Inbound: Deny all; Outbound: Deny  |
|  |                     | (Default SG allows internal VNet) | (Custom NACLs default to Deny all) |
|  +---------------------+-----------------------------------+------------------------------------+ |
+---------------------------------------------------------------------------------------------------+

Stateful vs. Stateless Mechanics

• Security Groups (Stateful): When an inbound connection is permitted by a Security Group rule (for example, TCP port 443 from 0.0.0.0/0), the cloud hypervisor automatically tracks the TCP connection state. Return traffic sent back to the client is automatically permitted regardless of outbound Security Group rules.
• Network ACLs (Stateless): Subnet NACLs maintain no connection state table. Inbound and outbound traffic streams are evaluated as independent, isolated packets. If an inbound NACL rule permits incoming HTTP traffic on port 80, the returning response packet will be dropped unless the outbound NACL explicitly allows traffic to the client's ephemeral port range (1024–65535 for Linux/Windows clients, or 32768–60999 for specific Linux kernels).

Rule Evaluation Order in NACLs

NACLs evaluate rules in strict numerical ascending order (e.g., Rule 10, Rule 20, Rule 100, Rule *).

• As soon as a packet matches a rule's CIDR, protocol, and port criteria, the ALLOW or DENY action is executed immediately, and no further rules are evaluated.
• A common misconfiguration occurs when an administrator adds a DENY rule (e.g., Rule 50: DENY 198.51.100.0/24) to block malicious traffic, but forgets that Rule 20 already specifies ALLOW 0.0.0.0/0, rendering the block completely ineffective.

---

2. VPC Flow Logs Deep Dive & Analysis

VPC Flow Logs capture metadata regarding IP traffic traversing Elastic Network Interfaces (ENIs) within a cloud virtual network. They do not capture packet payloads, but provide essential Layer 3 and Layer 4 telemetry to pinpoint packet drops.

+---------------------------------------------------------------------------------------------------+
|                             VPC FLOW LOG RECORD STRUCTURE                                         |
|                                                                                                   |
|  <version> <account-id> <interface-id> <srcaddr> <dstaddr> <srcport> <dstport> <protocol>         |
|  <packets> <bytes> <start> <end> <action> <log-status> [<pkt-srcaddr> <pkt-dstaddr>]             |
|                                                                                                   |
|  Sample Log 1 (Inbound Security Group or Inbound NACL Drop):                                      |
|  2 123456789012 eni-0a1b2c3d 203.0.113.50 10.0.1.25 54321 443 6 1 40 1629850000 1629850060 REJECT OK
|                                                                                                   |
|  Sample Log 2 (Outbound Stateless NACL Ephemeral Drop):                                           |
|  2 123456789012 eni-0a1b2c3d 203.0.113.50 10.0.1.25 54321 443 6 1 40 1629850000 1629850060 ACCEPT OK
|  2 123456789012 eni-0a1b2c3d 10.0.1.25 203.0.113.50 443 54321 6 1 40 1629850000 1629850060 REJECT OK
+---------------------------------------------------------------------------------------------------+

Interpreting ACCEPT vs. REJECT Patterns

1. Inbound REJECT: The destination ENI received an incoming packet, but either the Inbound Security Group or the Inbound NACL dropped it. Because Security Groups evaluate before the OS network stack and NACLs evaluate at the subnet edge, check both.
2. Inbound ACCEPT + Outbound REJECT: The inbound request was permitted by both the Inbound NACL and the Security Group. However, the outbound response from port 443 back to client ephemeral port 54321 was rejected by the Outbound NACL. (Security Groups cannot cause this drop because they are stateful).
3. pkt-srcaddr vs. srcaddr Discrepancies: When traffic traverses an intermediate Network Address Translation (NAT) Gateway or Application Load Balancer, srcaddr reflects the original packet header, while pkt-srcaddr reveals the physical or translated source address, aiding in multi-hop IP triage.

# Filter CloudWatch Logs Insights for all REJECT packets targeting a specific database ENI
fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, protocol, action
| filter interfaceId = 'eni-0a1b2c3d4e5f67890' and action = 'REJECT'
| sort @timestamp desc
| limit 50

---

3. Route Table Misconfigurations & Asymmetric Routing

Virtual routing tables dictate packet forwarding paths inside cloud VPCs. Routing failures typically stem from missing default gateways, overlapping Classless Inter-Domain Routing (CIDR) blocks, or asymmetric routing across hybrid links.

+---------------------------------------------------------------------------------------------------+
|                         COMMON CLOUD ROUTING MISCONFIGURATIONS                                    |
|                                                                                                   |
|  Failure Mode                   Root Cause Mechanism                  Symptom & Remediation       |
|  +----------------------------+-------------------------------------+---------------------------+ |
|  | Missing Default Gateway    | Subnet route table lacks 0.0.0.0/0  | Private subnet instances  | |
|  |                            | pointing to IGW or NAT Gateway      | cannot download OS patches| |
|  | Overlapping CIDR Peering   | Two VPCs or on-prem networks share  | Longest Prefix Match      | |
|  |                            | identical IP ranges (e.g. 10.0.0.0/16)| routes traffic improperly| |
|  | Asymmetric Hybrid Routing  | Outbound exits Direct Connect (DX), | Stateful firewalls drop   | |
|  |                            | Inbound returns via IPSec VPN       | return SYN/ACK packets    | |
|  | Blackhole Route (Inactive) | Target peering connection or ENI    | Packets dropped silently; | |
|  |                            | deleted, but route entry remains    | status shows 'Blackhole'  | |
|  +----------------------------+-------------------------------------+---------------------------+ |
+---------------------------------------------------------------------------------------------------+

Longest Prefix Match Routing

Cloud routing engines strictly enforce Longest Prefix Match (LPM). More specific CIDR prefixes (higher subnet mask bits) always override broader CIDR entries:

• If a route table contains 10.0.0.0/16 -> local and 10.0.1.0/24 -> tgw-attach123, traffic destined for 10.0.1.50 will always route to the Transit Gateway (/24), whereas traffic destined for 10.0.2.50 routes locally (/16).
• Overlapping CIDR peering conflicts occur when two interconnected VPCs use the exact same address space (e.g., both use 10.0.0.0/16). Direct VPC peering cannot be established between overlapping CIDRs; organizations must implement Private NAT (Source/Destination NAT) or re-architect subnets.

Asymmetric Routing in Dual-Homed Hybrid Connections

When an enterprise establishes redundant connections to the cloud using both a dedicated AWS Direct Connect / Azure ExpressRoute circuit and a backup IPSec VPN tunnel:

• Outbound traffic from the cloud may exit over Direct Connect due to BGP Local Preference settings.
• Return traffic from the on-premises network may be routed over the IPSec VPN tunnel due to internal router metric preferences.
• Failure Mechanism: On-premises and cloud stateful Next-Generation Firewalls (NGFWs) along the path inspect TCP handshakes. When the firewall on the VPN path sees a TCP SYN-ACK or ACK packet without having seen the initial SYN packet (which traversed Direct Connect), it flags the packet as out-of-state and silently drops it.
• Remediation: Enforce symmetric routing by adjusting BGP attributes (AS-Path prepending on VPN, BGP MED, or Local Preference) so both directions traverse the same link.

---

4. Cloud DNS Resolution Failures

DNS failures prevent compute instances from discovering backend databases, external APIs, and internal cloud service endpoints.

+---------------------------------------------------------------------------------------------------+
|                           CLOUD DNS DIAGNOSTIC TAXONOMY                                           |
|                                                                                                   |
|  [ 1. Split-Horizon Private Zone Association ]                                                    |
|    ├── Symptom: 'NXDOMAIN' or public IP returned instead of private 10.x.x.x IP                   |
|    ├── Cause: VPC is not associated with the Route 53 / Azure Private DNS zone                    |
|    └── Attribute: VPC settings 'enableDnsHostnames' or 'enableDnsSupport' set to false           |
|                                                                                                   |
|  [ 2. VPC DHCP Options Sets & Custom DNS ]                                                        |
|    ├── Symptom: Internal cloud service endpoints (e.g. s3.amazonaws.com) fail to resolve          |
|    ├── Cause: Custom on-premises DNS servers configured in DHCP options without forwarding rules |
|    └── Fix: Configure outbound conditional forwarders pointing to cloud resolver (169.254.169.253)|
|                                                                                                   |
|  [ 3. DNS Resolver Rate Limiting (Throttling) ]                                                   |
|    ├── Symptom: Intermittent lookup timeouts (SERVFAIL) during traffic spikes                     |
|    ├── Cause: AWS Route 53 Resolver ENI quota exceeded (1,024 packets/sec per network interface) |
|    └── Fix: Deploy local DNS caching daemon (e.g., NodeLocal DNSCache, dnsmasq, systemd-resolved)|
+---------------------------------------------------------------------------------------------------+

Split-Horizon DNS & VPC Association

In split-horizon DNS, an authoritative domain (e.g., corp.internal) has both a public zone and a private zone. For a cloud virtual machine to resolve private record sets:

1. The private hosted zone must be explicitly associated with the VM's specific VPC ID.
2. The VPC configuration flags enableDnsSupport (enables the cloud DNS resolver at base IP + 2 / 169.254.169.253) and enableDnsHostnames (assigns DNS hostnames to instances with public IPs) must both be set to true.

Route 53 Resolver Rate Limits (1,024 Packets/Second Limit)

Every Amazon VPC provides access to the Amazon Route 53 Resolver (the .2 resolver). However, AWS enforces a hard limit of 1,024 packets per second per Elastic Network Interface (ENI) to the resolver address.

• In high-throughput environments (such as Kubernetes microservices making un-cached DNS queries per HTTP request), applications exceed 1,024 QPS, causing packets to be throttled.
• Applications experience intermittent 5-second DNS timeouts and SERVFAIL errors.
• Remediation: Install and enable local DNS caching daemons (such as systemd-resolved, dnsmasq, or Kubernetes NodeLocal DNSCache) on container nodes to serve repeat queries locally without generating outbound network packets.

# Test DNS resolution against the local VPC resolver
dig @169.254.169.253 db.internal.production.local +stats +trace

# Validate reverse DNS resolution and query response time
nslookup -timeout=2 10.0.2.45 169.254.169.253

---

5. MTU Black Holes & Path MTU Discovery (PMTUD) Failures

A subtle and devastating networking issue in hybrid and multi-cloud architectures is the Maximum Transmission Unit (MTU) Black Hole.

+---------------------------------------------------------------------------------------------------+
|                         MTU BLACK HOLE & PMTUD FAILURE MECHANISM                                  |
|                                                                                                   |
|  Client (VPC 1)                       VPN / Internet Gateway               Destination Server     |
|  MTU = 9001 (Jumbo)                   MTU = 1500 (Standard) / 1420         MTU = 1500             |
|  +-----------------+                 +-----------------------+            +------------------+    |
|  | Sends TCP SYN   | (64 bytes) ────►| Forwards packet       |───────────►| Responds SYN-ACK |    |
|  | Sends Big Frame | (8900 bytes)───►| Packet Exceeds MTU!   |            +------------------+    |
|  | with DF=1 flag  |                 | Drops Packet &        |                                    |
|  |                 |                 | Generates ICMP Type 3 |                                    |
|  |                 |                 | Code 4 (Frag Needed)  |                                    |
|  |                 |                 +-----------┬-----------+                                    |
|  |                 |                             │                                                |
|  | Never Receives  |◄────────────────────────────┘                                                |
|  | ICMP Packet!    |  [ X ] BLOCKED BY FIREWALL / SECURITY GROUP / NACL (ICMP Denied)             |
|  | Connection HANGS|                                                                              |
|  +-----------------+                                                                              |
+---------------------------------------------------------------------------------------------------+

How PMTUD Works and Why It Fails

1. MTU Mismatch: Inside an AWS VPC or cloud backbone, virtual machines utilize Jumbo Frames (MTU 9001). Across the public Internet, Direct Connect links, or IPSec VPN tunnels (which add IPsec encapsulation overhead), the MTU is limited to 1500 or 1420 bytes.
2. Don't Fragment (DF) Bit: Modern operating systems set the DF=1 flag on TCP packets to optimize throughput.
3. The Drop: When an 8,900-byte packet arrives at a VPN router with an egress MTU of 1,420 bytes, the router cannot fragment the packet because DF=1. The router drops the packet and sends an ICMP Type 3, Code 4 message (Destination Unreachable: Fragmentation Needed and DF Set) back to the sender, indicating the required next-hop MTU.
4. The Black Hole: Network engineers frequently configure overly restrictive Security Groups or NACLs that block all ICMP traffic (protocol -1 or ICMP deny). Consequently, the sender never receives the ICMP Type 3 Code 4 alert. The initial small TCP 3-way handshake succeeds (SYN packets are ~60 bytes), but as soon as the client transmits actual application data (e.g., an SSL certificate exchange, HTTP POST payload, or database query), the connection hangs indefinitely and times out.
5. Remediation:
   • Permit ICMP Type 3, Code 4 inbound on all Security Groups and NACLs.
   • Configure TCP Maximum Segment Size (MSS) Clamping on intermediate VPN gateways (iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu).
   • Standardize cloud instance network interfaces to MTU 1500 when communicating over hybrid tunnels.

---

6. Load Balancer Health Check Failures

Application Load Balancers (ALBs) and Network Load Balancers (NLBs) route traffic exclusively to backend targets registered as Healthy. When all targets are marked Unhealthy, the load balancer returns HTTP 502 Bad Gateway or HTTP 503 Service Unavailable.

+---------------------------------------------------------------------------------------------------+
|                         LOAD BALANCER HEALTH CHECK DIAGNOSTIC MATRIX                              |
|                                                                                                   |
|  Health Check Symptom             Root Cause Analysis                 Corrective Action           |
|  +------------------------------+-----------------------------------+---------------------------+ |
|  | Status 301/302 Redirect      | Probe expects 200 OK, but backend | Update health check path  | |
|  |                              | forces HTTP -> HTTPS redirect     | or accept 200-399 codes   |
|  | Status 404 Not Found         | Health probe path typo (e.g., /   | Point probe to valid      | |
|  |                              | instead of /api/v1/healthz)       | lightweight endpoint      |
|  | Probe Connection Timeout     | Target Security Group blocks LB   | Allow LB security group on| |
|  |                              | private IP or probe timeout < app | backend listening port    |
|  | Flapping Health Status       | Health check interval too short;  | Increase probe timeout &  | |
|  | (Healthy <-> Unhealthy)      | backend CPU spikes delay probe    | consecutive threshold     |
|  | Targets Not In Service       | Instances registered in wrong AZ  | Verify subnets & VPC target|
|  |                              | or stopped by auto-scaling group  | group registration bounds |
|  +------------------------------+-----------------------------------+---------------------------+ |
+---------------------------------------------------------------------------------------------------+

Key Health Check Parameters

• Response Code Matchers: ALBs allow configuring valid HTTP response codes (default is 200). If a backend application returns 301 Moved Permanently (common when web servers enforce HTTPS redirection), the load balancer flags the target as unhealthy unless 200,301,302 or 200-399 is configured in the matcher.
• Health Probe Path: The health endpoint should be a dedicated, lightweight health check handler (e.g., /healthz) that checks basic process viability without running expensive, deep database queries that might time out under heavy load.
• Threshold Counts:
  • Healthy Threshold Count: Number of consecutive successful probes required before marking an unhealthy instance as healthy (e.g., 3).
  • Unhealthy Threshold Count: Number of consecutive failed probes required before taking an instance out of service (e.g., 2).
  • Timeout: Maximum time the load balancer waits for a probe response (must be less than the check interval).

---

7. Network Service Unavailability: DHCP, NTP, HTTP Status Codes & Switching Issues

Beyond routing tables and firewalls, Cloud+ Objective 6.2 tests four utility-service failure families that routinely masquerade as application problems.

DHCP Scope Exhaustion & Misconfiguration

• Mechanism: Cloud subnets and on-premises LANs hand out dynamic IPv4 leases from a fixed-size DHCP scope (for example, a /24 with 251 usable addresses). During auto-scaling surges or CI/CD deployment storms, newly launched instances can consume every remaining lease; subsequent launches then receive an APIPA self-assigned link-local address (169.254.0.0/16) or no address at all.
• Symptoms: Newly provisioned instances cannot reach any gateway and ip addr / ipconfig shows a 169.254.x.x address, while older instances keep working — misleading triage toward the "new" servers.
• Additional cloud nuance: In AWS, five addresses per subnet (the first four and the last) are permanently reserved, and overriding the cloud DHCP options set with custom DNS servers lacking forwarding rules silently breaks resolution of cloud service endpoints.
• Remediation: Expand the subnet CIDR or attach a secondary CIDR range, shorten lease durations for ephemeral fleets, and alert on address-pool utilization before it reaches 100%.

NTP Clock Drift

• Mechanism: Kerberos tickets, SAML assertions, OIDC tokens, TLS certificate notBefore/notAfter validation, and AWS Signature Version 4 request signing all assume the instance clock is synchronized via the Network Time Protocol (NTP, UDP 123). If the NTP source is blocked or misconfigured, clocks drift.
• Symptoms: Intermittent authentication failures ("token expired or not yet valid"), TLS validation errors, RequestTimeTooSkewed signing errors against AWS APIs, and misleading cross-system log timelines during forensic correlation.
• Remediation: Point every instance at the resilient cloud time endpoint (for example, the Amazon Time Sync Service at 169.254.169.123), permit outbound UDP 123 in security policies, and alert when clock skew approaches five minutes (the Kerberos tolerance limit).

HTTP Status Codes for Rapid Triage

Switching Issues: VLAN Tags & Access vs. Trunk Ports

In hybrid data centers feeding the cloud — and inside VMware/NSX virtual switching — connectivity failures frequently trace back to Layer 2 misconfiguration:

• Access ports carry untagged frames for exactly one VLAN; end devices and servers connect to access ports.
• Trunk ports (802.1Q) carry tagged frames for multiple VLANs between switches, routers, and virtualization hosts.
• Failure signatures: A link intended as a trunk but configured as an access port passes only one VLAN and isolates every other segment; a missing or misconfigured 802.1Q VLAN tag — or a native-VLAN mismatch between the two trunk ends — blackholes tagged traffic; a VM port group bound to the wrong VLAN ID silently lands the workload on the wrong network.
• Triage: Verify the VLAN assigned to the affected port group or interface first, then confirm both ends of the trunk agree on the allowed-VLAN list and the native VLAN, using the physical switch's show vlan / show interfaces trunk equivalents.

---

8. CompTIA Cloud+ Exam Traps: Cloud Networking