14.1 VPC Configuration Troubleshooting & Analysis Tools
Key Takeaways
- Security Groups provide stateful packet filtering at the elastic network interface (ENI) level, automatically admitting return traffic and evaluating all allow rules collectively without regard to rule ordering.
- Network Access Control Lists (NACLs) operate as stateless traffic filters evaluated sequentially by rule number; return traffic fails unless explicit inbound and outbound rules allow the ephemeral port range (1024–65535).
- VPC Reachability Analyzer executes static configuration analysis using formal reasoning algorithms to trace paths hop-by-hop across VPC components without injecting or transmitting packets across the network.
- Network Access Analyzer evaluates compliance against network access scopes to verify structural perimeter isolation, such as ensuring database subnets maintain no reachable paths to internet gateways.
- CloudWatch Network Monitor performs synthetic active probing across hybrid connections (Direct Connect and VPN) to measure packet loss and latency, while CloudWatch Internet Monitor tracks internet health and client-side degradation for global user endpoints.
14.1 VPC Configuration Troubleshooting & Analysis Tools
CloudOps Blueprint Focus: Network connectivity issues represent one of the most frequent operational failure modes in cloud infrastructure. For the AWS Certified CloudOps Engineer – Associate (SOA-C03) exam, you must master the architectural divergence between Security Groups and Network ACLs, diagnose ephemeral port traps, leverage VPC Reachability Analyzer for packet-free hop-by-hop path tracing, audit network segmentation using Network Access Analyzer, and monitor hybrid connectivity with CloudWatch Network Monitor and Internet Monitor.
Security Groups vs. Network Access Control Lists (NACLs)
When troubleshooting packet drops or unexpected connectivity loss within an Amazon VPC, the first line of investigation focuses on Security Groups and Network Access Control Lists (NACLs). While both filter IP traffic, their operational boundaries, statefulness, and evaluation logic differ fundamentally.
| Operational Dimension | Security Groups | Network Access Control Lists (NACLs) |
|---|---|---|
| Enforcement Layer | Elastic Network Interface (ENI) level | Subnet boundary level |
| Statefulness | Stateful (return traffic automatically allowed) | Stateless (return traffic requires explicit rule) |
| Rule Types | Allow rules only (implicit deny at end) | Allow and Deny rules |
| Evaluation Order | All rules evaluated collectively; order is irrelevant | Ordered evaluation by rule number (lowest evaluated first) |
| Default Configuration | Inbound denies all; outbound permits all | Default NACL allows all; Custom NACL denies all |
| Ephemeral Return Traffic | Managed automatically via connection tracking table | Must explicitly permit inbound ports 1024–65535 |
| Scope of Impact | Associated instances and ENIs only | All resources residing within the associated subnet |
Security Group Connection Tracking
Security groups operate at the virtual interface layer. When an instance initiates an outbound connection, the security group evaluates outbound rules. If permitted, the virtualization layer records an entry in its connection tracking table. When the remote server responds, inbound packets match the tracked session and are admitted automatically, regardless of inbound security group rules. Untracked flows (such as unrestricted 0.0.0.0/0 traffic across all protocols) bypass tracking tables to prevent table exhaustion during connection surges.
NACL Statelessness & The Ephemeral Port Trap
Because NACLs operate at the subnet boundary and are stateless, they maintain no memory of outbound sessions. Every packet crossing the subnet boundary is evaluated independently.
When an EC2 instance in a private subnet establishes an outbound connection to an external HTTP (port 80) or HTTPS (port 443) endpoint:
- The client operating system assigns a high random port from its ephemeral port range (Linux kernels use ports
32768–60999, Windows Server uses49152–65535, and NAT Gateways use1024–65535). - The outbound packet leaves the subnet; the outbound NACL allows destination port 80/443.
- The remote web server responds back to the client's allocated ephemeral port.
- The inbound response arrives at the subnet boundary. If the inbound NACL permits only ports 80 and 443, the response is dropped immediately.
To resolve this issue, any subnet hosting instances that initiate outbound connections must include an inbound NACL rule permitting TCP traffic on ephemeral ports 1024–65535 from 0.0.0.0/0.
NACL Rule Ordering & Precedence
NACL rules evaluate in ascending numerical order starting with the lowest number. Evaluation terminates upon the first match, ignoring subsequent rules. For example, if Rule 100 denies SSH from 10.0.0.0/8, a subsequent Rule 110 allowing SSH from 10.0.1.50/32 will never take effect. The asterisk (*) rule acts as an immutable default deny. Administrators should number rules in increments of 10 or 100 to allow inserting emergency exceptions.
VPC Reachability Analyzer: Hop-by-Hop Configuration Analysis
Historically, diagnosing VPC connectivity required running live ping or traceroute tests across production networks. VPC Reachability Analyzer performs static configuration analysis using formal mathematical reasoning to evaluate connectivity without sending live packets.
Operating Mechanics & Coverage
Reachability Analyzer inspects control-plane configurations without transmitting physical packets, running safely during peak production traffic:
- Supported Endpoints: ENIs, EC2 instances, ALBs, NLBs, Internet Gateways, NAT Gateways, Transit Gateways, and VPC Peering connections.
- Hop-by-Hop Path Tracing: Pinpoints the exact misconfigured component when connectivity fails (e.g., missing route in a route table, closed security group port, or blocking NACL deny rule).
# Trace connectivity between an EC2 instance and an RDS database ENI
aws ec2 create-network-insights-path \
--source i-0a1b2c3d4e5f6g7h8 \
--destination eni-0123456789abcdef0 \
--protocol tcp --destination-port 5432
aws ec2 start-network-insights-analysis \
--network-insights-path-id nip-0987654321fedcba0
Network Access Analyzer: Continuous Segmentation Auditing
While Reachability Analyzer troubleshoots point-to-point paths, Network Access Analyzer verifies network segmentation and compliance postures across entire VPCs.
Using automated reasoning, it evaluates network configurations against predefined Network Access Scopes:
- Perimeter Isolation: Verifies that no route or gateway provides an inbound path from the internet to database or payment processing subnets.
- Cross-Environment Boundaries: Guarantees that non-production VPCs cannot route into production VPCs across Transit Gateways or VPC peering links.
- Compliance Artifacts: Generates cryptographic evidence demonstrating strict structural network isolation for PCI-DSS, HIPAA, and SOC 2 audits.
CloudWatch Network Telemetry: Network Monitor & Internet Monitor
- Amazon CloudWatch Network Monitor: Deploys active synthetic probing agents within VPC subnets to continuously test connectivity to on-premises IP destinations over AWS Direct Connect or Site-to-Site VPN. It tracks
PacketLossPercentandRoundTripTime(RTT) to detect physical fiber degradation or routing churn before application performance suffers. - Amazon CloudWatch Internet Monitor: Provides visibility into how public internet performance impacts end users accessing AWS applications fronted by Amazon CloudFront, Route 53, or VPCs. It calculates an Internet Availability Score and Internet Performance Score, helping operators determine whether client latency stems from AWS infrastructure or regional ISP outages.
An engineer launches an Amazon EC2 instance in a public subnet with an allocated Elastic IP address. The instance can send ping requests to other internal instances, but running yum update or accessing external software repositories over HTTP (port 80) and HTTPS (port 443) fails with connection timeout errors. The instance's security group allows outbound traffic on all ports to 0.0.0.0/0. The subnet's Network ACL has Rule 100 allowing inbound TCP ports 80 and 443 from 0.0.0.0/0, Rule 100 allowing outbound TCP ports 80 and 443 to 0.0.0.0/0, and the default Rule * denying all traffic. What is the root cause of this connectivity failure?
An operations team needs to determine why application servers in Subnet A cannot establish database connections to an Amazon Aurora PostgreSQL cluster in Subnet B within the same VPC. The team must identify whether the failure is caused by a missing route, an unconfigured security group rule, or a blocking Network ACL rule, but corporate policy strictly prohibits transmitting synthetic test packets across the production database network. Which AWS diagnostic tool satisfies these requirements?
A financial services organization has strict regulatory mandates requiring continuous validation that no resources within their private PCI cardholder data environment subnets have a network path to an Internet Gateway or public VPC peering connection. The compliance director requires an automated auditing mechanism to generate findings whenever network paths violate these structural isolation boundaries. Which service fulfills this compliance requirement?