7.1 Layered Traffic Filtering: SGs, NACLs & AWS Network Firewall

7.1 Layered Traffic Filtering: SGs, NACLs & AWS Network Firewall

Cloud infrastructure security demands a defense-in-depth posture across the network stack. Rather than relying on a single perimeter moat, modern AWS network architectures deploy concentric filtering boundaries: virtual network interface (ENI) micro-segmentation, stateless subnet boundary controls, and centralized deep packet inspection (DPI). Each layer operates at a distinct tier of the Open Systems Interconnection (OSI) model, enforcing specific operational models, performance characteristics, and threat mitigations.

This section analyzes the mechanics, capabilities, and trade-offs of the three primary network filtering mechanisms in Amazon Virtual Private Cloud (Amazon VPC): Security Groups, Network Access Control Lists (NACLs), and AWS Network Firewall.

---

Security Groups: Stateful Micro-Segmentation at the ENI Level

A Security Group (SG) functions as a virtual distributed firewall operating at the hypervisor layer for individual Elastic Network Interfaces (ENIs). Every packet entering or leaving an ENI attached to an Amazon EC2 instance, AWS Lambda function in a VPC, Amazon ECS task, or Application Load Balancer is evaluated against the associated security group rules.

Loading diagram...

1. Stateful Connection Tracking Mechanics

Security groups are stateful. They utilize an underlying connection tracking engine (conntrack) at the Nitro or Xen hypervisor level:

• Automatic Return Traffic: When an inbound packet matches an allow rule, the hypervisor creates an entry in its connection tracking table. All return traffic for that specific session is automatically permitted outbound, regardless of any outbound security group rules.
• Outbound Request Tracking: Conversely, if an instance initiates an outbound connection (e.g., calling an external API over TCP port 443), the return traffic arriving from the remote server is automatically permitted inbound, even if no inbound security group rule allows traffic on that port.
• Untracked Connections: A TCP or UDP flow is untracked when a rule allows it from all addresses (0.0.0.0/0 or ::/0) and a rule in the other direction allows all response traffic (0.0.0.0/0 or ::/0) on all ports (0–65535). ICMP is always tracked, and flows through NAT gateways, Network Load Balancers, Network Firewall endpoints, Global Accelerator, interface endpoints, and Lambda Hyperplane ENIs are automatically tracked. Untracked flows don't consume tracking allowance, but removing or changing the rule that allows them cuts them off immediately.
• Rule Changes and Existing Sessions: Changing a security group rule does not interrupt tracked connections; they continue until they time out. To stop tracked traffic immediately, add a network ACL deny rule.
• Connection Tracking Exhaustion: Each EC2 instance size has a finite connection tracking table capacity. If an instance experiences a massive volume of concurrent short-lived connections (such as during a SYN flood or distributed denial-of-service attack), the conntrack table can become exhausted. When exhausted, the hypervisor drops new connection attempts, causing packet loss even if CPU and memory are underutilized. The ENA driver metrics conntrack_allowance_exceeded (packets dropped because the limit was reached) and conntrack_allowance_available (remaining tracked connections), read with ethtool -S or published through the CloudWatch agent, identify this condition. Remedies include larger instance sizes, more instances behind the load balancer, shorter idle tracking timeouts, and making high-volume flows untracked.

2. Rule Evaluation and Grammar

Security group rules conform to a strict structural model:

• Permissive / Allow-Only Model: Security groups support allow rules only. You cannot create explicit DENY rules within a security group. Any traffic that is not explicitly allowed is dropped by the implicit default deny rule.
• Simultaneous Evaluation: Unlike firewall engines with sequential rule ordering, all security group rules are evaluated simultaneously. If multiple rules apply to a packet, the most permissive match applies.
• Security Group Referencing: Rather than hardcoding static IPv4 or IPv6 CIDR blocks, security groups support referencing another Security Group ID (sg-xxxx) as the source or destination. This enables dynamic micro-segmentation:
  • For example, an RDS database security group can specify sg-web-servers as the only permitted source on port 5432.
  • As instances scale in or out in the web tier, their dynamic private IP addresses are automatically recognized without manual rule updates.
  • Peering Support: Security group referencing works across VPC peering connections in the same Region (including cross-account peers, written as account-id/sg-id); it doesn't work across inter-Region peering.
  • Transit Gateway Support: Since 2024, VPCs attached to the same transit gateway can reference each other's security groups in inbound rules, once security group referencing support is enabled on the transit gateway and on the VPC attachments.

3. Security Group Limits and Performance Quotas

• Default Rules Quota: By default, a security group allows up to 60 inbound rules and 60 outbound rules.
• ENI Association Limit: A single ENI can have up to 5 security groups attached by default.
• Cross-Product Formula: AWS enforces an operational quota where (Number of Security Groups per ENI) x (Number of Rules per Security Group) cannot exceed 1,000. Increasing the rules per group decreases the maximum number of groups that can be attached to an interface.

---

Network Access Control Lists (NACLs): Stateless Subnet Boundaries

A Network ACL is an optional layer of security for a VPC that acts as a firewall for controlling traffic in and out of one or more subnets. While security groups protect individual ENIs, NACLs protect the entire subnet boundary.

Loading diagram...

1. Stateless Inspection Model

NACLs are strictly stateless. The NACL engine inspects each packet in isolation, retaining no memory or state of previously transmitted packets:

• An inbound packet allowed by an inbound NACL rule generates a response from the host instance that must be explicitly permitted by an outbound NACL rule.
• If an outbound rule is missing, return traffic is discarded at the subnet boundary, causing connection timeouts.

2. Sequential Rule Evaluation (Rule Numbers 1–32,766)

NACL rules are processed in numerical order from lowest to highest:

• Sequential Processing: When a packet arrives, the NACL evaluates rules starting with the lowest rule number (e.g., 10, 20, 100). As soon as a rule matches the packet's 5-tuple attributes (protocol, source IP, source port, destination IP, destination port), that rule is immediately applied (ALLOW or DENY), and evaluation terminates (short-circuit evaluation).
• The Default Asterisk (*) Rule: Every NACL contains an unnumbered asterisk rule (*) at the end of the rule list. This rule denies all traffic not matched by prior numbered rules. It cannot be modified or deleted.
• Explicit Denials: Unlike security groups, NACLs support explicit DENY rules. To block a malicious IP address or compromised subnet (e.g., 198.51.100.23/32), an administrator can create a rule with a lower number (e.g., Rule 50: DENY 198.51.100.23/32) before a general allow rule (e.g., Rule 100: ALLOW 0.0.0.0/0).

3. The Ephemeral Port Requirement

When a client initiates a connection to a server (e.g., a web browser connecting to HTTPS on port 443), the client operating system allocates a temporary port from its ephemeral port range to receive return traffic:

• Many Linux kernels, including Amazon Linux: 32768–61000
• NAT gateways, Elastic Load Balancing, and AWS Lambda: 1024–65535
• Windows Server 2008 and later: 49152–65535 (Windows Server 2003 and earlier used 1025–5000)

Because the client chooses the port, AWS examples commonly open 1024–65535 for return traffic to cover every client type.

Because NACLs are stateless:

• For a web server subnet accepting HTTPS: Inbound must allow port 443 from clients. Outbound must allow the clients' ephemeral ports (commonly 1024–65535).
• For a private subnet instance initiating outbound updates (e.g., running dnf update or apt-get over port 443 via a NAT Gateway): Outbound must allow port 443 to 0.0.0.0/0. Inbound must allow ephemeral ports (1024–65535, the range a NAT gateway uses) to accept the repository's server responses.

Failure to configure outbound ephemeral port allowances is one of the most common operational root causes for hanging TCP handshakes in AWS networking.

---

AWS Network Firewall: Managed Stateful Inspection at Scale

While Security Groups and NACLs provide Layer 3 and Layer 4 packet filtering, they cannot inspect application-layer payloads, evaluate regular expressions across packet streams, filter outbound web requests by domain name (FQDN), or detect intrusion signatures. AWS Network Firewall provides fully managed, scalable Layer 3 through Layer 7 firewall protection for Amazon VPCs.

1. Architecture and VPC Endpoint Deployment

AWS Network Firewall deploys as a dedicated gateway endpoint in an isolated subnet in each Availability Zone:

• Dedicated Firewall Subnet: Best practice mandates dedicating an exclusive subnet (e.g., /28 CIDR) per AZ solely for the Network Firewall endpoint. Workload instances must never share this subnet.
• Route Table Redirection (Gateway Routing): Traffic is directed through the firewall endpoint via Amazon VPC Route Tables:
  • Ingress Traffic: An Ingress Route Table associated with the Internet Gateway (IGW) contains edge association routes pointing the VPC CIDR to the Network Firewall VPC endpoint IDs.
  • Egress Traffic: The application subnet route table directs 0.0.0.0/0 to the Network Firewall endpoint in its local AZ, which then routes inspected traffic to the NAT Gateway or Internet Gateway.

Loading diagram...

2. Dual-Engine Architecture: Stateless vs Stateful Engines

AWS Network Firewall processes packets using a two-stage evaluation pipeline:

Loading diagram...

Stage 1: The Stateless Engine

• Inspects individual packets without retaining connection history.
• Evaluates standard 5-tuple attributes: source IP, source port, destination IP, destination port, and protocol.
• Rules are assigned explicit numerical priorities (evaluated from lowest to highest number).
• Possible actions:
  • Pass: Permits the packet immediately, completely bypassing the stateful inspection engine (useful for trusted, high-volume protocols like internal backups).
  • Drop: Immediately discards the packet without generating return ICMP or TCP notifications.
  • Forward to Stateful Rule Groups: Passes the packet to the stateful engine for deep packet inspection.

Stage 2: The Stateful Engine

• Maintains full TCP connection state, reassembles packet streams, and analyzes Layer 7 payloads.
• Powered by an open-source Suricata-compatible engine, supporting standard Suricata rule syntax.
• Supported rule actions:
  • Pass: Permits the entire traffic flow.
  • Drop: Silently discards packets belonging to the flow.
  • Reject: Drops the traffic and sends a TCP reset (RST) back to the sender. Reject applies only to TCP traffic and doesn't support FTP or IMAP.
  • Alert: Generates an alert event sent to Amazon CloudWatch Logs, Amazon S3, or Amazon Data Firehose, while allowing traffic to continue traversing the firewall.

3. Stateful Rule Evaluation Order: Default vs Strict

A critical architectural distinction tested on the SCS-C03 exam is the Rule Evaluation Order of the stateful engine:

4. Advanced Inspection Capabilities

Domain List Filtering

AWS Network Firewall can inspect outbound HTTP and HTTPS requests to restrict traffic to an approved list of Fully Qualified Domain Names (FQDNs):

• HTTP Inspection: Parses the unencrypted Host request header.
• HTTPS Inspection (without decryption): Parses the Server Name Indication (SNI) extension during the initial TLS Client Hello handshake before encryption begins.
• Supports prefix matching and wildcards (e.g., .amazonaws.com matches s3.amazonaws.com and dynamodb.us-east-1.amazonaws.com).
• Target actions: ALLOWLIST (drops all traffic not matching listed domains) or DENYLIST (blocks listed domains, permitting all others).

TLS Deep Packet Inspection

While SNI inspection identifies the destination hostname, it cannot inspect the encrypted HTTPS payload for data exfiltration, command-and-control (C2) beacons, or malicious scripts. Furthermore, emerging standards like Encrypted Client Hello (ECH) obscure the SNI header.

AWS Network Firewall supports TLS Inspection:

• Operates as a forward proxy decrypting outbound client traffic and re-encrypting it before sending it to the external destination.
• Leverages AWS Private CA or an imported enterprise intermediate CA certificate.
• The client instance must trust the private root CA certificate installed in its local operating system trust store.
• Once decrypted, the stateful Suricata engine inspects raw application payloads against signatures, detects embedded DLP patterns, and checks certificate revocation status via OCSP/CRL before re-encrypting.

---

Comparison: Security Groups vs NACLs vs AWS Network Firewall

---

Specialty Exam Pitfalls & Architectural Traps

1. The NACL Ephemeral Return Port Trap: An administrator creates a secure private subnet NACL that allows outbound traffic on TCP port 443 to permit EC2 instances to pull updates from external repositories. However, instances time out when connecting. The root cause is the absence of an inbound NACL rule allowing ephemeral ports (1024–65535). Because NACLs are stateless, return packets from external servers are dropped at the subnet boundary.
2. The Network Firewall Default Evaluation Order Trap: In AWS Network Firewall stateful rule groups, the Default Action Order executes Pass before Drop. If an engineer writes a Suricata rule to drop known malicious HTTP user agents but has a domain list rule set to pass traffic to *.example.com, any request to example.com carrying the malicious user agent will be passed, because the Pass action overrides the Drop action. Security architects must specify Strict Rule Order to enforce drops sequentially.
3. Where Security Group Referencing Works: Referencing works within a VPC, across same-Region VPC peering, and (inbound rules only) between VPCs attached to the same transit gateway with referencing support enabled. It does not work across inter-Region peering or for outbound rules over a transit gateway; those cases need CIDR-based rules.
4. SNI Spoofing Without TLS Inspection: Relying strictly on Network Firewall Domain List filtering without enabling TLS decryption leaves organizations vulnerable to SNI spoofing. Advanced adversaries can craft TLS handshakes with an allowlisted SNI (e.g., update.microsoft.com) while routing the underlying HTTP request or TCP connection to an arbitrary malicious endpoint once the handshake completes. True payload validation requires active TLS inspection.