14.2 Network Traffic Analysis: VPC Flow Logs & Log Analytics

14.2 Network Traffic Analysis: VPC Flow Logs & Log Analytics

CloudOps Blueprint Focus: Packet payload inspection is rarely viable in high-throughput cloud environments. The AWS Certified CloudOps Engineer – Associate (SOA-C03) exam tests your ability to configure Amazon VPC Flow Logs, interpret default and custom log formats, decode TCP flags, run targeted CloudWatch Logs Insights queries, and correlate network flows with Application Load Balancer and AWS WAF telemetry.

VPC Flow Logs Architecture & Capture Scope

Amazon VPC Flow Logs captures IP traffic flow metadata entering and leaving network interfaces across an Amazon VPC. Operating out-of-band at the hypervisor level, Flow Logs impose zero latency overhead and zero compute impact on running workloads.

Capture Levels & Publishing Destinations

Flow logs can be provisioned at three architectural scopes:

1. VPC Level: Monitors all current and future ENIs across all subnets in the VPC.
2. Subnet Level: Captures traffic across all ENIs within a specific subnet.
3. Network Interface (ENI) Level: Surgically targets an individual ENI for focused troubleshooting.

Traffic Filters & Non-Captured Traffic

Engineers select one of three traffic filters:

• ALL: Records both permitted and blocked flows; required for complete visibility.
• ACCEPT: Records only traffic permitted by Security Groups and NACLs; used for capacity planning and baselining.
• REJECT: Records only traffic blocked by Security Groups or NACLs; optimizes cost for threat hunting.

[!IMPORTANT] Non-Captured Traffic: Flow Logs do not record traffic to the Amazon DNS Route 53 Resolver (VPC base CIDR + 2, e.g., 10.0.0.2), EC2 Instance Metadata Service (169.254.169.254), Amazon Time Sync (169.254.169.123), DHCP client traffic, or hypervisor licensing communication.

---

Log Formats & Extended Telemetry Fields

Flow logs aggregate packets over 10-minute intervals (or 1-minute intervals for accelerated troubleshooting).

Default vs. Custom Formats

The default format outputs 14 fields:

<version> <account-id> <interface-id> <srcaddr> <dstaddr> <srcport> <dstport> <protocol> <packets> <bytes> <start> <end> <action> <log-status>

Standard protocol numbers include 6 (TCP), 17 (UDP), and 1 (ICMP).

To troubleshoot complex topologies, engineers deploy a Custom Log Format incorporating extended attributes:

• tcp-flags: Bitmask identifying TCP control flags observed during the flow window.
• pkt-srcaddr & pkt-dstaddr: Captures the original packet source and destination IP addresses before Network Address Translation (NAT) through a NAT Gateway.
• flow-direction: Identifies whether the flow was ingress or egress relative to the interface.
• traffic-path: Shows the routing path (e.g., Internet Gateway, Transit Gateway, or Peering).

Decoding TCP Flag Bitmasks

TCP flags reveal connection handshake stages:

• 2 (SYN): Connection initiation. Repeated tcp-flags = 2 with action = REJECT indicates blocked connection attempts or port scanning.
• 18 (SYN-ACK = 16 + 2): Handshake acknowledged; confirms bidirectional connectivity.
• 4 (RST): Connection reset; indicates abnormal session termination by a host or firewall.
• 1 (FIN) / 16 (ACK): Standard graceful connection teardown.

---

CloudWatch Logs Insights Queries for Network Troubleshooting

Amazon CloudWatch Logs Insights provides an interactive engine to scan millions of flow records rapidly.

1. Identifying Port Scanning & Reconnaissance

To identify hostile internal or external scanning activity:

filter action = "REJECT"
| stats count(*) as DropCount by srcaddr, dstport, protocol
| sort DropCount desc
| limit 20

A high DropCount on ports 22 (SSH), 3389 (RDP), or 445 (SMB) highlights brute-force or scanning attempts.

2. Finding Top Talkers & Bandwidth Consumers

To isolate instances driving high cross-AZ or NAT Gateway data transfer charges:

filter action = "ACCEPT"
| stats sum(bytes) as TotalBytes, sum(packets) as TotalPackets by srcaddr, dstaddr
| sort TotalBytes desc
| limit 10

3. Diagnosing Stateless NACL Return Traffic Drops

When an application establishes outbound connections that fail to receive responses:

filter (srcaddr = "10.0.1.25" and dstaddr = "198.51.100.10") or 
       (srcaddr = "198.51.100.10" and dstaddr = "10.0.1.25")
| fields @timestamp, srcaddr, dstaddr, srcport, dstport, protocol, action, bytes
| sort @timestamp desc

If the outbound request (10.0.1.25:52140 -> 198.51.100.10:443) is ACCEPTed, but the return packet (198.51.100.10:443 -> 10.0.1.25:52140) displays REJECT, a stateless NACL is dropping the response on ephemeral port 52140.

---

Correlating Multi-Layer Network Telemetry

Isolating application issues requires correlating Layer 3/4 flow logs with Layer 7 load balancer and security telemetry.

Elastic Load Balancing (ELB) Access Logs

ELB access logs provide three critical timing metrics:

• request_processing_time: Latency from client request reception to dispatch to backend.
• target_processing_time: Time spent by backend EC2/container instances processing the request before sending headers. High values indicate application code or database query latency.
• response_processing_time: Time taken to transmit the response to the client.
• Status Code Discrepancies: If elb_status_code is 504 and target_status_code is -, the backend target timed out before responding.

AWS WAF Web ACL Logs & CloudFront Logs

• AWS WAF Logs: Capture the terminatingRuleId and action (BLOCK, ALLOW, COUNT). When users encounter HTTP 403 errors, WAF logs confirm whether managed rules (e.g., SQLi or rate-limiting) blocked the client IP.
• CloudFront Access Logs: Record edge cache hits/misses, origin latency, and TLS protocol negotiation, pinpointing whether latency occurs at the edge or origin.