2.3 Network Telemetry: VPC Flow Logs & Route 53 Resolver Logs

Foundations of Network Telemetry in AWS

Network telemetry forms the first line of defense in detecting unauthorized perimeter reconnaissance, lateral movement, data exfiltration, and command-and-control (C2) communication. In AWS, network traffic inspection relies primarily on two complementary telemetry streams:

1. Amazon VPC Flow Logs: Captures IP traffic flow metadata for Elastic Network Interfaces (ENIs), subnets, and VPCs. The companion Transit Gateway Flow Logs feature records the same kind of metadata at the transit gateway, per attachment, which is where inter-VPC and hybrid (VPN or Direct Connect) traffic is visible in a hub-and-spoke network. Both publish to CloudWatch Logs, Amazon S3, or Amazon Data Firehose.
2. Amazon Route 53 Resolver Query Logs: Captures internal DNS resolution queries submitted to the Amazon-provided DNS resolver (Route 53 Resolver or the VPC .2 address).

Both services operate out-of-band at the hypervisor layer. They introduce zero impact on network latency, packet throughput, or compute instance performance.

---

VPC Flow Logs Architecture & Capture Mechanics

VPC Flow Logs can be attached at three hierarchical levels:

• Individual ENI: Captures traffic on a single virtual network card.
• Subnet: Automatically captures traffic across all existing and future ENIs within the subnet.
• VPC: Automatically captures traffic across all existing and future ENIs across all subnets in the VPC.

Aggregation Intervals & Destinations

• Aggregation Window: Flow logs aggregate packets into connection records. The default aggregation interval is 10 minutes. For security monitoring and incident response, you should always configure a 1-minute aggregation interval to capture high-frequency bursts, port scans, and short-lived connections with lower detection latency.
• Delivery Destinations:
  • Amazon CloudWatch Logs: Ideal for real-time metric filters, CloudWatch alarms, and interactive threat hunting via CloudWatch Logs Insights.
  • Amazon S3: Optimal for cost-effective, long-term archival, compliance evidence storage, and Athena/Security Lake analytics.
  • Amazon Data Firehose: Designed for streaming delivery to third-party real-time SIEMs and streaming analytics platforms.

---

Anatomy of Flow Log Fields: Default vs. Custom Formats

The default format includes 14 basic network fields. However, the default format is insufficient for advanced security forensics because it obscures packet paths and middlebox transformations. Security engineers must deploy custom format flow logs to capture advanced metadata.

The pkt-* vs. srcaddr/dstaddr Distinction

At a middlebox such as a NAT gateway, srcaddr and dstaddr describe the network interface's view of each hop, while pkt-srcaddr and pkt-dstaddr preserve the original packet addresses. On the NAT gateway's network interface, the record for traffic arriving from a private instance shows srcaddr = the instance IP but dstaddr = the NAT gateway's own private IP; only pkt-dstaddr shows the real internet destination. The onward records show srcaddr = the NAT gateway's IP. Adding pkt-srcaddr and pkt-dstaddr to a custom format lets one record tie the internal instance IP to the external C2 address, so responders isolate the right host.

Decoding tcp-flags for Threat Hunting

The tcp-flags field represents a bitmask of TCP control bits:

• 2 (SYN): Connection initiation.
• 18 (SYN-ACK: 2 + 16): Response from open port.
• 4 (RST): Connection refused / reset.
• 1 (FIN): Normal connection teardown.
• If a flow log shows action: REJECT and tcp-flags: 2 across hundreds of sequential ports, an external actor is actively conducting a TCP SYN port scan against the host.

---

Critical Blind Spots: What VPC Flow Logs DO NOT Capture

A critical objective on the AWS Security Specialty exam is knowing the operational limitations and blind spots of VPC Flow Logs. Flow logs do NOT record the following categories of traffic:

1. Amazon Route 53 Resolver Queries: Traffic to the VPC DNS resolver (169.254.169.253 or the VPC network base address plus two, e.g., 10.0.0.2) is explicitly omitted from VPC Flow Logs.
2. Instance Metadata Service (IMDS): Traffic to the link-local address 169.254.169.254 is not logged by VPC Flow Logs.
3. Amazon Time Sync Service: Traffic to 169.254.169.123 is excluded.
4. DHCP Traffic: Communications between instances and the Amazon VPC DHCP server are omitted.
5. Windows License Activation: Traffic to AWS Windows licensing servers is not recorded.

Exam Trap: If an exam scenario describes an attacker performing DNS tunneling or exfiltrating data via DNS queries to the VPC .2 resolver, reviewing VPC Flow Logs will reveal nothing. You must enable Route 53 Resolver Query Logging to gain visibility into DNS traffic.

---

Route 53 Resolver Query Logging

Amazon Route 53 Resolver Query Logs record every DNS query initiated by resources within your VPCs (EC2 instances, Lambda functions in VPCs, ECS tasks, and EKS pods) directed to the Route 53 Resolver.

Query Log Attributes

Route 53 Resolver query logs are output in JSON format containing crucial threat hunting attributes:

• query_timestamp: Exact timestamp of the DNS query.
• query_name: The fully qualified domain name (FQDN) requested (e.g., malicious-c2.attacker-domain.com).
• query_type: Resource record type requested (e.g., A, AAAA, TXT, CNAME).
• rcode: DNS response code (NOERROR, NXDOMAIN, SERVFAIL, REFUSED).
• answers: The DNS answer payload returned, including resolved IP addresses.
• srcaddr: The private IP of the ENI that submitted the DNS query.
• firewall_rule_action: When a Route 53 Resolver DNS Firewall rule with an alert or block action matched, the action applied (ALERT or BLOCK), alongside firewall_rule_group_id and firewall_domain_list_id.

Threat Hunting with DNS Telemetry

1. Detecting DNS Tunneling: Attackers encapsulate stolen data (credit cards, credentials) into encoded subdomains (e.g., base64data.exfil.attacker.com) and issue DNS lookups. Query logs capture these queries, revealing unusually long subdomain strings, high entropy, and an excessive volume of TXT queries.
2. Detecting Domain Generation Algorithms (DGAs): Malware infected hosts often query hundreds of algorithmically generated domain names daily seeking active C2 servers. Query logs reveal high volumes of consecutive NXDOMAIN (Non-Existent Domain) responses originating from a single internal IP.

---

Threat Hunting with CloudWatch Logs Insights

When flow logs and DNS query logs are streamed to CloudWatch Logs, CloudWatch Logs Insights provides an interactive, highly scalable query engine to hunt for threats.

Query 1: Hunting for Inbound Port Scans & Brute Force Reconnaissance

Identifies external IP addresses generating the highest volume of rejected connections:

fields @timestamp, srcAddr, dstPort, protocol
| filter action = "REJECT"
| stats count(*) as rejectCount by srcAddr, dstPort
| sort rejectCount desc
| limit 20

Query 2: Hunting for High-Volume Outbound Data Exfiltration

Aggregates outbound bytes transferred to non-RFC1918 public IP addresses to identify potential data exfiltration:

fields @timestamp, srcAddr, dstAddr, dstPort, bytes
| filter flowDirection = "egress" and action = "ACCEPT"
| filter not (isIpv4InSubnet(dstAddr, "10.0.0.0/8") or isIpv4InSubnet(dstAddr, "172.16.0.0/12") or isIpv4InSubnet(dstAddr, "192.168.0.0/16"))
| stats sum(bytes) as totalBytesSent by srcAddr, dstAddr, dstPort
| sort totalBytesSent desc
| limit 25

Query 3: Hunting for High-Volume NXDOMAIN Spikes in Route 53 Resolver Logs

Identifies internal hosts generating excessive NXDOMAIN errors indicative of DGA malware:

fields @timestamp, srcaddr, query_name, rcode
| filter rcode = "NXDOMAIN"
| stats count(*) as nxCount by srcaddr
| sort nxCount desc
| limit 15