10.2 NSG Flow Logs, VNet Flow Logs & Traffic Analytics

NSG Flow Logs, VNet Flow Logs & Traffic Analytics

Deep visibility into packet flows across cloud virtual networks is critical for security compliance, threat detection, network capacity management, and auditing. In Azure, flow logging provides the raw telemetry required to reconstruct conversations traversing Network Security Groups and virtual network subnets.

Combined with Traffic Analytics, raw JSON flow records are enriched with Azure topology context, geographical geolocation data, and the Microsoft Threat Intelligence database to deliver comprehensive visibility into network communications.

1. NSG Flow Logs: Version 1 vs. Version 2

NSG Flow Logs capture IP traffic flowing through Network Security Groups associated with subnets and NICs. Flow logs are written to an Azure Storage account in JSON format.

+-----------------------------------------------------------------------------+
|                     NSG FLOW LOG FORMAT COMPARISON                          |
|                                                                             |
|   VERSION 1 FLOW RECORD STRUCTURE:                                          |
|   "1590000000,10.1.0.4,198.51.100.2,443,51234,T,I,A"                       |
|   - Timestamp: 1590000000                                                   |
|   - Source IP: 10.1.0.4                                                     |
|   - Destination IP: 198.51.100.2                                            |
|   - Source Port: 443                                                        |
|   - Destination Port: 51234                                                 |
|   - Protocol: T (TCP) / U (UDP)                                             |
|   - Direction: I (Inbound) / O (Outbound)                                   |
|   - Decision: A (Allow) / D (Deny)                                          |
|                                                                             |
|   VERSION 2 FLOW RECORD STRUCTURE (ADDS THROUGHPUT & STATE):                |
|   "1590000000,10.1.0.4,198.51.100.2,443,51234,T,I,A,B,12,6144,8,4096"       |
|   - Flow State: B (Begin) / C (Continuing) / E (End)                        |
|   - Packets Sent (Src -> Dst): 12                                           |
|   - Bytes Sent (Src -> Dst): 6144                                           |
|   - Packets Received (Dst -> Src): 8                                        |
|   - Bytes Received (Dst -> Src): 4096                                       |
+-----------------------------------------------------------------------------+

Key Architectural Differences

2. VNet Flow Logs: Evolution Beyond NSGs

Historically, flow logging in Azure was tied strictly to Network Security Groups. However, modern enterprise topologies frequently include specialized subnets where NSGs cannot be associated or are restricted—such as AzureFirewallSubnet, GatewaySubnet, AzureBastionSubnet, or subnets hosting Private Endpoints.

VNet Flow Logs (configured as Azure Network Watcher Flow Log resources) elevate flow logging to the Virtual Network or Subnet scope:

• Broader Coverage: Logs traffic traversing virtual networks and subnets regardless of whether an NSG is attached.
• Private Endpoint Visibility: Captures traffic flows targeting Azure Private Link Private Endpoints residing within subnets.
• Simplified Governance: Enables organizations to configure a single flow log resource at the VNet level that automatically captures all subnets within the address space, eliminating the operational overhead of managing dozens of individual NSG flow log associations.

3. Storage Account Integration & Blob Partitioning

Flow logs are written to an Azure Storage account in block blobs using a deterministic, highly partitioned path hierarchy:

resourceId=/SUBSCRIPTIONS/{subscriptionId}/RESOURCEGROUPS/{resourceGroupName}/
  PROVIDERS/MICROSOFT.NETWORK/{resourceType}/{resourceName}/
  y={YYYY}/m={MM}/d={DD}/h={HH}/m={00}/macAddress={MAC}/PT1H.json

Storage Architecture Details

• Hourly Partitioning (PT1H.json): Logs are aggregated and written in one-hour chunks per network interface MAC address. Each file contains JSON-formatted records representing all flows processed during that hour.
• Storage Sizing & Retention Policies: Administrators can configure automatic retention policies ranging from 1 to 365 days, or set retention to 0 (indefinite retention). Standard General Purpose v2 (GPv2) storage accounts are recommended.
• Read-Only Ingestion: Storage accounts holding flow logs can be ingested by third-party Security Information and Event Management (SIEM) systems (such as Splunk or Datadog) via Event Hub or Storage Blob triggers.

4. Traffic Analytics Architecture & Capabilities

Traffic Analytics is an Azure-native analytics solution that provides rich visibility into user and application activity across cloud networks. It processes raw flow logs from the storage account, cross-references them with Azure network topology and Microsoft Threat Intelligence, and writes structured records into an Azure Log Analytics workspace.

Processing Intervals: 10 Minutes vs. 60 Minutes

When enabling Traffic Analytics, administrators choose a data aggregation processing interval:

• 60-Minute Processing Interval (Default): Raw flow logs are processed every hour. Recommended for general capacity planning, trend reporting, and monthly auditing.
• 10-Minute Processing Interval: Raw flow logs are processed every 10 minutes. Recommended for security operations (SecOps) teams, rapid incident response, and active network troubleshooting where near-real-time visibility is required.

Core Analytical Capabilities

1. Threat Intelligence & Malicious Traffic Identification:
   • Identifies inbound and outbound communication with known malicious IP addresses, command-and-control (C2) servers, botnets, crypto-miners, and TOR exit nodes.
   • Visualizes malicious IP origins on interactive global geographic maps.
2. NSG Rule Optimization & Inactive Rule Pruning:
   • Identifies "zero hit" NSG rules (rules that have processed zero traffic over the evaluation period) that can be safely deprecated to simplify firewall policies.
   • Flags overly permissive security rules (e.g., rules allowing inbound traffic from 0.0.0.0/0 to sensitive administrative ports like 22 or 3389).
3. Top Talkers & Bandwidth Management:
   • Ranks top communicating host pairs, top talker virtual machines, and highest-bandwidth network protocols.
   • Tracks inter-region and inter-zone egress data transfer to optimize data transfer billing and architecture.

5. KQL Queries for Network Traffic Analysis

Traffic Analytics writes structured telemetry into custom tables inside the Log Analytics workspace, primarily AzureNetworkAnalytics_CL (and AzureNetworkAnalyticsIP_CL). Network engineers can execute powerful Kusto Query Language (KQL) queries for diagnostic auditing.

Example 1: Identifying Blocked Malicious Inbound Traffic

// Find all inbound flows from known malicious public IPs that were blocked by NSG rules
AzureNetworkAnalytics_CL
| where FlowType_s == "MaliciousFlow"
| where FlowDirection_s == "I"
| where FlowStatus_s == "D" // Denied by NSG
| project TimeGenerated, SrcIP_s, DestIP_s, DestPort_d, L7Protocol_s, NSGList_s, Country_s
| order by TimeGenerated desc
| take 50

Example 2: Top 10 Internal Talkers by Byte Volume (Capacity Planning)

// Identify internal VMs generating the highest outbound byte volume
AzureNetworkAnalytics_CL
| where FlowType_s == "IntraVNet" or FlowType_s == "InterVNet"
| summarize TotalBytesSent = sum(OutboundBytes_d), TotalBytesReceived = sum(InboundBytes_d) 
    by SrcIP_s, VM_s
| extend TotalTrafficBytes = TotalBytesSent + TotalBytesReceived
| top 10 by TotalTrafficBytes desc
| project VM_s, SrcIP_s, TotalBytesSent, TotalBytesReceived, TotalTrafficBytes

Example 3: Auditing Cross-Region Egress Data Transfer

// Detect cross-region egress flows to optimize data transfer costs
AzureNetworkAnalytics_CL
| where FlowType_s == "InterRegion"
| summarize EgressGigabytes = sum(OutboundBytes_d) / (1024 * 1024 * 1024) 
    by SourceRegion_s, DestRegion_s, VM_s
| order by EgressGigabytes desc