4.1 Azure ExpressRoute Architecture & Peering Models

Azure ExpressRoute creates a private, dedicated, high-throughput connection between enterprise on-premises infrastructure and Microsoft Cloud services. Unlike Site-to-Site VPN connections that traverse the public internet, ExpressRoute traffic travels over a private optical network fabric, delivering predictable performance, lower latency, and enhanced security for mission-critical enterprise workloads.

---

1. ExpressRoute Connectivity Models & Physical Topology

To establish an ExpressRoute circuit, organizations partner with network service providers or connect directly to Microsoft Enterprise Edge (MSEE) routers at designated ExpressRoute peering locations (meet-me facilities).

+---------------------------------------------------------------------------------------------------+
|                                 EXPRESSROUTE CONNECTIVITY MODELS                                  |
|                                                                                                   |
|  1. Cloud Exchange Co-location (Layer 2 / Layer 3 Ethernet Exchange)                              |
|     [ On-Premises Meet-Me ] <---> [ Cloud Exchange (e.g. Equinix, Megaport) ] <---> [ MSEE Pair ] |
|                                                                                                   |
|  2. Point-to-Point Ethernet (Dedicated Layer 2 Connection)                                       |
|     [ Corporate Datacenter ] <============== Dedicated Point-to-Point Fiber ==============> [ MSEE Pair ] |
|                                                                                                   |
|  3. Any-to-Any IPVPN (Layer 3 MPLS / IP-VPN Network)                                              |
|     [ Branch 1 ] ----+                                                                            |
|     [ Branch 2 ] ----+---> [ Telco MPLS / WAN Provider Backbone ] <-----------------------> [ MSEE Pair ] |
|     [ HQ Core  ] ----+                                                                            |
|                                                                                                   |
|  4. ExpressRoute Direct (Dedicated Dual 10G / 100G Direct Fiber Ports)                             |
|     [ Customer Edge Router 1 ] <============== 10G/100G Direct Cross-Connect ==============> [ MSEE 1 ]   |
|     [ Customer Edge Router 2 ] <============== 10G/100G Direct Cross-Connect ==============> [ MSEE 2 ]   |
+---------------------------------------------------------------------------------------------------+

Connectivity Models Breakdown

ExpressRoute Direct Architecture & MACsec Encryption

ExpressRoute Direct provides direct physical connectivity into Microsoft's global backbone without an intermediary exchange provider. Key architectural capabilities include:

1. Dual Physical Ports: Provisioned as active-active port pairs (e.g., $2 \times 100\text{ Gbps}$ or $2 \times 10\text{ Gbps}$) landing on distinct MSEE chassis in a specific peering location.
2. Multi-Circuit Hosting: On a 100 Gbps ExpressRoute Direct port pair, administrators can carve out multiple isolated 1 Gbps, 2 Gbps, 5 Gbps, 10 Gbps, 40 Gbps, or 100 Gbps standard ExpressRoute circuits across different Azure subscriptions and routing domains.
3. Line-Rate MACsec (802.1AE) Encryption: ExpressRoute Direct supports point-to-point Layer 2 hardware encryption between customer edge routers and MSEEs. Pre-shared keys (CAK/CKN) are securely stored in Azure Key Vault and rotated seamlessly without dropping BGP peering sessions.

---

2. ExpressRoute Peering Models: Private vs. Microsoft Peering

ExpressRoute supports two distinct BGP routing domains over a single physical circuit: Azure Private Peering and Microsoft Peering (Public Peering is legacy and deprecated).

+---------------------------------------------------------------------------------------------------+
|                                 EXPRESSROUTE PEERING ARCHITECTURES                                |
|                                                                                                   |
|  +---------------------------------------------------------------------------------------------+  |
|  |                                      Microsoft Cloud                                        |  |
|  |                                                                                             |  |
|  |   +--------------------------------------------------------------------------------------+  |  |
|  |   | Azure Private Peering (RFC 1918 / Private IPs)                                       |  |  |
|  |   | -> Virtual Networks, Spoke VNets, Virtual Network Gateways, FastPath VM NICs         |  |  |
|  |   +--------------------------------------------------------------------------------------+  |  |
|  |                                                                                             |  |
|  |   +--------------------------------------------------------------------------------------+  |  |
|  |   | Microsoft Peering (Public IPs / BGP Communities / Route Filters)                     |  |  |
|  |   | -> Microsoft 365 (Exchange, Teams, SharePoint) [Requires Authorization]              |  |  |
|  |   | -> Azure PaaS (Storage, SQL DB, Key Vault, Azure App Services) [Public Endpoints]    |  |  |
|  |   +--------------------------------------------------------------------------------------+  |  |
|  +----------------------------------------------+----------------------------------------------+  |
|                                                 | (Redundant BGP /30 Subnets)                     |
|                                                 v                                                 |
|  +---------------------------------------------------------------------------------------------+  |
|  |                              Microsoft Enterprise Edge (MSEE)                               |  |
|  +----------------------------------------------+----------------------------------------------+  |
|                                                 | (Primary & Secondary VLANs)                     |
|                                                 v                                                 |
|  +---------------------------------------------------------------------------------------------+  |
|  | On-Premises Customer Edge (CE) Routers                                                      |  |
|  | * NAT Gateway / Public IP Pool (Required for Microsoft Peering Outbound SNAT)               |  |
|  | * Private IP Addressing (Private Peering)                                                   |  |
|  +---------------------------------------------------------------------------------------------+  |
+---------------------------------------------------------------------------------------------------+

Peering Comparison Matrix

Microsoft Peering NAT & Route Filter Mechanics

1. Public IP Validation: Public IP prefixes used for BGP peering and outbound NAT must be registered to the enterprise in a Regional Internet Registry (RIR) such as ARIN, RIPE, or APNIC, or validated through Routing Assets Database (RADb).
2. NAT Architecture: Customer Edge routers must implement Source NAT (SNAT) so that internal client private IPs (e.g., 10.1.0.0/16) are translated to public IPs before packets cross the Microsoft Peering BGP boundary. The NAT IP pool must not overlap with the /30 peering IP addresses.
3. Route Filters & BGP Communities: By default, an activated Microsoft Peering session receives zero routes. To receive BGP routes, administrators create an Azure Route Filter resource containing specific BGP community strings and link it to the ExpressRoute circuit:
   • 12076:51004 — Azure Global Services (Storage, SQL, Cosmos DB, Azure Monitor)
   • 12076:5010 — Microsoft 365 Exchange Online (Requires Microsoft approval)
   • 12076:5020 — Microsoft 365 SharePoint Online / OneDrive (Requires Microsoft approval)
   • 12076:5030 — Microsoft 365 Teams / Skype for Business (Requires Microsoft approval)

---

3. ExpressRoute Virtual Network Gateways & SKUs

When connecting an Azure Virtual Network to an ExpressRoute circuit via Private Peering, you must deploy an ExpressRoute Virtual Network Gateway inside the dedicated GatewaySubnet.

ExpressRoute Gateway SKU Specifications

[!IMPORTANT] GatewaySubnet Sizing for ExpressRoute: The GatewaySubnet must be sized at /27 minimum, but /26 is the recommended best practice. A /26 subnet provides 59 usable host IPs, allowing concurrent deployment of an ExpressRoute Gateway, an Azure VPN Gateway (for co-existence failover), and necessary maintenance instances allocated during platform host updates.

---

4. ExpressRoute FastPath Architecture

In standard ExpressRoute Private Peering, all packets entering the Azure VNet from an on-premises network must pass through the virtual network gateway VM instances in the GatewaySubnet before reaching target virtual machines.

+-------------------------------------------------------------------------------------------------+
|                            STANDARD GATEWAY ROUTING vs. FASTPATH                                |
|                                                                                                 |
|   STANDARD ROUTING (Bottleneck at Gateway CPU):                                                 |
|   [ On-Premises ] ---> [ MSEE ] ---> [ GatewaySubnet (VGW VMs: 1-9.5 Gbps) ] ---> [ VM NIC ]   |
|                                                                                                 |
|   FASTPATH ROUTING (Bypasses Gateway CPU completely):                                           |
|   [ On-Premises ] ---> [ MSEE ] ================================================> [ VM NIC ]   |
|                                        (Direct Host SmartNIC Data Plane)                        |
|                                                                                                 |
|   * Note: ExpressRoute Gateway is still required for BGP Control Plane route exchange!          |
+-------------------------------------------------------------------------------------------------+

FastPath Operational Mechanics

1. Data Plane Bypass: When FastPath is enabled on the ExpressRoute Connection resource, Azure programs routing tables directly into the host hypervisors and Azure SmartNICs. Packets travel directly between the MSEE router and the destination VM NIC at full wire speed.
2. Throughput Scaling: FastPath removes the gateway CPU bottleneck, scaling network throughput to 10 Gbps, 40 Gbps, and 100 Gbps when paired with ExpressRoute Direct.
3. Latency Reduction: Bypassing gateway VM hops reduces inter-packet jitter and improves round-trip latency for high-frequency trading and big-data analytics.
4. Prerequisites & SKUs: FastPath requires an UltraPerformance or ErGw3AZ (or ErGw1AZ/2AZ for standard FastPath configurations) gateway SKU.

FastPath Feature Support & Limitations

• Virtual Network Peering Transit: FastPath supports peered VNets. Traffic from on-premises to a peered spoke VNet bypasses the gateway completely.
• Private Endpoints: FastPath routes traffic directly to Azure Private Endpoints deployed in the VNet without gateway throttling.
• Internal Load Balancers (ILB): Supports backend pools behind ILBs across all SKUs.
• Control Plane Dependency: The ExpressRoute Gateway cannot be deleted; it is strictly required to maintain eBGP neighbor peering sessions with the MSEEs.
• UDR Bypassing: By default, User-Defined Routes (UDRs) associated with the GatewaySubnet are bypassed by FastPath. To force FastPath traffic through an NVA/Firewall, specific UDR-on-GatewaySubnet FastPath support must be explicitly enabled.

---

5. MTU, Packet Size & Fragmentation

ExpressRoute does not support jumbo frames. This is one of the most commonly mis-remembered ExpressRoute facts, and it is worth memorising precisely because plausible-sounding 9,000-byte answers appear as distractors.

Fragmentation Behaviour

• ExpressRoute gateways do not support fragmented packets. A fragmented packet arriving at the gateway is dropped rather than reassembled, so the application must avoid producing IP fragmentation in the first place.
• The supported mitigations are application-side: reduce the sending MTU, rely on Path MTU Discovery (PMTUD), or apply TCP MSS clamping on the on-premises edge router.
• If your workload genuinely requires IP fragmentation support, the documented option is to enable ExpressRoute FastPath, which bypasses the ExpressRoute gateway data plane entirely.

[!CAUTION] The classic MTU failure signature: small requests (a TCP handshake, an ICMP ping, a DNS lookup) succeed, but large transfers — a file copy, a database bulk load, a TLS handshake carrying a large certificate chain — hang or reset. That asymmetry between "small works, large fails" over a hybrid link is the exam's cue for an MTU/fragmentation problem, not a BGP, NSG, or firewall problem.