4.2 ExpressRoute Routing, Resiliency & Global Reach
Key Takeaways
- Each ExpressRoute circuit consists of an active-active pair of independent physical connections terminating on two separate MSEE routers, providing a 99.95% single-circuit availability SLA.
- To influence Azure-to-on-premises routing, advertise AS Path prepending or MED from on-premises; to influence on-premises-to-Azure routing, configure BGP Local Preference on customer edge routers.
- When identical address prefixes are advertised over both ExpressRoute and a co-existing Site-to-Site VPN, Azure strictly prefers ExpressRoute due to higher internal route preference.
- Standard ExpressRoute circuits support advertising up to 4,000 routes from on-premises; enabling ExpressRoute Premium increases this limit to 10,000 routes.
- ExpressRoute Global Reach connects disparate on-premises branch offices directly across the Microsoft global backbone by establishing BGP peering between MSEEs across ExpressRoute Premium circuits.
Enterprise hybrid architectures demand continuous connectivity and deterministic traffic engineering. Achieving high resiliency with Azure ExpressRoute requires a thorough understanding of Border Gateway Protocol (BGP) routing mechanics, active-active MSEE hardware topologies, ExpressRoute and Site-to-Site VPN co-existence failover rules, and ExpressRoute Global Reach.
1. BGP Routing Mechanics over ExpressRoute
ExpressRoute utilizes external Border Gateway Protocol (eBGP) to dynamically exchange network prefixes between on-premises Customer Edge (CE) routers and Microsoft Enterprise Edge (MSEE) routers.
Core BGP Peering Parameters
- Microsoft ASN:
12076(Fixed globally for all ExpressRoute Private and Microsoft Peering BGP sessions). - Customer ASN: Can use a public 16-bit or 32-bit ASN registered to the customer, or a private ASN in the ranges
64512 - 65534or4200000000 - 4294967294. - BGP MD5 Authentication: Supported and strongly recommended to prevent spoofing and unauthorized route injection across the peering links.
- Point-to-Point Peering Subnets: Each peering requires two dedicated
/30(or/31) subnets for the primary and secondary links. In Private Peering, these can be RFC 1918 addresses; in Microsoft Peering, they must be publicly registered IPv4 addresses.
Route Advertisement Limits & ExpressRoute Premium
| Feature / Metric | Standard ExpressRoute Circuit | ExpressRoute with Premium Add-on |
|---|---|---|
| Max Routes Advertised to Azure | 4,000 routes (BGP session drops if exceeded) | 10,000 routes |
| Geographic VNet Connectivity | VNets within the same geopolitical region | Global VNet connectivity (connect to VNets in any Azure region worldwide) |
| Max VNet Links per Circuit | 10 VNets (Standard) | Up to 100 VNets (depending on bandwidth tier) |
| ExpressRoute Global Reach | Not supported | Supported (Enables branch-to-branch data transit) |
| Microsoft 365 Peering | Not supported | Included (subject to Microsoft authorization) |
[!WARNING] Route Limit Enforcement: If an on-premises router advertises more than 4,000 prefixes on a standard circuit, Azure will immediately reset the BGP session and drop all traffic. Network engineers must configure route summarization (aggregation) at the on-premises edge or enable the ExpressRoute Premium add-on.
2. Active-Active MSEE Resiliency Architecture
Every ExpressRoute circuit is provisioned by default with two redundant physical connections terminating on two distinct Microsoft Enterprise Edge (MSEE) routers in the same peering facility.
+---------------------------------------------------------------------------------------------------+
| EXPRESSROUTE ACTIVE-ACTIVE MSEE PAIR ARCHITECTURE |
| |
| +---------------------------------------------------------------------------------------------+ |
| | Azure Virtual Network | |
| | [ ExpressRoute Gateway (ErGw3AZ) ] | |
| +-----------------------------------+-------------------------------------+-------------------+ |
| | | |
| v v |
| +-----------------------------------+-------------------------------------+-------------------+ |
| | Meet-Me Peering Facility | | | |
| | +--------------+--------------+ +--------------+--------------+ | |
| | | MSEE 1 | | MSEE 2 | | |
| | | (Primary) | | (Secondary) | | |
| | +--------------+--------------+ +--------------+--------------+ | |
| +-----------------------------------|-------------------------------------|-------------------+ |
| | | |
| Primary Link (VLAN 100) Secondary Link (VLAN 100) |
| Active BGP Session Active BGP Session |
| | | |
| v v |
| +-----------------------------------+-------------------------------------+-------------------+ |
| | On-Premises Edge +--------------+--------------+ +--------------+--------------+ | |
| | | Customer Router 1 (CE1) | <===> | Customer Router 2 (CE2) | | |
| | +-----------------------------+ (iBGP)+-----------------------------+ | |
| +---------------------------------------------------------------------------------------------+ |
+---------------------------------------------------------------------------------------------------+
Single Circuit SLA vs. Multi-Circuit Disaster Recovery
- Single Circuit Redundancy (99.95% SLA):
- Connecting to both MSEE routers (Primary and Secondary) across dual Customer Edge routers provides an Azure SLA of 99.95%.
- If Customer Edge routers connect to only one MSEE (single-homed), Microsoft does not provide an availability SLA, and planned MSEE firmware upgrades will cause total circuit downtime.
- Dual-Circuit Multi-Site Disaster Recovery (99.99% SLA):
- To protect against catastrophic peering facility outages (e.g., fiber cut, power grid failure at meet-me site), enterprises deploy two separate ExpressRoute circuits in geographically diverse peering locations (e.g., Silicon Valley and Ashburn).
- Connecting the Azure VNet Gateway to both circuits provides maximum resilience with a 99.99% SLA.
3. BGP Path Selection & Traffic Engineering
When multiple paths exist between Azure and on-premises networks, network engineers use standard BGP attributes to control symmetrical traffic flow.
1. Controlling Inbound Traffic to On-Premises (Azure -> On-Premises)
To direct traffic exiting Azure toward a preferred primary on-premises path:
- AS Path Prepending (Recommended): The on-premises edge router prepends its own Autonomous System Number multiple times (e.g.,
65001 65001 65001) when advertising routes over the secondary/backup path. Azure BGP path selection prefers the path with the shortest AS Path length. - Multi-Exit Discriminator (MED): On-premises routers can advertise a lower MED value on the primary link (e.g., MED 50) and a higher MED on the secondary link (e.g., MED 100). Azure honors MED when paths originate from the same ASN.
2. Controlling Outbound Traffic from On-Premises (On-Premises -> Azure)
To direct traffic exiting the corporate network toward Azure across the primary link:
- BGP Local Preference: Set a higher Local Preference (e.g.,
Local-Pref 200) on the customer edge router receiving routes from the primary circuit, and a lower Local Preference (e.g.,Local-Pref 100) on the secondary circuit. Higher Local Preference wins inside the on-premises autonomous system.
BGP Decision Hierarchy Summary
| Direction | Preferred BGP Attribute | Configuration Location | Rule |
|---|---|---|---|
| Azure $\rightarrow$ On-Premises | AS Path Prepending | On-premises egress BGP route-map | Shorter AS Path is preferred by MSEE/Azure. |
| Azure $\rightarrow$ On-Premises | MED | On-premises egress BGP route-map | Lower MED is preferred (if AS paths are equal). |
| On-Premises $\rightarrow$ Azure | Local Preference | On-premises ingress BGP route-map | Higher Local Preference is preferred inside enterprise AS. |
| Both Directions | Longest Prefix Match | IP Routing Table | Longest prefix match always wins over any BGP attribute. |
4. ExpressRoute & Site-to-Site VPN Co-existence and Failover
Many enterprises deploy an Azure Site-to-Site (S2S) VPN Gateway alongside an ExpressRoute Gateway in the same VNet to serve as an automatic, cost-effective backup path.
+---------------------------------------------------------------------------------------------------+
| EXPRESSROUTE & S2S VPN CO-EXISTENCE ARCHITECTURE |
| |
| +---------------------------------------------------------------------------------------------+ |
| | Azure Virtual Network (10.200.0.0/16) | |
| | | |
| | +---------------------------------------------------------------------------------------+ | |
| | | GatewaySubnet (10.200.255.0/26) | | |
| | | [ ExpressRoute Gateway (ErGw3AZ) ] [ Route-Based VPN Gateway (VpnGw3AZ) ] | | |
| | +-----------------------+-----------------------------------+---------------------------+ | |
| +--------------------------|-----------------------------------|------------------------------+ |
| | | |
| PRIMARY PATH (ExpressRoute) BACKUP PATH (Encrypted IPsec / Internet) |
| Route Preference: HIGHER (Default) Route Preference: LOWER (Default) |
| | | |
| v v |
| +---------------------------------------------------------------------------------------------+ |
| | On-Premises Datacenter (192.168.0.0/16) | |
| | * Advertises 192.168.0.0/16 over ExpressRoute BGP | |
| | * Advertises 192.168.0.0/16 over VPN BGP (with AS Path Prepend / Lower Local-Pref) | |
| +---------------------------------------------------------------------------------------------+ |
+---------------------------------------------------------------------------------------------------+
Azure Route Preference Hierarchy in Co-existence
When a VNet contains both an ExpressRoute Gateway and a VPN Gateway, Azure applies a strict built-in routing hierarchy:
- Identical Prefixes (Automatic Failover):
- If the exact same prefix (e.g.,
192.168.0.0/16) is advertised over both ExpressRoute and S2S VPN, Azure automatically prefers ExpressRoute. - ExpressRoute has a higher default administrative route preference in the Azure SDN stack than VPN Gateway routes.
- If the ExpressRoute circuit fails or its BGP session drops, Azure automatically and instantly redirects outbound traffic through the S2S VPN Gateway.
- If the exact same prefix (e.g.,
- Specific vs. Summary Prefixes (The LPM Trap):
- If on-premises advertises a summary route (e.g.,
192.168.0.0/16) over ExpressRoute, but advertises specific subnet routes (e.g.,192.168.1.0/24) over the S2S VPN, Azure will route traffic to192.168.1.0/24over the VPN because Longest Prefix Match overrides route type preference. - Best Practice: Always advertise identical prefix masks over both paths to ensure deterministic failover.
- If on-premises advertises a summary route (e.g.,
5. ExpressRoute Global Reach
ExpressRoute Global Reach allows enterprises to interconnect disparate on-premises branch offices and regional datacenters directly over Microsoft's dark fiber backbone using their existing ExpressRoute circuits.
Global Reach Architecture & Capabilities
- Backbone Transit: Links two ExpressRoute circuits in different peering locations (e.g., London and Singapore). On-premises traffic entering the London MSEE travels across the Microsoft global backbone directly to the Singapore MSEE and exits to the Singapore branch.
- BGP Peering over MSEEs: The on-premises routers establish end-to-end BGP peering through the MSEEs. Prefixes learned at one datacenter are automatically propagated to the other.
- Bypasses Azure VNets: Global Reach traffic does not transit through an Azure Virtual Network Gateway or VNet. Communication occurs entirely at the edge on the MSEE routing plane.
- Prerequisites: Requires ExpressRoute Premium SKU on all participating circuits and connectivity through supported Global Reach peering locations.
An enterprise maintains two ExpressRoute circuits in different peering locations (Primary Circuit in Chicago and Secondary Circuit in Dallas) connecting to the same Azure VNet. Network operations reports that outbound traffic from Azure to the corporate datacenter is unpredictably splitting across both circuits. The company requires all Azure-to-on-premises traffic to prefer the Chicago circuit unless it experiences an outage. How should the network engineer configure BGP routing?
A healthcare organization implements hybrid connectivity to Azure using an ExpressRoute circuit as the primary link and an active-active Route-Based Site-to-Site VPN as an automated backup. Both the ExpressRoute Gateway and VPN Gateway reside in the same GatewaySubnet (/26). On-premises routers dynamically advertise prefix 10.50.0.0/16 over BGP across both the ExpressRoute circuit and the VPN tunnels. Under normal operational conditions, how does Azure route outbound traffic destined for 10.50.0.0/16?
A global manufacturing firm has two regional datacenters in London and Tokyo, each connected to Azure via a dedicated ExpressRoute circuit. The company needs to establish direct, high-bandwidth data replication between the London and Tokyo datacenters. Corporate policy forbids hair-pinning traffic through an Azure Virtual Network or deploying software-based transit virtual appliances. Which Azure ExpressRoute feature meets this requirement?
An enterprise with a standard ExpressRoute circuit attempts to advertise 4,800 network prefixes from its core MPLS network to Azure over an active Private Peering BGP session. Shortly after applying the route redistribution script, network administrators report complete loss of connectivity to all Azure workloads. Diagnostic metrics show the BGP session state is 'Idle'. What is the root cause of this failure and how should it be resolved?