Azure : BGP ExpressRoute and BGP over IPSec Tunnel
Note: Special thanks to Navisite leader and my Manager John Rudenauer for his continued support on this blog series.
Azure BGP ExpressRoute
Summary
This is the fourth blog in the Azure Networking Blog series. Microsoft Azure BGP ExpressRoute lets you extend your on-premises networks into the Microsoft cloud over a private connection facilitated by a connectivity provider. ExpressRoute connections do not go over the public Internet, and offer higher security, reliability, and speeds with lower latencies than typical connections over the Internet.
Check out other blogs in this series:
- Azure Traffic Manager and Load Balancer Design
- Citrix VPX in Azure – Third Party Network Devices (Part-1)
- Cisco ASAv in Azure – Third Party Network Devices (Part-2)
This blog article is a deep dive on implementing BGP ExpressRoute with Private Peering and setting up site-to-site VPN (with BGP) in Azure. Microsoft has some great documentation and this blog is intended to give you complete configuration snippets for a real world use-case along with troubleshooting and verification techniques.
This BGP ExpressRoute use case is for a business that needs high-speed, secure connection with predictable performance and latency. This secure connection bypasses the Internet and if needed can co-exist with a backup VPN failover design. The requirement is to use Azure Compute (IaaS and PaaS) using Private peering as trusted extension of core on-prem networks into Azure VNets.
BGP ExpressRoute Design and Configuration
Reference Architecture Diagram
ExpressRoute Terminology:
First some new terminology to get familiar with. Microsoft has some some great documentation and I’ve provided the link.
Connectivity Models: CloudExchange Co-location, Point-to-Point Ethernet Connection, and Any-to-Any (IPVPN) Connection. This use case is for Point-to-Point Ethernet Connection
Express Route Routing Domains: Azure public, Azure private, and Microsoft.
Bandwidth and Speed options: You can purchase ExpressRoute circuits for a wide range of bandwidth 50Mbps ->10Gbps.
ExpressRoute virtual network gateway SKUs: Standard, HighPerformance, UltraPerformance. Check out the link for estimated performance by gateway SKU.
ExpressRoute Pricing: Metered Data plan, Unlimited Data plan, ExpressRoute Premium Add-on. ExpressRoute Premium is Global connectivity for services.
ExpressRoute Configuration:
Reference Documentation:
https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-circuit-portal-resource-manager
1.Create an ExpressRoute Circuit
All Services -> ExpressRoute Circuits -> Add
Note: The Peering Location indicates the physical location where you are peering with Microsoft. This is not linked to “Location” property, which refers to the geography where the Azure network resource provider is located. While they are not related, it is a good practice to choose a network resource provider geographically close to the peering location of the circuit. In my recent experience troubleshooting an issue, I’ve found that this adds extra latency if not architected correctly. Notice the latency graph generated by Azure Network Performance Monitoring
2.View the Properties of the Circuit
Click on the newly-created circuit. The Provider status provides information on the current state of provisioning on the service-provider side. Circuit status provides the state on the Microsoft side. When you create a new ExpressRoute circuit, the circuit is in the following state:
Provider status: Not provisioned
Circuit status: Enabled
For you to be able to use an ExpressRoute circuit, it must be in the following state:
Provider status: Provisioned
Circuit status: Enabled
3. Send the Service Key to Your Connectivity Provider for Provisioning
4. Periodically Check the Status and the State of the Circuit Key
You can view the properties of the circuit that you’re interested in by selecting it. Check the Provider status and ensure that it has moved to Provisioned before you continue.
5. Create Your Routing Configuration
The circuit has to be in enabled and provisioned state. Go to Peering-> Private Peering.
6. Configure Azure Private Peering for the Circuit.
6. Create the Virtual Network Gateway
7. Link VNG to ExpressRoute Circuit
Note: The ExpressRoute Circuit has to be in provisioned state
Verifications Using Powershell:
# # Get ExpressRoute Circuit Information # Get-AzureRmExpressRouteCircuit -ResourceGroupName nn-rg # # Validate Peering Configuration # $ckt = Get-AzureRmExpressRouteCircuit -ResourceGroupName nn-rg -Name nn-ExpressRoute-circuit Get-AzureRmExpressRouteCircuitPeeringConfig -Name "AzurePrivatePeering" -ExpressRouteCircuit $ckt # # Validate ARP between Microsoft and service provider # Get-AzureRmExpressRouteCircuitARPTable -ResourceGroupName nn-rg -ExpressRouteCircuitName nn-ExpressRoute-circuit -PeeringType AzurePrivatePeering -DevicePath Primary # # Validate BGP and routes # Get-AzureRmExpressRouteCircuitRouteTable -ResourceGroupName nn-rg -ExpressRouteCircuitName nn-ExpressRoute-circuit -PeeringType AzurePrivatePeering -DevicePath Primary # # Check status # Get-AzureRmExpressRouteCircuitStats -ResourceGroupName nn-rg -ExpressRouteCircuitName nn-ExpressRoute-circuit -PeeringType AzurePrivatePeering # # Get Route table information # Get-AzureRmRouteTable -ResourceGroupName nn-rg # Get BGP status Get-AzureRmVirtualNetworkGatewayBGPPeerStatus -VirtualNetworkGatewayName VNG-SC-EXPROUTE-GW-01 -ResourceGroupName ExpRoute
Azure : BGP Over IPSec VPN
Use Case:
BGP is an optional feature on the Azure Route-Based VPN gateways. You can continue to use Azure VPN gateways and your on-premises VPN devices without BGP. If your on-premises VPN devices support BGP, it enables the Azure VPN Gateways and your on-premises VPN devices to become BGP peers or neighbors, to exchange “routes”. This use case is the equivalent of using static routes (without BGP) vs. using dynamic routing with BGP between your networks and Azure.
Design:
We will build the configuration shown in the diagram below
Pre-requisites:
Resource Group (nn-east-rg) and a vNET (nn-east-vnet) with Gateway Subnet Exists
1. Create the Virtual Network Gateway
More Services -> Virtual Network Gateway -> Create. Creating Virtual Network gateway can take upto 45 minutes. Configure BGP with ASN Number.
From Azure BGP FAQ:
Can I use the same ASN for both on-premises VPN networks and Azure VNets?
No, you must assign different ASNs between your on-premises networks and your Azure VNets, if you are connecting them together with BGP. Azure VPN Gateways have a default ASN of 65515 assigned, whether BGP is enabled or not for your cross-premises connectivity. You can override this default by assigning a different ASN when creating the VPN gateway, or change the ASN after the gateway is created. You will need to assign your on-premises ASNs to the corresponding Azure Local Network Gateways.
2. Create the Local Network Gateway
This defines the gateway parameters for the On-Prem Firewall/VPN Gateway. The minimum prefix that you need to declare for the local network gateway is the host address of your BGP peer IP address on your VPN device. In this case, it’s a /32 prefix of 10.255.78.9/32. You must use different BGP ASNs between your on-premises networks and the Azure virtual network. Refer to Azure BGP FAQ for more information on ASNs.
More Services -> Local Network Gateway -> Create
3. Connect the VNet Gateway and the local network gateway using a Connection
More Services -> Connection -> Create
It is possible to have both BGP and non-BGP connections on the same Azure VPN gateway. Unless BGP is enabled in the connection property, Azure will not enable BGP for this connection, even though BGP parameters are already configured on both gateways.
4. On –Prem ASA Configuration
Create route Route based VPN Configuration on the Cisco ASA
On-Prem Cisco ASA Configuration:
asav982# sh int ip brief | i up GigabitEthernet0/0 192.168.1.78 YES CONFIG up up GigabitEthernet0/1 192.168.190.78 YES CONFIG up up Management0/0 unassigned YES unset administratively down up Tunnel2 10.255.78.9 YES CONFIG up up asav982# sh ip add System IP Addresses: Interface Name IP address Subnet mask Method GigabitEthernet0/0 outside 192.168.1.78 255.255.255.0 CONFIG GigabitEthernet0/1 inside 192.168.190.78 255.255.255.0 CONFIG Tunnel2 vti-azure4 10.255.78.9 255.255.255.252 CONFIG asav982# sh run interface tunnel 2 ! interface Tunnel2 nameif vti-azure4 ip address 10.255.78.9 255.255.255.252 tunnel source interface outside tunnel destination 52.167.227.163 tunnel mode ipsec ipv4 tunnel protection ipsec profile azure asav982# sh run tunnel-group tunnel-group 52.167.227.163 type ipsec-l2l tunnel-group 52.167.227.163 ipsec-attributes ikev2 remote-authentication pre-shared-key ***** ikev2 local-authentication pre-shared-key ***** asav982# sh run route route outside 0.0.0.0 0.0.0.0 192.168.1.1 1 route vti-azure4 10.5.0.254 255.255.255.255 52.167.227.163 1 asav982# sh run router bgp router bgp 65525 bgp log-neighbor-changes bgp graceful-restart bgp router-id 10.255.78.9 address-family ipv4 unicast neighbor 10.5.0.254 remote-as 65515 neighbor 10.5.0.254 ebgp-multihop 255 neighbor 10.5.0.254 activate network 192.168.190.0 maximum-paths 2 no auto-summary no synchronization exit-address-family
Verification:
asav982# sh vpn-sessiondb l2l Session Type: LAN-to-LAN Connection : 52.167.227.163 Index : 62735 IP Addr : 52.167.227.163 Protocol : IKEv2 IPsecOverNatT Encryption : IKEv2: (1)AES256 IPsecOverNatT: (1)AES256 Hashing : IKEv2: (1)SHA1 IPsecOverNatT: (1)SHA1 Bytes Tx : 6637 Bytes Rx : 6074 Login Time : 12:05:47 UTC Sun Mar 18 2018 Duration : 0h:26m:09s asav982# sh bgp summary BGP router identifier 10.255.78.9, local AS number 65525 BGP table version is 6, main routing table version 6 4 network entries using 800 bytes of memory 4 path entries using 320 bytes of memory 2/2 BGP path/bestpath attribute entries using 416 bytes of memory 1 BGP AS-PATH entries using 24 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 1560 total bytes of memory BGP activity 34/30 prefixes, 34/30 paths, scan interval 60 secs Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 10.5.0.254 4 65515 27 23 6 0 0 00:21:47 3 asav982# sh route bgp Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2, V - VPN i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, + - replicated route Gateway of last resort is 192.168.1.1 to network 0.0.0.0 B 10.5.0.0 255.255.0.0 [20/0] via 10.5.0.254, 00:20:47 B 192.168.45.0 255.255.255.0 [20/0] via 10.5.0.254, 00:20:47 asav982# sh bgp neighbors 10.5.0.254 advertised-routes BGP table version is 6, local router ID is 10.255.78.9 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path *> 192.168.190.0 0.0.0.0 0 32768 i
5. Powershell Commands to Validate
PS C:\Users\nehali> Get-AzureRmVirtualNetworkGatewayBGPPeerStatus -VirtualNetworkGatewayName nn-east-vng-route-based -ResourceGroupName nn-rg-east Asn : 65525 ConnectedDuration : 00:41:46.0062819 LocalAddress : 10.5.0.254 MessagesReceived : 42 MessagesSent : 52 Neighbor : 10.255.78.9 RoutesReceived : 1 State : Connected PS C:\Users\nehali> Get-AzureRmVirtualNetworkGatewayLearnedRoute -VirtualNetworkGatewayName nn-east-vng-route-based -ResourceGroupName nn-rg-east AsPath : LocalAddress : 10.5.0.254 Network : 10.5.0.0/16 NextHop : Origin : Network SourcePeer : 10.5.0.254 Weight : 32768 AsPath : LocalAddress : 10.5.0.254 Network : 192.168.45.0/24 NextHop : Origin : Network SourcePeer : 10.5.0.254 Weight : 32768 AsPath : LocalAddress : 10.5.0.254 Network : 10.255.78.9/32 NextHop : Origin : Network SourcePeer : 10.5.0.254 Weight : 32768 AsPath : 65525 LocalAddress : 10.5.0.254 Network : 192.168.190.0/24 NextHop : 10.255.78.9 Origin : EBgp SourcePeer : 10.255.78.9 Weight : 32768 PS C:\Users\nehali> Get-AzureRmVirtualNetworkGatewayAdvertisedRoute -VirtualNetworkGatewayName nn-east-vng-route-based -ResourceGroupName nn-rg-east -peer 10.255.78.9 AsPath : 65515 LocalAddress : 10.5.0.254 Network : 10.5.0.0/16 NextHop : 10.5.0.254 Origin : Igp SourcePeer : Weight : 0 AsPath : 65515 LocalAddress : 10.5.0.254 Network : 192.168.45.0/24 NextHop : 10.5.0.254 Origin : Igp SourcePeer : Weight : 0 AsPath : 65515 LocalAddress : 10.5.0.254 Network : 10.255.78.9/32 NextHop : 10.5.0.254 Origin : Igp SourcePeer : Weight : 0
Putting It All Together
For an advanced Failover setup, Azure Site-to-Site VPN and ExpressRoute can co-exist and has several advantages. Check out the following Link
https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-coexist-classic
Note: From Azure Documentation:
Limits and Limitations
- Transit routing is not supported.You cannot route (via Azure) between your local network connected via Site-to-Site VPN and your local network connected via ExpressRoute.
- Point-to-Site is not supported.You can’t enable point-to-site VPN connections to the same VNet that is connected to ExpressRoute. Point-to-Site VPN and ExpressRoute cannot coexist for the same VNet.
- Forced tunneling cannot be enabled on the Site-to-Site VPN gateway.You can only “force” all Internet-bound traffic back to your on-premises network via ExpressRoute.
- Basic SKU gateway is not supported.You must use a Non-Basic SKU gateway for both the ExpressRoute gateway and the VPN gateway.
- Only route-based VPN gateway is supported.You must use a route-based VPN Gateway.
- Static route should be configured for your VPN gateway.If your local network is connected to both ExpressRoute and a Site-to-Site VPN, you must have a static route configured in your local network to route the Site-to-Site VPN connection to the public Internet.
- ExpressRoute gateway must be configured first.You must create the ExpressRoute gateway first before you add the Site-to-Site VPN gateway.