Preferring MPLS VPN BGP Path with IGP Backup

Summary

Over the last few years MPLS VPN services have gained popularity as an alternative network connectivity transport option over legacy TDM networks. One of the most popular challenges with the MPLS VPN design is the Layer3 routing interaction between the customer network and the service provider routing. A common scenario is when there is a primary BGP path over the MPLS VPN network and a redundant routing path over a non MPLS VPN network. This is exposed in many networks that have an eBGP peering session with the MPLS VPN provider and routes are learned to remote locations but also have a backup path to those same locations over a redundant IGP path. Typically the IGP path learns the routes via a dynamic routing protocol such as EIGRP or OSPF. This TAC Tip describes how to configure the routing such that the preferred path is always selected in both the primary path failure condition as well as the reroute on primary path recovery. Typically with the default configuration the failover works to the backup IGP path. However, the problem comes when the primary recovers.

Sample Network

When an IGP (in this example OSPF) route is redistributed in to BGP it is considered locally generated by BGP and gets assigned a weight of 32768. By default, all routes received from a BGP peer are assigned a local weight of 0. When doing BGP path comparison weight is the first attribute compared. Therefore, if the same prefix must be compared, the locally originated prefix with the higher weight will be installed in the routing table based on the BGP best path selection process. Let's first walk through an example of how the problem surfaces.

Take this simple network example:

R1 is the end customer (CPE) router that has two parallel paths to reach the remote 192.168.1.0/24 subnet. One path is an OSPF learned path and the other is an eBGP learned route from the MPLS PE router over the MPLS VPN network.

When the MPLS VPN network is up the eBGP route is selected as the best path based on the higher administrative distance (20 for eBGP and 110 for OSPF).

R1#show ip bgp 192.168.1.0 255.255.255.0

BGP routing table entry for 192.168.1.0/24, version 1562

Paths: (1 available, best #1, table default)

Flag: 0x820

Not advertised to any peer

65000

172.16.56.6 from 172.16.56.6 (192.168.8.1)

Origin IGP, metric 0, localpref 100, valid, external, best

R1#show ip route 192.168.1.0

Routing entry for 192.168.1.0/24

Known via "bgp 65001", distance 20, metric 0

Tag 65000, type external

Last update from 172.16.56.6 00:08:14 ago

Routing Descriptor Blocks:

* 172.16.56.6, from 172.16.56.6, 00:08:14 ago

Route metric is 0, traffic share count is 1

AS Hops 1

Route tag 65000

The OSPF learned route to 192.168.1.0/24 is there as a candidate path in the OSPF database.

R1#show ip ospf data router 3.3.3.3

OSPF Router with ID (5.5.5.5) (Process ID 1)

Router Link States (Area 0)

LS age: 225

Options: (No TOS-capability, DC)

LS Type: Router Links

Link State ID: 3.3.3.3

Advertising Router: 3.3.3.3

LS Seq Number: 8000138E

Checksum: 0x3AF3

Length: 48

Number of Links: 2

Link connected to: a Transit Network

(Link ID) Designated Router address: 172.16.35.5

(Link Data) Router Interface address: 172.16.35.3

Number of MTID metrics: 0

TOS 0 Metrics: 10

Link connected to: a Stub Network

(Link ID) Network/subnet number: 192.168.1.0

(Link Data) Network Mask: 255.255.255.0

Number of MTID metrics: 0

TOS 0 Metrics: 1

Now assume the link to the MPLS VPN network fails and we lose the eBGP route. Under this condition the OSPF backup route will be installed in the routing table. Here is the routing table debug showing the backup OSPF route going in the routing table.

RT: del 192.168.1.0 via 172.16.56.6, bgp metric [20/0]

RT: delete network route to 192.168.1.0/24

RT: updating ospf 192.168.1.0/24 (0x0) via 172.16.35.3 Et1/0

RT: add 192.168.1.0/24 via 172.16.35.3, ospf metric [110/11]

R1#show ip route 192.168.1.0

Routing entry for 192.168.1.0/24

Known via "ospf 1", distance 110, metric 11, type intra area

Redistributing via bgp 65001

Advertised by bgp 65001 match internal external 1 & 2

Last update from 172.16.35.3 on Ethernet1/0, 00:00:09 ago

Routing Descriptor Blocks:

* 172.16.35.3, from 3.3.3.3, 00:00:09 ago, via Ethernet1/0

Route metric is 11, traffic share count is 1

At this stage the routing has reconverged to the IGP backup path and everything is ok. However, notice that the output above shows the route is being redistributed in to BGP. This is because the router is doing OSPF to BGP redistribution to get the local OSPF learned routes in to BGP in order to be advertised over the MPLS VPN network.

Here is the entry in the BGP table showing it is now locally sourced with a weight of 32768.

R1#show ip bgp 192.168.1.0 255.255.255.0

BGP routing table entry for 192.168.1.0/24, version 1564

Paths: (1 available, best #1, table default)

Flag: 0x820

Not advertised to any peer

Local

172.16.35.3 from 0.0.0.0 (5.5.5.5)

Origin incomplete, metric 11, localpref 100, weight 32768, valid, sourced, best

Now let's say that the primary link to the MPLS VPN router comes back up and the eBGP session recovers such that we learn the 192.168.1.0/24 network over the eBGP session again.

BGP(0): 172.16.56.6 rcvd UPDATE w/ attr: nexthop 172.16.56.6, origin i, metric 0, path 65000

BGP(0): 172.16.56.6 rcvd 192.168.1.0/24

R1#show ip bgp 192.168.1.0 255.255.255.0

BGP routing table entry for 192.168.1.0/24, version 1564

Paths: (2 available, best #2, table default)

Flag: 0x820

Advertised to update-groups:

1

65000

172.16.56.6 from 172.16.56.6 (192.168.8.1)

Origin IGP, metric 0, localpref 100, valid, external

Local

172.16.35.3 from 0.0.0.0 (5.5.5.5)

Origin incomplete, metric 11, localpref 100, weight 32768, valid, sourced, best

Even though the AD of the eBGP path (20) is lower than OSPF path (110), we do not install the eBGP learned route into the routing table. Since this prefix is in the routing table via OSPF and is being redistributed into BGP, the BGP table will have both paths and must use the Best Path Selection Algorithm. Routes redistributed into BGP are considered locally originated and get a default weight of 32768. The BGP learned prefix is assigned a weight of 0 by default. Since weight is the first BGP attribute that we compare on Cisco routers, the route with the higher weight is considered the best.

R1#show ip route 192.168.1.0

Routing entry for 192.168.1.0/24

Known via "ospf 1", distance 110, metric 11, type intra area

Redistributing via bgp 65001

Advertised by bgp 65001 match internal external 1 & 2

Last update from 172.16.35.3 on Ethernet1/0, 00:03:05 ago

Routing Descriptor Blocks:

* 172.16.35.3, from 3.3.3.3, 00:03:05 ago, via Ethernet1/0

Route metric is 11, traffic share count is 1

Now the problem is that, even though the BGP link is back up and we are learning prefixes, traffic is still routing over the backup path via OSPF. To resolve this, we need to force the eBGP path to be preferred.

Resolution

One common way to resolve this issue is to set the weight on routes learned from the eBGP peer higher than 32768. When the paths are compared by BGP, the path with the highest weight will be preferred and installed in the routing table.

router bgp 65001

bgp log-neighbor-changes

neighbor 172.16.56.6 remote-as 65000

!

address-family ipv4

no synchronization

redistribute ospf 1 match internal external 1 external 2

neighbor 172.16.56.6 activate

neighbor 172.16.56.6 weight 32769

no auto-summary

exit-address-family

To update the weight on the received update, we must force the peer to send the update again so that we can apply the change inbound.

R1#clear ip bgp 172.16.56.6 soft in

*Feb 10 00:32:01.279: BGP(0): 172.16.56.6 rcvd UPDATE w/ attr: nexthop 172.16.56.6, origin i, metric 0, path 65000

*Feb 10 00:32:01.279: BGP(0): 172.16.56.6 rcvd 192.168.1.0/24

*Feb 10 00:32:01.291: RT: closer admin distance for 192.168.1.0, flushing 1 routes

*Feb 10 00:32:01.291: RT: add 192.168.1.0/24 via 172.16.56.6, bgp metric [20/0]

R1#show ip route 192.168.1.0

Routing entry for 192.168.1.0/24

Known via "bgp 65001", distance 20, metric 0

Tag 65000, type external

Last update from 172.16.56.6 00:01:06 ago

Routing Descriptor Blocks:

* 172.16.56.6, from 172.16.56.6, 00:01:06 ago

Route metric is 0, traffic share count is 1

AS Hops 1

Route tag 65000

R1#show ip bgp 192.168.1.0

BGP routing table entry for 192.168.1.0/24, version 1565

Paths: (1 available, best #1, table default)

Flag: 0x820

Not advertised to any peer

65000

172.16.56.6 from 172.16.56.6 (192.168.8.1)

Origin IGP, metric 0, localpref 100, weight 32769, valid, external, best

To demonstrate how this works, let's assume that the BGP route has been lost and the OSPF route is installed in the routing table.

R1#show ip route 192.168.1.0

Routing entry for 192.168.1.0/24

Known via "ospf 1", distance 110, metric 11, type intra area

Redistributing via bgp 65001

Advertised by bgp 65001 match internal external 1 & 2

Last update from 172.16.35.3 on Ethernet1/0, 00:00:08 ago

Routing Descriptor Blocks:

* 172.16.35.3, from 3.3.3.3, 00:00:08 ago, via Ethernet1/0

Route metric is 11, traffic share count is 1

R1#show ip bgp 192.168.1.0

BGP routing table entry for 192.168.1.0/24, version 1567

Paths: (1 available, best #1, table default)

Flag: 0x820

Not advertised to any peer

Local

172.16.35.3 from 0.0.0.0 (5.5.5.5)

Origin incomplete, metric 11, localpref 100, weight 32768, valid, sourced, best

Once the eBGP peer comes back up, we learn the 192.168.1.0/24 again. Now we can see that the eBGP path is immediately installed in the routing table as the best path.

*Feb 10 00:37:33.259: BGP(0): 172.16.56.6 rcvd UPDATE w/ attr: nexthop 172.16.56.6, origin i, metric 0, path 65000

*Feb 10 00:37:33.259: BGP(0): 172.16.56.6 rcvd 192.168.1.0/24

*Feb 10 00:37:33.271: BGP(0): Revise route installing 1 of 1 routes for 192.168.1.0/24 -> 172.16.56.6(global) to main IP table

*Feb 10 00:37:33.271: RT: updating bgp 192.168.1.0/24 (0x0) via 172.16.56.6

*Feb 10 00:37:33.271: RT: closer admin distance for 192.168.1.0, flushing 1 routes

*Feb 10 00:37:33.271: RT: add 192.168.1.0/24 via 172.16.56.6, bgp metric [20/0]

R1#show ip route 192.168.1.0

Routing entry for 192.168.1.0/24

Known via "bgp 65001", distance 20, metric 0

Tag 65000, type external

Last update from 172.16.56.6 00:00:11 ago

Routing Descriptor Blocks:

* 172.16.56.6, from 172.16.56.6, 00:00:11 ago

Route metric is 0, traffic share count is 1

AS Hops 1

Route tag 65000

R1#show ip bgp 192.168.1.0

BGP routing table entry for 192.168.1.0/24, version 1568

Paths: (1 available, best #1, table default)

Flag: 0x820

Not advertised to any peer

65000

172.16.56.6 from 172.16.56.6 (192.168.8.1)

Origin IGP, metric 0, localpref 100, weight 32769, valid, external, best

If you do not want to apply the weight to all updates received from the neighbor, you can use a route-map to change the weight for only certain updates from the peer. Please see the configuration example below.

router bgp 65001

bgp log-neighbor-changes

neighbor 172.16.56.6 remote-as 65000

!

address-family ipv4

no synchronization

redistribute ospf 1 match internal external 1 external 2

neighbor 172.16.56.6 activate

neighbor 172.16.56.6 route-map set_weight in

no auto-summary

exit-address-family

route-map set_weight permit 10

match ip address 1

set weight 32769

access-list 1 permit 192.168.1.0 0.0.0.255

When the update is received, you can check the ACL for matches.

R1#show access-list 1

Standard IP access list 1

10 permit 192.168.1.0, wildcard bits 0.0.0.255 (2 matches)

Route Process on CSR-1


Cisco Announced that Guinness World Records has certified the Cisco Carrier Routing System (CRS-1) as the highest capacity Internet router ever developed at 92 terabits (92 trillion bits per second) of total throughput, designed to afford up to 100 times more capacity than previously available. The Cisco CRS-1 becomes the first networking technology to be recognized by Guinness World Records, which is the authority on record-breaking achievement around the world.


Cisco's CRS-1, announced in May 2004, is a new class of routing system designed for telecommunications service providers to deliver next-generation data, voice and video services over a converged Internet Protocol (IP) network. This system is ideally suited for large-scale, high-bandwidth applications such as video on demand, online gaming, and real-time interactive services. The IP router, introduced by Cisco in 1984, is the foundation of the public Internet and one of the most important technologies of the past 20 years.

Front View of CRS-1


"Cisco is proud to have Guinness World Records recognize the Cisco CRS-1 as the world's highest capacity Internet router," said Tony Bates, architect of the CRS-1 and vice-president and general manager of Cisco's Carrier Core and Multiservice business unit. "The technological advancement of the CRS-1 will provide the capability for Internet services and applications that have never before been available on such a large scale. Millions of people can now look forward to a new suite of exciting, multimedia services both at work and home."

Cisco's innovative and breakthrough routing system achieves the greatest level of speed and network capacity ever attained. The technological advancement of the CRS-1, for example, could make possible the delivery of high-bandwidth services and applications to entire nations. The CRS-1 with 92 terabits of bandwidth capacity, when augmented by adequate network and transmission capacity, would be capable of the following:

--Download the entire printed collection of the U. S. Library of Congress in 4.6 seconds, as opposed to a dial-up modem transfer rate of 56 thousand bps which would take 82 years

--1 billion people at the same time could play an online game, using real-time voice & chat

--Give 872Kbps broadband connection to every household in the United States (105,480,101 households)

--15 million people could watch quality video on demand simultaneously, which requires 6Mbps (6 million bits per second) per user

--The population of Phoenix, Ariz. (1.3 million) could download a 4 MB song instantaneously. With today's technology, only 9,200 people could download the same song at once

--Approximately 12,415 people could download a 7.4 GB movie in one second

--The entire global population (6.4 billion) could have a simultaneous phone call using Voice-over-IP technology

"As the world advances technologically, it is important that we recognize the innovations that will revolutionize global communications," said David Hawksett, science and technology editor at Guinness World Records. "As soon as we saw the numbers, we were delighted to create a brand new category to acknowledge Cisco Systems and this great achievement in networking technology."

The Cisco CRS-1 complements Cisco's existing portfolio of routers that support the world's largest carrier and enterprise networks. The new routing system is the result of Cisco's proven 20-year track record in routing innovations through focused research and development programs. The Cisco CRS-1 provides carriers with high-performance core routing capabilities, and rounds out Cisco's existing carrier-class routing portfolio, which is ideally suited to deliver cost effective and innovative IP services. For information about the Cisco CRS-1 Carrier Routing System and Cisco's routing strategy, visit: www.cisco.com/go/crs

Availability and Pricing

The Cisco CRS-1 is currently in field trials now with carriers and service providers worldwide and will be available in July 2004. The starting system list price is $450,000 USD.

Editor's Note:

Further press information on the CRS-1 including: in-depth, executive commentary; customer and partner information; videos; interviews with the technologists who developed the Cisco CRS-1; and downloadable, high-resolution product and event photos is available at: http://newsroom.cisco.com/presskit/crs/

About Guinness World Records:

First published in 1955, Guinness World Records(TM) has developed its annual book into an international phenomenon published in more than 100 countries and 20 languages. Guinness World Records has become a household name and the global leader in world records. No other enterprise collects, confirms, accredits and presents world record data with the same investment in comprehensiveness and authenticity. Guinness World Records is the universally recognized authority on record-breaking achievement. For more information, contact Kate White at +44 (0)20 7891 4516 or press@guinnessworldrecords.com.