• The Interconnect Ethernet Segment (I-ES), an Ethernet Segment that can be associated to a set of pseudowires (PWs) or other tunnels. The I-ES defined in this document is not associated with a set of Ethernet links, as per [RFC 7432], but rather with a set of virtual tunnels (e.g., a set of PWs). This set of virtual tunnels is referred to as vES [VIRTUAL-ES].
• The use of the Unknown MAC Route (UMR) in a DCI scenario.
• The processing of EVPN routes on Gateways with MAC-VRFs connecting EVPN-Overlay and EVPN-MPLS networks, or EVPN-Overlay and EVPN-Overlay networks.

AC:

Attachment Circuit

ARP:

Address Resolution Protocol

BUM:

Broadcast, Unknown Unicast and Multicast (traffic)

CE:

Customer Equipment

CFM:

Connectivity Fault Management

DC:

Data Center

DCI:

Data Center Interconnect

DF:

Designated Forwarder

EVI:

EVPN Instance

EVPN:

Ethernet Virtual Private Network, as in [RFC 7432]

EVPN Tunnel binding:

refers to a tunnel to a remote PE/NVE for a given EVI. Ethernet packets in these bindings are encapsulated with the Overlay or MPLS encapsulation and the EVPN label at the bottom of the stack.

ES:

Ethernet Segment

ESI:

Ethernet Segment Identifier

GW:

Gateway or Data Center Gateway

I-ES and I-ESI:

Interconnect Ethernet Segment and Interconnect Ethernet Segment Identifier. An I-ES is defined on the GWs for multihoming to/from the WAN.

MAC

Media Access Control

MAC-VRF:

refers to an EVI instance in a particular node

MP2P and LSM tunnels:

refer to multipoint-to-point and label switched multicast tunnels

ND:

Neighbor Discovery

NDF:

Non-Designated Forwarder

NVE:

Network Virtualization Edge

NVGRE:

Network Virtualization using Generic Routing Encapsulation

NVO:

Network Virtualization Overlay

OAM:

Operations, Administration, and Maintenance

PBB:

Provider Backbone Bridging

PE:

Provider Edge

PW:

Pseudowire

RD:

Route Distinguisher

RR:

Route Reflector

RT:

Route Target

S/C-TAG:

refers to a combination of Service Tag and Customer Tag in a 802.1Q frame

TOR:

Top-Of-Rack

UMR:

Unknown MAC Route

vES:

virtual Ethernet Segment

VNI/VSID:

refers to VXLAN/NVGRE virtual identifiers

VPLS:

Virtual Private LAN Service

VSI:

Virtual Switch Instance or VPLS instance in a particular PE

VXLAN:

Virtual eXtensible LAN

• A simple connectivity handoff between the EVPN-Overlay network provider and the WAN provider so that QoS and security enforcement are easily accomplished.
• Independence of the L2VPN technology deployed in the WAN.
• Multihoming between GW and WAN Edge routers, including per-service load balancing. Per-flow load balancing is not a strong requirement, since a deterministic path per service is usually required for an easy QoS and security enforcement.
• Support of Ethernet OAM and Connectivity Fault Management (CFM) [IEEE.802.1AG] [Y.1731] functions between the GW and the WAN Edge router to detect individual AC failures.
• Support for the following optimizations at the GW:
  • Flooding reduction of unknown unicast traffic sourced from the DC Network Virtualization Edge (NVE) devices.
  • Control of the WAN MAC addresses advertised to the DC.
  • Address Resolution Protocol (ARP) and Neighbor Discovery (ND) flooding control for the requests coming from the WAN.

• If a FEC128-based PW is used between the MAC-VRF (GW) and the VSI (WAN Edge), the corresponding Virtual Connection Identifier (VCID) MUST be provisioned on the MAC-VRF and match the VCID used in the peer VSI at the WAN Edge router.
• If BGP Auto-discovery [RFC 6074] and FEC129-based PWs are used between the GW MAC-VRF and the WAN Edge VSI, the provisioning of the VPLS-ID MUST be supported on the MAC-VRF and MUST match the VPLS-ID used in the WAN Edge VSI.

• If the interconnect solution is based on a VLAN handoff, Ethernet-CFM [IEEE.802.1AG] [Y.1731] may be used to detect individual AC failures on both the GW and WAN Edge router. An individual AC failure will trigger the withdrawal of the corresponding A-D per EVI route as well as the MACs learned on that AC.
• If the interconnect solution is based on a PW handoff, the Label Distribution Protocol (LDP) PW Status bits TLV [RFC 6870] may be used to detect individual PW failures on both the GW and WAN Edge router.

• Control plane and data plane interworking between the EVPN-Overlay network and the L2VPN technology supported in the WAN, irrespective of the technology choice -- i.e., (PBB-)VPLS or (PBB-)EVPN, as depicted in Figure 2.
• Multihoming, including single-active multihoming with per-service load balancing or all-active multihoming -- i.e., per-flow load-balancing -- as long as the technology deployed in the WAN supports it.
• Support for end-to-end MAC Mobility, Static MAC protection and other procedures (e.g., proxy-arp) described in [RFC 7432] as long as EVPN-MPLS is the technology of choice in the WAN.
• Independent inclusive multicast trees in the WAN and in the DC. That is, the inclusive multicast tree type defined in the WAN does not need to be the same as in the DC.

• VPLS PWs will be instantiated in the WAN split-horizon group.
• Overlay tunnel bindings (e.g., VXLAN, NVGRE) will be instantiated in the DC split-horizon group.

• Attempts to set up a PW between the two GWs within the B-component context will never be blocked.
• If a PW exists between two GWs for the B-component and an attempt is made to set up an EVPN binding on an I-component linked to that B-component, the EVPN binding will be kept down operationally. Note that the BGP EVPN routes will still be valid but not used.
• The EVPN binding will only be up and used as long as there is no PW to the same far end in the corresponding B-component. The EVPN bindings in the I-components will be brought down before the PW in the B-component is brought up.

• Ethernet A-D routes, ES routes, and Inclusive Multicast routes are consumed by the GWs and processed locally for the corresponding [RFC 7432] procedures.
• MAC/IP advertisement routes will be received and imported, and if they become active in the MAC-VRF, the information will be re-advertised as new routes with the following fields:
  • The RD will be the GW's RD for the MAC-VRF.
  • The ESI will be set to the I-ESI.
  • The Ethernet-tag value will be kept from the received NLRI the received NLRI.
  • The MAC length, MAC address, IP Length, and IP address values will be kept from the received NLRI.
  • The MPLS label will be a local 20-bit value (when sent to the WAN) or a DC-global 24-bit value (when sent to the DC for encapsulations using a VNI).
  • The appropriate Route Targets (RTs) and [RFC 9012] BGP encapsulation extended community will be used according to [RFC 8365].

• ES route(s) for the I-ESI(s).
• Ethernet A-D routes per ES and EVI for the I-ESI(s). The A-D per-EVI routes sent to the WAN and the DC will have consistent Ethernet-Tag values.
• Inclusive Multicast routes with independent tunnel-type value for the WAN and DC. For example, a P2MP Label Switched Path (LSP) may be used in the WAN, whereas ingress replication may be used in the DC. The routes sent to the WAN and the DC will have a consistent Ethernet-Tag.
• MAC/IP advertisement routes for MAC addresses learned in local attachment circuits. Note that these routes will not include the I-ESI value in the ESI field. These routes will include a zero ESI or a non-zero ESI for local multihomed Ethernet Segments (ES). The routes sent to the WAN and the DC will have a consistent Ethernet-Tag.

• Inclusive multicast routes: When setting up the flooding lists for a given MAC-VRF, each GW will include its DC peer GW only in the EVPN-MPLS flooding list (by default) and not the EVPN-Overlay flooding list. That is, GW2 will import two Inclusive Multicast routes from GW1 (from set-DC and set-WAN) but will only consider one of the two, giving the set-WAN route higher priority. An administrative option MAY change this preference so that the set-DC route is selected first.
• MAC/IP advertisement routes for local attachment circuits: As above, the GW will select only one, giving the route from the set-WAN a higher priority. As with the Inclusive multicast routes, an administrative option MAY change this priority.

• EVPN-Overlay flooding list (composed of bindings to the remote NVEs or multicast tunnel to the NVEs).
• EVPN-MPLS flooding list (composed of MP2P or LSM tunnel to the remote PEs).

• Single-active multihoming; assuming a WAN split-horizon group (comprised of EVPN-MPLS bindings), a DC split-horizon group (comprised of EVPN-Overlay bindings), and local ACs on the GWs:
  • Forwarding behavior on the non-DF: The non-DF MUST block ingress and egress forwarding on the EVPN-Overlay bindings associated to the I-ES. The EVPN-MPLS network is considered to be the core network, and the EVPN-MPLS bindings to the remote PEs and GWs will be active.
  • Forwarding behavior on the DF: The DF MUST NOT forward BUM or unicast traffic received from a given split-horizon group to a member of its own split-horizon group. Forwarding to other split-horizon groups and local ACs is allowed (as long as the ACs are not part of an ES for which the node is non-DF). As per [RFC 7432] and for split-horizon purposes, when receiving BUM traffic on the EVPN-Overlay bindings associated to an I-ES, the DF GW SHOULD add the I-ESI label when forwarding to the peer GW over EVPN-MPLS.
  • When receiving EVPN MAC/IP routes from the WAN, the non-DF MUST NOT reoriginate the EVPN routes and advertise them to the DC peers. In the same way, EVPN MAC/IP routes received from the DC MUST NOT be advertised to the WAN peers. This is consistent with [RFC 7432] and allows the remote PE/NVEs to know who the primary GW is, based on the reception of the MAC/IP routes.

• All-active multihoming; assuming a WAN split-horizon group (comprised of EVPN-MPLS bindings), a DC split-horizon group (comprised of EVPN-Overlay bindings), and local ACs on the GWs:
  • Forwarding behavior on the non-DF: The non-DF follows the same behavior as the non-DF in the single-active case, but only for BUM traffic. Unicast traffic received from a split-horizon group MUST NOT be forwarded to a member of its own split-horizon group but can be forwarded normally to the other split-horizon groups and local ACs. If a known unicast packet is identified as a "flooded" packet, the procedures for BUM traffic MUST be followed.
  • Forwarding behavior on the DF: The DF follows the same behavior as the DF in the single-active case, but only for BUM traffic. Unicast traffic received from a split-horizon group MUST NOT be forwarded to a member of its own split-horizon group but can be forwarded normally to the other split-horizon group and local ACs. If a known unicast packet is identified as a "flooded" packet, the procedures for BUM traffic MUST be followed. As per [RFC 7432] and for split-horizon purposes, when receiving BUM traffic on the EVPN-Overlay bindings associated to an I-ES, the DF GW MUST add the I-ESI label when forwarding to the peer GW over EVPN-MPLS.
  • Contrary to the single-active multihoming case, both DF and non-DF reoriginate and advertise MAC/IP routes received from the WAN/DC peers, adding the corresponding I-ESI so that the remote PE/NVEs can perform regular aliasing, as per [RFC 7432].

     |<--EVPN-Overlay--->|<--EVPN-MPLS-->|

             +---------+ +--------------+
      +----+ BUM       +---+             |
      |NVE1+----+----> |   +-+-----+     |
      +----+  | |   DF |GW1| |     |     |
              | |      +-+-+ |     |    ++--+
              | |        |   |     +--> |PE1|
              | +--->X +-+-+ |          ++--+
              |     NDF|   | |           |
      +----+  |        |GW2<-+           |
      |NVE2+--+        +-+-+             |
      +----+  +--------+ |  +------------+
                         v
                       +--+
                       |CE|
                       +--+

(a)

MAC Address M1 is advertised from NVE3 in EVI-1.

(b)

GW3 and GW4 learn M1 for EVI-1 and re-advertise M1 to the WAN with I-ESI-2 in the ESI field.

(c)

GW1 and GW2 learn M1 and install GW3/GW4 as next hops following the EVPN aliasing procedures.

(d)

Before NVE1 learns M1, a packet arrives at NVE1 with destination M1. If the Unknown MAC Route had not been advertised into the DC, NVE1 would have flooded the packet throughout the DC, in particular to both GW1 and GW2. If the same VNI/VSID is used for both known unicast and BUM traffic, as is typically the case, there is no indication in the packet that it is a BUM packet, and both GW1 and GW2 would have forwarded it, creating packet duplication. However, because the Unknown MAC Route had been advertised into the DC, NVE1 will unicast the packet to either GW1 or GW2.

(e)

Since both GW1 and GW2 know M1, the GW receiving the packet will forward it to either GW3 or GW4.

• As in any of the described GW models, this solution supports the connectivity of local attachment circuits on the GWs. This is not possible in a "DCI using ASBRs" solution.
• Different data plane encapsulations can be supported in the DC and the WAN, while a uniform encapsulation is needed in the "DCI using ASBRs" solution.
• Optimized multicast solution, with independent inclusive multicast trees in DC and WAN.
• MPLS label aggregation: For the case where MPLS labels are signaled from the NVEs for MAC/IP advertisement routes, this solution provides label aggregation. A remote PE MAY receive a single label per GW MAC-VRF, as opposed to a label per NVE/MAC-VRF connected to the GW MAC-VRF. For instance, in Figure 2, PE would receive only one label for all the routes advertised for a given MAC-VRF from GW1, as opposed to a label per NVE/MAC-VRF.
• The GW will not propagate MAC Mobility for the MACs moving within a DC. Mobility intra-DC is solved by all the NVEs in the DC. The MAC Mobility procedures on the GWs are only required in case of mobility across DCs.
• Proxy-ARP/ND function on the DC GWs can be leveraged to reduce ARP/ND flooding in the DC or/and the WAN.

1. Globally unique VNIs in the interconnect network. In this case, the GWs and PEs in the interconnect network will agree on a common VNI for a given EVI. The RT to be used in the interconnect network can be autoderived from the agreed-upon interconnect VNI. The VNI used inside each DC MAY be the same as the interconnect VNI.
2. Downstream-assigned VNIs in the interconnect network. In this case, the GWs and PEs MUST use the proper RTs to import/export the EVPN routes. Note that even if the VNI is downstream assigned in the interconnect network, and unlike option (a), it only identifies the <Ethernet Tag, GW> pair and not the <Ethernet Tag, egress PE> pair. The VNI used inside each DC MAY be the same as the interconnect VNI. GWs SHOULD support multiple VNI spaces per EVI (one per interconnect network they are connected to).

[RFC2119]

S. Bradner, "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.

[RFC4761]

K. Kompella, and Y. Rekhter, "Virtual Private LAN Service (VPLS) Using BGP for Auto-Discovery and Signaling", RFC 4761, DOI 10.17487/RFC4761, January 2007,
<https://www.rfc-editor.org/info/rfc4761>.

[RFC4762]

M. Lasserre, and V. Kompella, "Virtual Private LAN Service (VPLS) Using Label Distribution Protocol (LDP) Signaling", RFC 4762, DOI 10.17487/RFC4762, January 2007,
<https://www.rfc-editor.org/info/rfc4762>.

[RFC6074]

E. Rosen, B. Davie, V. Radoaca, and W. Luo, "Provisioning, Auto-Discovery, and Signaling in Layer 2 Virtual Private Networks (L2VPNs)", RFC 6074, DOI 10.17487/RFC6074, January 2011,
<https://www.rfc-editor.org/info/rfc6074>.

[RFC7041]

F. Balus, A. Sajassi, and N. Bitar, "Extensions to the Virtual Private LAN Service (VPLS) Provider Edge (PE) Model for Provider Backbone Bridging", RFC 7041, DOI 10.17487/RFC7041, November 2013,
<https://www.rfc-editor.org/info/rfc7041>.

[RFC7432]

A. Sajassi, R. Aggarwal, N. Bitar, A. Isaac, J. Uttaro, J. Drake, and W. Henderickx, "BGP MPLS-Based Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 2015,
<https://www.rfc-editor.org/info/rfc7432>.

[RFC7543]

H. Jeng, L. Jalil, R. Bonica, K. Patel, and L. Yong, "Covering Prefixes Outbound Route Filter for BGP-4", RFC 7543, DOI 10.17487/RFC7543, May 2015,
<https://www.rfc-editor.org/info/rfc7543>.

[RFC7623]

A. Sajassi, S. Salam, N. Bitar, A. Isaac, and W. Henderickx, "Provider Backbone Bridging Combined with Ethernet VPN (PBB-EVPN)", RFC 7623, DOI 10.17487/RFC7623, September 2015,
<https://www.rfc-editor.org/info/rfc7623>.

[RFC8174]

B. Leiba, "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017,
<https://www.rfc-editor.org/info/rfc8174>.

[RFC8365]

A. Sajassi, J. Drake, N. Bitar, R. Shekhar, J. Uttaro, and W. Henderickx, "A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, DOI 10.17487/RFC8365, March 2018,
<https://www.rfc-editor.org/info/rfc8365>.

[RFC9012]

K. Patel, G. Van de Velde, S. Sangli, and J. Scudder, "The BGP Tunnel Encapsulation Attribute", RFC 9012, DOI 10.17487/RFC9012, April 2021,
<https://www.rfc-editor.org/info/rfc9012>.

[IEEE.802.1AG]

IEEE, "IEEE Standard for Local and Metropolitan Area Networks Virtual Bridged Local Area Networks Amendment 5: Connectivity Fault Management", IEEE standard 802.1ag-2007, January 2008.

[IEEE.802.1Q]

IEEE, "IEEE Standard for Local and metropolitan area networks--Bridges and Bridged Networks", IEEE standard 802.1Q-2014, DOI 10.1109/IEEESTD.2014.6991462, December 2014,
<https://doi.org/10.1109/IEEESTD.2014.6991462>.

[RFC3031]

E. Rosen, A. Viswanathan, and R. Callon, "Multiprotocol Label Switching Architecture", RFC 3031, DOI 10.17487/RFC3031, January 2001,
<https://www.rfc-editor.org/info/rfc3031>.

[RFC4023]

T. Worster, Y. Rekhter, and E. Rosen, "Encapsulating MPLS in IP or Generic Routing Encapsulation (GRE)", RFC 4023, DOI 10.17487/RFC4023, March 2005,
<https://www.rfc-editor.org/info/rfc4023>.

[RFC4684]

P. Marques, R. Bonica, L. Fang, L. Martini, R. Raszuk, K. Patel, and J. Guichard, "Constrained Route Distribution for Border Gateway Protocol/MultiProtocol Label Switching (BGP/MPLS) Internet Protocol (IP) Virtual Private Networks (VPNs)", RFC 4684, DOI 10.17487/RFC4684, November 2006,
<https://www.rfc-editor.org/info/rfc4684>.

[RFC6870]

P. Muley, and M. Aissaoui, "Pseudowire Preferential Forwarding Status Bit", RFC 6870, DOI 10.17487/RFC6870, February 2013,
<https://www.rfc-editor.org/info/rfc6870>.

[RFC7348]

M. Mahalingam, D. Dutt, K. Duda, P. Agarwal, L. Kreeger, T. Sridhar, M. Bursell, and C. Wright, "Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
<https://www.rfc-editor.org/info/rfc7348>.

[RFC7637]

P. Garg, and Y. Wang, "NVGRE: Network Virtualization Using Generic Routing Encapsulation", RFC 7637, DOI 10.17487/RFC7637, September 2015,
<https://www.rfc-editor.org/info/rfc7637>.

[VIRTUAL-ES]

Nokia, , , , , and , "EVPN Virtual Ethernet Segment", Internet-Draft draft-ietf-bess-evpn-virtual-eth-segment-06, March 2020,
<https://tools.ietf.org/html/draft-ietf-bess-evpn-virtual-eth-segment-06>.

[Y.1731]

ITU-T, "OAM functions and mechanisms for Ethernet based networks", ITU-T Recommendation Y.1731, August 2019.

Ravi Shekhar

Anil Lohiya

Wen Lin

Florin Balus

Patrice Brissette

Senad Palislamovic

Dennis Cai

Jorge Rabadan

Senthil Sathappan

Wim Henderickx

Ali Sajassi

John Drake