AC:

Attachment Circuit

ARP:

Address Resolution Protocol

BD:

Broadcast Domain. As per [RFC 7432], an EVI consists of a single BD or multiple BDs. In case of VLAN-bundle and VLAN-based service models (see [RFC 7432]), a BD is equivalent to an EVI. In case of a VLAN-aware bundle service model, an EVI contains multiple BDs. Also, in this document, "BD" and "subnet" are equivalent terms.

BD Route Target:

Refers to the broadcast-domain-assigned Route Target [RFC 4364]. In case of a VLAN-aware bundle service model, all the BD instances in the MAC-VRF share the same Route Target.

BT:

Bridge Table. The instantiation of a BD in a MAC-VRF, as per [RFC 7432].

CE:

Customer Edge

DA:

Destination Address

DGW:

Data Center Gateway

Ethernet A-D Route:

Ethernet Auto-Discovery (A-D) route, as per [RFC 7432].

Ethernet NVO Tunnel:

Refers to Network Virtualization Overlay tunnels with Ethernet payload. Examples of this type of tunnel are VXLAN or GENEVE.

EVI:

EVPN Instance spanning the NVE/PE devices that are participating on that EVPN, as per [RFC 7432].

EVPN:

Ethernet VPN, as per [RFC 7432].

GENEVE:

Generic Network Virtualization Encapsulation, as per [RFC 8926].

GRE:

Generic Routing Encapsulation

GW IP:

Gateway IP address

IPL:

IP Prefix Length

IP NVO Tunnel:

Refers to Network Virtualization Overlay tunnels with IP payload (no MAC header in the payload).

IP-VRF:

A Virtual Routing and Forwarding table for IP routes on an NVE/PE. The IP routes could be populated by EVPN and IP-VPN address families. An IP-VRF is also an instantiation of a Layer 3 VPN in an NVE/PE.

IRB:

Integrated Routing and Bridging interface. It connects an IP-VRF to a BD (or subnet).

MAC:

Media Access Control

MAC-VRF:

A Virtual Routing and Forwarding table for MAC addresses on an NVE/PE, as per [RFC 7432]. A MAC-VRF is also an instantiation of an EVI in an NVE/PE.

ML:

MAC Address Length

ND:

Neighbor Discovery

NVE:

Network Virtualization Edge

NVO:

Network Virtualization Overlay

PE:

Provider Edge

RT-2:

EVPN Route Type 2, i.e., MAC/IP Advertisement route, as defined in [RFC 7432].

RT-5:

EVPN Route Type 5, i.e., IP Prefix route, as defined in Section 3.

SBD:

Supplementary Broadcast Domain. A BD that does not have any ACs, only IRB interfaces, and is used to provide connectivity among all the IP-VRFs of the tenant. The SBD is only required in IP-VRF-to-IP-VRF use cases (see Section 4.4).

SN:

Subnet

TS:

Tenant System

VA:

Virtual Appliance

VM:

Virtual Machine

VNI:

Virtual Network Identifier. As in [RFC 8365], the term is used as a representation of a 24-bit NVO instance identifier, with the understanding that "VNI" will refer to a VXLAN Network Identifier in VXLAN, or a Virtual Network Identifier in GENEVE, etc., unless it is stated otherwise.

VSID:

Virtual Subnet Identifier

VTEP:

VXLAN Termination End Point, as per [RFC 7348].

VXLAN:

Virtual eXtensible Local Area Network, as per [RFC 7348].

• The Tenant Systems may be VMs that generate traffic from their own MAC and IP.
• The Tenant Systems may be VA entities that forward traffic to/from IP addresses of different end devices sitting behind them.
  • These VAs can be firewalls, load balancers, NAT devices, other appliances, or virtual gateways with virtual routing instances.
  • These VAs do not necessarily participate in dynamic routing protocols and hence rely on the EVPN NVEs to advertise the routes on their behalf.
  • In all these cases, the VA will forward traffic to other TSs using its own source MAC, but the source IP will be the one associated with the end device sitting behind the VA or a translated IP address (part of a public NAT pool) if the VA is performing NAT.
  • Note that the same IP address and endpoint could exist behind two of these TSs. One example of this would be certain appliance resiliency mechanisms, where a virtual IP or floating IP can be owned by one of the two VAs running the resiliency protocol (the Master VA). The Virtual Router Redundancy Protocol (VRRP) [RFC 5798] is one particular example of this. Another example is multihomed subnets, i.e., the same subnet is connected to two VAs.
  • Although these VAs provide IP connectivity to VMs and the subnets behind them, they do not always have their own IP interface connected to the EVPN NVE; Layer 2 firewalls are examples of VAs not supporting IP interfaces.

                    NVE1
                 +-----------+
        TS1(VM)--|  (BD-10)  |-----+
          M1/IP1 +-----------+     |               DGW1
                               +---------+    +-------------+
                               |         |----|  (BD-10)    |
  SN1---+           NVE2       |         |    |    IRB1\    |
        |        +-----------+ |         |    |     (IP-VRF)|---+
  SN2---TS2(VA)--|  (BD-10)  |-|         |    +-------------+  _|_
        | M2/IP2 +-----------+ |  VXLAN/ |                    (   )
  IP4---+  <-+                 |  GENEVE |         DGW2      ( WAN )
             |                 |         |    +-------------+ (___)
          vIP23 (floating)     |         |----|  (BD-10)    |   |
             |                 +---------+    |    IRB2\    |   |
  SN1---+  <-+      NVE3         |  |  |      |     (IP-VRF)|---+
        | M3/IP3 +-----------+   |  |  |      +-------------+
  SN3---TS3(VA)--|  (BD-10)  |---+  |  |
        |        +-----------+      |  |
  IP5---+                           |  |
                                    |  |
                 NVE4               |  |      NVE5            +--SN5
           +---------------------+  |  | +-----------+        |
  IP6------|  (BD-1)             |  |  +-|  (BD-10)  |--TS4(VA)--SN6
           |       \             |  |    +-----------+        |
           |    (IP-VRF)         |--+                ESI4     +--SN7
           |       /  \IRB3      |
       |---|  (BD-2)  (BD-10)    |
    SN4|   +---------------------+


  Note: 
  ESI4 = Ethernet Segment Identifier 4

• TS1 is a VM that generates/receives traffic to/from IP1, where IP1 belongs to the BD-10 subnet.
• TS2 and TS3 are VAs that send/receive traffic to/from the subnets and hosts sitting behind them (SN1, SN2, SN3, IP4, and IP5). Their IP addresses (IP2 and IP3) belong to the BD-10 subnet, and they can also generate/receive traffic. When these VAs receive packets destined to their own MAC addresses (M2 and M3), they will route the packets to the proper subnet or host. These VAs do not support routing protocols to advertise the subnets connected to them and can move to a different server and NVE when the cloud management system decides to do so. These VAs may also support redundancy mechanisms for some subnets, similar to VRRP, where a floating IP is owned by the Master VA and only the Master VA forwards traffic to a given subnet. For example, vIP23 in Figure 1 is a floating IP that can be owned by TS2 or TS3 depending on which system is the Master. Only the Master will forward traffic to SN1.
• Integrated Routing and Bridging interfaces IRB1, IRB2, and IRB3 have their own IP addresses that belong to the BD-10 subnet too. These IRB interfaces connect the BD-10 subnet to Virtual Routing and Forwarding (IP-VRF) instances that can route the traffic to other subnets for the same tenant (within the DC or at the other end of the WAN).
• TS4 is a Layer 2 VA that provides connectivity to subnets SN5, SN6, and SN7 but does not have an IP address itself in the BD-10. TS4 is connected to a port on NVE5 that is assigned to Ethernet Segment Identifier 4 (ESI4).

a)

must be advertised from the NVEs (since VAs and VMs do not participate in dynamic routing protocols) and

b)

may be associated with an Overlay Index that can be a VA IP address, a floating IP address, a MAC address, or an ESI. The Overlay Index is further discussed in Section 3.2.

a)

The clean and clear advertisements of IPv4 or IPv6 prefixes in a Network Layer Reachability Information (NLRI) message without MAC addresses are allowed.

b)

Since the route type is different from the MAC/IP Advertisement route, the current procedures described in [RFC 7432] do not need to be modified.

c)

A flexible implementation is allowed where the prefix can be linked to different types of Overlay/Underlay Indexes: overlay IP addresses, overlay MAC addresses, overlay ESIs, underlay BGP next hops, etc.

d)

An EVPN implementation not requiring IP prefixes can simply discard them by looking at the route type value.

    +-----------------------------------+
    |    Route Type (1 octet)           |
    +-----------------------------------+
    |     Length (1 octet)              |
    +-----------------------------------+
    | Route Type specific (variable)    |
    +-----------------------------------+

• Value:

  5

  Description:

  IP Prefix

    +---------------------------------------+
    |      RD (8 octets)                    |
    +---------------------------------------+
    |Ethernet Segment Identifier (10 octets)|
    +---------------------------------------+
    |  Ethernet Tag ID (4 octets)           |
    +---------------------------------------+
    |  IP Prefix Length (1 octet, 0 to 32)  |
    +---------------------------------------+
    |  IP Prefix (4 octets)                 |
    +---------------------------------------+
    |  GW IP Address (4 octets)             |
    +---------------------------------------+
    |  MPLS Label (3 octets)                |
    +---------------------------------------+

    +---------------------------------------+
    |      RD (8 octets)                    |
    +---------------------------------------+
    |Ethernet Segment Identifier (10 octets)|
    +---------------------------------------+
    |  Ethernet Tag ID (4 octets)           |
    +---------------------------------------+
    |  IP Prefix Length (1 octet, 0 to 128) |
    +---------------------------------------+
    |  IP Prefix (16 octets)                |
    +---------------------------------------+
    |  GW IP Address (16 octets)            |
    +---------------------------------------+
    |  MPLS Label (3 octets)                |
    +---------------------------------------+

• The Length field of the BGP EVPN NLRI for an EVPN IP Prefix route MUST be either 34 (if IPv4 addresses are carried) or 58 (if IPv6 addresses are carried). The IP prefix and gateway IP address MUST be from the same IP address family.
• The Route Distinguisher (RD) and Ethernet Tag ID MUST be used as defined in [RFC 7432] and [RFC 8365]. In particular, the RD is unique per MAC-VRF (or IP-VRF). The MPLS Label field is set to either an MPLS label or a VNI, as described in [RFC 8365] for other EVPN route types.
• The Ethernet Segment Identifier MUST be a non-zero 10-octet identifier if the ESI is used as an Overlay Index (see the definition of "Overlay Index" in Section 3.2). It MUST be all bytes zero otherwise. The ESI format is described in [RFC 7432].
• The IP prefix length can be set to a value between 0 and 32 (bits) for IPv4 and between 0 and 128 for IPv6, and it specifies the number of bits in the prefix. The value MUST NOT be greater than 128.
• The IP prefix is a 4- or 16-octet field (IPv4 or IPv6).
• The GW IP Address field is a 4- or 16-octet field (IPv4 or IPv6) and will encode a valid IP address as an Overlay Index for the IP prefixes. The GW IP field MUST be all bytes zero if it is not used as an Overlay Index. Refer to Section 3.2 for the definition and use of the Overlay Index.
• The MPLS Label field is encoded as 3 octets, where the high-order 20 bits contain the label value, as per [RFC 7432]. When sending, the label value SHOULD be zero if a recursive resolution based on an Overlay Index is used. If the received MPLS label value is zero, the route MUST contain an Overlay Index, and the ingress NVE/PE MUST perform a recursive resolution to find the egress NVE/PE. If the received label is zero and the route does not contain an Overlay Index, it MUST be "treat as withdraw" [RFC 7606].

• An Overlay Index can be an ESI or IP address in the address space of the tenant or MAC address, and it is used by an NVE as the next hop for a given IP prefix. An Overlay Index always needs a recursive route resolution on the NVE/PE that installs the RT-5 into one of its IP-VRFs so that the NVE knows to which egress NVE/PE it needs to forward the packets. It is important to note that recursive resolution of the Overlay Index applies upon installation into an IP-VRF and not upon BGP propagation (for instance, on an ASBR). Also, as a result of the recursive resolution, the egress NVE/PE is not necessarily the same NVE that originated the RT-5.
• The Overlay Index is indicated along with the RT-5 in the ESI field, GW IP field, or EVPN Router's MAC Extended Community, depending on whether the IP prefix next hop is an ESI, an IP address, or a MAC address in the tenant space. The Overlay Index for a given IP prefix is set by local policy at the NVE that originates an RT-5 for that IP prefix (typically managed by the cloud management system).
• In order to enable the recursive lookup resolution at the ingress NVE, an NVE that is a possible egress NVE for a given Overlay Index must originate a route advertising itself as the BGP next hop on the path to the system denoted by the Overlay Index. For instance:
  • If an NVE receives an RT-5 that specifies an Overlay Index, the NVE cannot use the RT-5 in its IP-VRF unless (or until) it can recursively resolve the Overlay Index.
  • If the RT-5 specifies an ESI as the Overlay Index, a recursive resolution can only be done if the NVE has received and installed an RT-1 (auto-discovery per EVI) route specifying that ESI.
  • If the RT-5 specifies a GW IP address as the Overlay Index, a recursive resolution can only be done if the NVE has received and installed an RT-2 (MAC/IP Advertisement route) specifying that IP address in the IP Address field of its NLRI.
  • If the RT-5 specifies a MAC address as the Overlay Index, a recursive resolution can only be done if the NVE has received and installed an RT-2 (MAC/IP Advertisement route) specifying that MAC address in the MAC Address field of its NLRI.
  Note that the RT-1 or RT-2 routes needed for the recursive resolution may arrive before or after the given RT-5 route.
• Irrespective of the recursive resolution, if there is no IGP or BGP route to the BGP next hop of an RT-5, BGP MUST NOT install the RT-5 even if the Overlay Index can be resolved.
• The ESI and GW IP fields may both be zero at the same time. However, they MUST NOT both be non-zero at the same time. A route containing a non-zero GW IP and a non-zero ESI (at the same time) SHOULD be treat as withdraw [RFC 7606].
• If either the ESI or the GW IP are non-zero, then the non-zero one is the Overlay Index, regardless of whether the EVPN Router's MAC Extended Community is present or the value of the label. In case the GW IP is the Overlay Index (hence, ESI is zero), the EVPN Router's MAC Extended Community is ignored if present.
• A route where ESI, GW IP, MAC, and Label are all zero at the same time SHOULD be treat as withdraw.

*

MAC with "Zero" value means no EVPN Router's MAC Extended Community is present along with the RT-5. "Non-Zero" indicates that the extended community is present and carries a valid MAC address. The encoding of a MAC address MUST be the 6-octet MAC address specified by [IEEE-802.1Q]. Examples of invalid MAC addresses are broadcast or multicast MAC addresses. The route MUST be treat as withdraw in case of an invalid MAC address. The presence of the EVPN Router's MAC Extended Community alone is not enough to indicate the use of the MAC address as the Overlay Index since the extended community can be used for other purposes.

**

In this case, the Overlay Index may be the RT-5's MAC address or "None", depending on the local policy of the receiving NVE/PE. Note that the advertising NVE/PE that sets the Overlay Index SHOULD advertise an RT-2 for the MAC Overlay Index if there are receiving NVE/PEs configured to use the MAC as the Overlay Index. This case in Table 1 is used in the IP-VRF-to-IP-VRF implementations described in Sections [4.4.1] and [4.4.3]. The support of a MAC Overlay Index in this model is OPTIONAL.

***

The Overlay Index is "None". This is a special case used for IP-VRF-to-IP-VRF where the NVE/PEs are connected by IP NVO tunnels as opposed to Ethernet NVO tunnels.

IP4---+           NVE2                            DGW1
      |        +-----------+ +---------+    +-------------+
SN2---TS2(VA)--|  (BD-10)  |-|         |----|  (BD-10)    |
      | M2/IP2 +-----------+ |         |    |    IRB1\    |
 -+---+                      |         |    |     (IP-VRF)|---+
  |                          |         |    +-------------+  _|_
 SN1                         |  VXLAN/ |                    (   )
  |                          |  GENEVE |         DGW2      ( WAN )
 -+---+           NVE3       |         |    +-------------+ (___)
      | M3/IP3 +-----------+ |         |----|  (BD-10)    |   |
SN3---TS3(VA)--|  (BD-10)  |-|         |    |    IRB2\    |   |
      |        +-----------+ +---------+    |     (IP-VRF)|---+
IP5---+                                     +-------------+

(1)

NVE2 advertises the following BGP routes on behalf of TS2:

• Route type 2 (MAC/IP Advertisement route) containing: ML = 48 (MAC address length), M = M2 (MAC address), IPL = 32 (IP prefix length), IP = IP2, and BGP Encapsulation Extended Community [RFC 9012] with the corresponding tunnel type. The MAC and IP addresses may be learned via ARP snooping.
• Route type 5 (IP Prefix route) containing: IPL = 24, IP = SN1, ESI = 0, and GW IP address = IP2. The prefix and GW IP are learned by policy.

(2)

Similarly, NVE3 advertises the following BGP routes on behalf of TS3:

• Route type 2 (MAC/IP Advertisement route) containing: ML = 48, M = M3, IPL = 32, IP = IP3 (and BGP Encapsulation Extended Community).
• Route type 5 (IP Prefix route) containing: IPL = 24, IP = SN1, ESI = 0, and GW IP address = IP3.

(3)

DGW1 and DGW2 import both received routes based on the Route Targets:

• Based on the BD-10 Route Target in DGW1 and DGW2, the MAC/IP Advertisement route is imported, and M2 is added to the BD-10 along with its corresponding tunnel information. For instance, if VXLAN is used, the VTEP will be derived from the MAC/IP Advertisement route BGP next hop and VNI from the MPLS Label1 field. M2/IP2 is added to the ARP table. Similarly, M3 is added to BD-10, and M3/IP3 is added to the ARP table.
• Based on the BD-10 Route Target in DGW1 and DGW2, the IP Prefix route is also imported, and SN1/24 is added to the IP-VRF with Overlay Index IP2 pointing at the local BD-10. In this example, it is assumed that the RT-5 from NVE2 is preferred over the RT-5 from NVE3. If both routes were equally preferable and ECMP enabled, SN1/24 would also be added to the routing table with Overlay Index IP3.

(4)

When DGW1 receives a packet from the WAN with destination IPx, where IPx belongs to SN1/24:

• A destination IP lookup is performed on the DGW1 IP-VRF table, and Overlay Index = IP2 is found. Since IP2 is an Overlay Index, a recursive route resolution is required for IP2.
• IP2 is resolved to M2 in the ARP table, and M2 is resolved to the tunnel information given by the BD FIB (e.g., remote VTEP and VNI for the VXLAN case).
• The IP packet destined to IPx is encapsulated with:
  • Inner source MAC = IRB1 MAC.
  • Inner destination MAC = M2.
  • Tunnel information provided by the BD (VNI, VTEP IPs, and MACs for the VXLAN case).

(5)

When the packet arrives at NVE2:

• Based on the tunnel information (VNI for the VXLAN case), the BD-10 context is identified for a MAC lookup.
• Encapsulation is stripped off and, based on a MAC lookup (assuming MAC forwarding on the egress NVE), the packet is forwarded to TS2, where it will be properly routed.

(6)

Should TS2 move from NVE2 to NVE3, MAC Mobility procedures will be applied to the MAC route M2/IP2, as defined in [RFC 7432]. Route type 5 prefixes are not subject to MAC Mobility procedures; hence, no changes in the DGW IP-VRF table will occur for TS2 mobility -- i.e., all the prefixes will still be pointing at IP2 as the Overlay Index. There is an indirection for, e.g., SN1/24, which still points at Overlay Index IP2 in the routing table, but IP2 will be simply resolved to a different tunnel based on the outcome of the MAC Mobility procedures for the MAC/IP Advertisement route M2/IP2.

                 NVE2                           DGW1
              +-----------+ +---------+    +-------------+
 +---TS2(VA)--|  (BD-10)  |-|         |----|  (BD-10)    |
 |     M2/IP2 +-----------+ |         |    |    IRB1\    |
 |      <-+                 |         |    |     (IP-VRF)|---+
 |        |                 |         |    +-------------+  _|_
SN1    vIP23 (floating)     |  VXLAN/ |                    (   )
 |        |                 |  GENEVE |         DGW2      ( WAN )
 |      <-+      NVE3       |         |    +-------------+ (___)
 |     M3/IP3 +-----------+ |         |----|  (BD-10)    |   |
 +---TS3(VA)--|  (BD-10)  |-|         |    |    IRB2\    |   |
              +-----------+ +---------+    |     (IP-VRF)|---+
                                           +-------------+

(1)

NVE2 advertises the following BGP routes for TS2:

• Route type 2 (MAC/IP Advertisement route) containing: ML = 48, M = M2, IPL = 32, and IP = vIP23 (as well as BGP Encapsulation Extended Community). The MAC and IP addresses may be learned via ARP snooping.
• Route type 5 (IP Prefix route) containing: IPL = 24, IP = SN1, ESI = 0, and GW IP address = vIP23. The prefix and GW IP are learned by policy.

(2)

NVE3 advertises the following BGP route for TS3 (it does not advertise an RT-2 for M3/vIP23):

• Route type 5 (IP Prefix route) containing: IPL = 24, IP = SN1, ESI = 0, and GW IP address = vIP23. The prefix and GW IP are learned by policy.

(3)

DGW1 and DGW2 import both received routes based on the Route Target:

• M2 is added to the BD-10 FIB along with its corresponding tunnel information. For the VXLAN use case, the VTEP will be derived from the MAC/IP Advertisement route BGP next hop and VNI from the VNI field. M2/vIP23 is added to the ARP table.
• SN1/24 is added to the IP-VRF in DGW1 and DGW2 with Overlay Index vIP23 pointing at M2 in the local BD-10.

(4)

When DGW1 receives a packet from the WAN with destination IPx, where IPx belongs to SN1/24:

• A destination IP lookup is performed on the DGW1 IP-VRF table, and Overlay Index = vIP23 is found. Since vIP23 is an Overlay Index, a recursive route resolution for vIP23 is required.
• vIP23 is resolved to M2 in the ARP table, and M2 is resolved to the tunnel information given by the BD (remote VTEP and VNI for the VXLAN case).
• The IP packet destined to IPx is encapsulated with:
  • Inner source MAC = IRB1 MAC.
  • Inner destination MAC = M2.
  • Tunnel information provided by the BD FIB (VNI, VTEP IPs, and MACs for the VXLAN case).

(5)

When the packet arrives at NVE2:

• Based on the tunnel information (VNI for the VXLAN case), the BD-10 context is identified for a MAC lookup.
• Encapsulation is stripped off and, based on a MAC lookup (assuming MAC forwarding on the egress NVE), the packet is forwarded to TS2, where it will be properly routed.

(6)

When the redundancy protocol running between TS2 and TS3 appoints TS3 as the new active TS for SN1, TS3 will now own the floating vIP23 and will signal this new ownership using a gratuitous ARP REPLY message (explained in [RFC 5227]) or similar. Upon receiving the new owner's notification, NVE3 will issue a route type 2 for M3/vIP23, and NVE2 will withdraw the RT-2 for M2/vIP23. DGW1 and DGW2 will update their ARP tables with the new MAC resolving the floating IP. No changes are made in the IP-VRF table.

                   NVE2                           DGW1
            M2 +-----------+ +---------+    +-------------+
  +---TS2(VA)--|  (BD-10)  |-|         |----|  (BD-10)    |
  |      ESI23 +-----------+ |         |    |    IRB1\    |
  |        +                 |         |    |     (IP-VRF)|---+
  |        |                 |         |    +-------------+  _|_
 SN1       |                 |  VXLAN/ |                    (   )
  |        |                 |  GENEVE |         DGW2      ( WAN )
  |        +      NVE3       |         |    +-------------+ (___)
  |      ESI23 +-----------+ |         |----|  (BD-10)    |   |
  +---TS3(VA)--|  (BD-10)  |-|         |    |    IRB2\    |   |
            M3 +-----------+ +---------+    |     (IP-VRF)|---+
                                            +-------------+

a)

an ESI, i.e., ESI23, that can be provisioned on the attachment ports of NVE2 and NVE3, as shown in Figure 7 or

b)

the VA's MAC address, which can be added to NVE2 and NVE3 by policy.

(1)

Assuming TS2 is the active TS in ESI23, NVE2 advertises the following BGP routes:

• Route type 1 (Ethernet A-D route for BD-10) containing: ESI = ESI23 and the corresponding tunnel information (VNI field), as well as the BGP Encapsulation Extended Community as per [RFC 8365].
• Route type 5 (IP Prefix route) containing: IPL = 24, IP = SN1, ESI = ESI23, and GW IP address = 0. The EVPN Router's MAC Extended Community defined in [RFC 9135] is added and carries the MAC address (M2) associated with the TS behind which SN1 sits. M2 may be learned by policy; however, the MAC in the Extended Community is preferred if sent with the route.

(2)

NVE3 advertises the following BGP route for TS3 (no AD per EVI route is advertised):

• Route type 5 (IP Prefix route) containing: IPL = 24, IP = SN1, ESI = 23, and GW IP address = 0. The EVPN Router's MAC Extended Community is added and carries the MAC address (M3) associated with the TS behind which SN1 sits. M3 may be learned by policy; however, the MAC in the Extended Community is preferred if sent with the route.

(3)

DGW1 and DGW2 import the received routes based on the Route Target:

• The tunnel information to get to ESI23 is installed in DGW1 and DGW2. For the VXLAN use case, the VTEP will be derived from the Ethernet A-D route BGP next hop and VNI from the VNI/VSID field (see [RFC 8365]).
• The RT-5 coming from the NVE that advertised the RT-1 is selected, and SN1/24 is added to the IP-VRF in DGW1 and DGW2 with Overlay Index ESI23 and MAC = M2.

(4)

When DGW1 receives a packet from the WAN with destination IPx, where IPx belongs to SN1/24:

• A destination IP lookup is performed on the DGW1 IP-VRF table, and Overlay Index = ESI23 is found. Since ESI23 is an Overlay Index, a recursive route resolution is required to find the egress NVE where ESI23 resides.
• The IP packet destined to IPx is encapsulated with:
  • Inner source MAC = IRB1 MAC.
  • Inner destination MAC = M2 (this MAC will be obtained from the EVPN Router's MAC Extended Community received along with the RT-5 for SN1). Note that the EVPN Router's MAC Extended Community is used in this case to carry the TS's MAC address, as opposed to the MAC address of the NVE/PE.
  • Tunnel information for the NVO tunnel is provided by the Ethernet A-D route per EVI for ESI23 (VNI and VTEP IP for the VXLAN case).

(5)

When the packet arrives at NVE2:

• Based on the tunnel demultiplexer information (VNI for the VXLAN case), the BD-10 context is identified for a MAC lookup (assuming a MAC-based disposition model [RFC 7432]), or the VNI may directly identify the egress interface (for an MPLS-based disposition model, which in this context is a VNI-based disposition model).
• Encapsulation is stripped off and, based on a MAC lookup (assuming MAC forwarding on the egress NVE) or a VNI lookup (in case of VNI forwarding), the packet is forwarded to TS2, where it will be forwarded to SN1.

(6)

If the redundancy protocol running between TS2 and TS3 follows an active/standby model and there is a failure, TS3 is appointed as the new active TS for SN1. TS3 will now own the connectivity to SN1 and will signal this new ownership. Upon receiving the new owner's notification, NVE3's AC will become active and issue a route type 1 for ESI23, whereas NVE2 will withdraw its Ethernet A-D route for ESI23. DGW1 and DGW2 will update their tunnel information to resolve ESI23. The inner destination MAC will be changed to M3.

1. Traffic destined to the IRB or TS IP interfaces, as well as
2. Traffic destined to IP subnets sitting behind the TS, e.g., SN1 or SN2.

1. Interface-less model: no SBD and no Overlay Indexes required.
2. Interface-ful with an SBD IRB model: requires SBD as well as GW IP addresses as Overlay Indexes.
3. Interface-ful with an unnumbered SBD IRB model: requires SBD as well as MAC addresses as Overlay Indexes.

                   NVE1(M1)
          +------------+
  IP1+----|  (BD-1)    |                DGW1(M3)
          |      \     |    +---------+ +--------+
          |    (IP-VRF)|----|         |-|(IP-VRF)|----+
          |      /     |    |         | +--------+    |
      +---|  (BD-2)    |    |         |              _+_
      |   +------------+    |         |             (   )
   SN1|                     |  VXLAN/ |            ( WAN )--H1
      |            NVE2(M2) |  GENEVE/|             (___)
      |   +------------+    |  MPLS   |               +
      +---|  (BD-2)    |    |         | DGW2(M4)      |
          |       \    |    |         | +--------+    |
          |    (IP-VRF)|----|         |-|(IP-VRF)|----+
          |       /    |    +---------+ +--------+
  SN2+----|  (BD-3)    |
          +------------+

a)

The NVEs and DGWs must provide connectivity between hosts in SN1, SN2, and IP1 and hosts sitting at the other end of the WAN -- for example, H1. It is assumed that the DGWs import/export IP and/or VPN-IP routes to/from the WAN.

b)

The IP-VRF instances in the NVE/DGWs are directly connected through NVO tunnels, and no IRBs and/or BD instances are instantiated to connect the IP-VRFs.

c)

The solution must provide Layer 3 connectivity among the IP-VRFs for Ethernet NVO tunnels -- for instance, VXLAN or GENEVE.

d)

The solution may provide Layer 3 connectivity among the IP-VRFs for IP NVO tunnels -- for example, GENEVE (with IP payload).

• RD as per [RFC 7432].
• Ethernet Tag ID = 0.
• IP prefix length and IP address, as explained in the previous sections.
• GW IP address = 0.
• ESI = 0.
• MPLS label or VNI corresponding to the IP-VRF.

• The first one is the BGP Encapsulation Extended Community, as per [RFC 9012], identifying the tunnel type.
• The second one is the EVPN Router's MAC Extended Community, as per [RFC 9135], containing the MAC address associated with the NVE advertising the route. This MAC address identifies the NVE/DGW and MAY be reused for all the IP-VRFs in the NVE. The EVPN Router's MAC Extended Community must be sent if the route is associated with an Ethernet NVO tunnel -- for instance, VXLAN. If the route is associated with an IP NVO tunnel -- for instance, GENEVE with an IP payload -- the EVPN Router's MAC Extended Community should not be sent.

(1)

NVE1 advertises the following BGP route:

• Route type 5 (IP Prefix route) containing:
  • IPL = 24, IP = SN1, Label = 10.
  • GW IP = set to 0.
  • BGP Encapsulation Extended Community [RFC 9012].
  • EVPN Router's MAC Extended Community that contains M1.
  • Route Target identifying the tenant (IP-VRF).

(2)

DGW1 imports the received routes from NVE1:

• DGW1 installs SN1/24 in the IP-VRF identified by the RT-5 Route Target.
• Since GW IP = ESI = 0, the label is a non-zero value, and the local policy indicates this interface-less model, DGW1, will use the label and next hop of the RT-5, as well as the MAC address conveyed in the EVPN Router's MAC Extended Community (as the inner destination MAC address) to set up the forwarding state and later encapsulate the routed IP packets.

(3)

When DGW1 receives a packet from the WAN with destination IPx, where IPx belongs to SN1/24:

• A destination IP lookup is performed on the DGW1 IP-VRF table. The lookup yields SN1/24.
• Since the RT-5 for SN1/24 had a GW IP = ESI = 0, a non-zero label, and a next hop, and since the model is interface-less, DGW1 will not need a recursive lookup to resolve the route.
• The IP packet destined to IPx is encapsulated with: inner source MAC = DGW1 MAC, inner destination MAC = M1, outer source IP (tunnel source IP) = DGW1 IP, and outer destination IP (tunnel destination IP) = NVE1 IP. The source and inner destination MAC addresses are not needed if IP NVO tunnels are used.

(4)

When the packet arrives at NVE1:

• NVE1 will identify the IP-VRF for an IP lookup based on the label (the inner destination MAC is not needed to identify the IP-VRF).
• An IP lookup is performed in the routing context, where SN1 turns out to be a local subnet associated with BD-2. A subsequent lookup in the ARP table and the BD FIB will provide the forwarding information for the packet in BD-2.

                 NVE1
        +------------+                       DGW1
IP10+---+(BD-1)      | +---------------+ +------------+
        |  \         | |               | |            |
        |(IP-VRF)-(SBD)|               |(SBD)-(IP-VRF)|-----+
        |  /    IRB(M1/IP1)           IRB(M3/IP3)     |     |
    +---+(BD-2)      | |               | +------------+    _+_
    |   +------------+ |               |                  (   )
 SN1|                  |     VXLAN/    |                 ( WAN )--H1
    |            NVE2  |     GENEVE/   |                  (___)
    |   +------------+ |     MPLS      |     DGW2           +
    +---+(BD-2)      | |               | +------------+     |
        |  \         | |               | |            |     |
        |(IP-VRF)-(SBD)|               |(SBD)-(IP-VRF)|-----+
        |  /    IRB(M2/IP2)           IRB(M4/IP4)     |
SN2+----+(BD-3)      | +---------------+ +------------+
        +------------+

a)

As in Section 4.4.1, the NVEs and DGWs must provide connectivity between hosts in SN1, SN2, and IP10 and in hosts sitting at the other end of the WAN.

b)

However, the NVE/DGWs are now connected through Ethernet NVO tunnels terminated in the SBD instance. The IP-VRFs use IRB interfaces for their connectivity to the SBD.

c)

Each SBD IRB has an IP and a MAC address, where the IP address must be reachable from other NVEs or DGWs.

d)

The SBD is attached to all the NVE/DGWs in the tenant domain BDs.

e)

The solution must provide Layer 3 connectivity for Ethernet NVO tunnels -- for instance, VXLAN or GENEVE (with Ethernet payload).

• RD as per [RFC 7432].
• Ethernet Tag ID = 0.
• IP prefix length and IP address, as explained in the previous sections.
• GW IP address = IRB-IP of the SBD (this is the Overlay Index that will be used for the recursive route resolution).
• ESI = 0.
• Label value should be zero since the RT-5 route requires a recursive lookup resolution to an RT-2 route. It is ignored on reception, and the MPLS label or VNI from the RT-2's MPLS Label1 field is used when forwarding packets.

(1)

NVE1 advertises the following BGP routes:

• Route type 5 (IP Prefix route) containing:
  • IPL = 24, IP = SN1, Label = SHOULD be set to 0.
  • GW IP = IP1 (SBD IRB's IP).
  • Route Target identifying the tenant (IP-VRF).
• Route type 2 (MAC/IP Advertisement route for the SBD IRB) containing:
  • ML = 48, M = M1, IPL = 32, IP = IP1, Label = 10.
  • A BGP Encapsulation Extended Community [RFC 9012].
  • Route Target identifying the SBD. This Route Target may be the same as the one used with the RT-5.

(2)

DGW1 imports the received routes from NVE1:

• DGW1 installs SN1/24 in the IP-VRF identified by the RT-5 Route Target.
  • Since GW IP is different from zero, the GW IP (IP1) will be used as the Overlay Index for the recursive route resolution to the RT-2 carrying IP1.

(3)

When DGW1 receives a packet from the WAN with destination IPx, where IPx belongs to SN1/24:

• A destination IP lookup is performed on the DGW1 IP-VRF table. The lookup yields SN1/24, which is associated with the Overlay Index IP1. The forwarding information is derived from the RT-2 received for IP1.
• The IP packet destined to IPx is encapsulated with: inner source MAC = M3, inner destination MAC = M1, outer source IP (source VTEP) = DGW1 IP, and outer destination IP (destination VTEP) = NVE1 IP.

(4)

When the packet arrives at NVE1:

• NVE1 will identify the IP-VRF for an IP lookup based on the label and the inner MAC DA.
• An IP lookup is performed in the routing context, where SN1 turns out to be a local subnet associated with BD-2. A subsequent lookup in the ARP table and the BD FIB will provide the forwarding information for the packet in BD-2.

                 NVE1
        +------------+                       DGW1
IP1+----+(BD-1)      | +---------------+ +------------+
        |  \         | |               | |            |
        |(IP-VRF)-(SBD)|               (SBD)-(IP-VRF) |-----+
        |  /    IRB(M1)|               | IRB(M3)      |     |
    +---+(BD-2)      | |               | +------------+    _+_
    |   +------------+ |               |                  (   )
 SN1|                  |     VXLAN/    |                 ( WAN )--H1
    |            NVE2  |     GENEVE/   |                  (___)
    |   +------------+ |     MPLS      |     DGW2           +
    +---+(BD-2)      | |               | +------------+     |
        |  \         | |               | |            |     |
        |(IP-VRF)-(SBD)|               (SBD)-(IP-VRF) |-----+
        |  /    IRB(M2)|               | IRB(M4)      |
SN2+----+(BD-3)      | +---------------+ +------------+
        +------------+

a)

As in Sections [4.4.1] and [4.4.2], the NVEs and DGWs must provide connectivity between hosts in SN1, SN2, and IP1 and in hosts sitting at the other end of the WAN.

b)

As in Section 4.4.2, the NVE/DGWs are connected through Ethernet NVO tunnels terminated in the SBD instance. The IP-VRFs use IRB interfaces for their connectivity to the SBD.

c)

However, each SBD IRB has a MAC address only and no IP address (which is why the model refers to an "unnumbered" SBD IRB). In this model, there is no need to have IP reachability to the SBD IRB interfaces themselves, and there is a requirement to limit the number of IP addresses used.

d)

As in Section 4.4.2, the SBD is composed of all the NVE/DGW BDs of the tenant that need inter-subnet forwarding.

e)

As in Section 4.4.2, the solution must provide Layer 3 connectivity for Ethernet NVO tunnels -- for instance, VXLAN or GENEVE (with Ethernet payload).

• GW IP address = set to 0.

(1)

NVE1 advertises the following BGP routes:

• Route type 5 (IP Prefix route) containing the same values as in the example in Section 4.4.2, except:
  • GW IP = SHOULD be set to 0.
  • EVPN Router's MAC Extended Community containing M1 (this will be used for the recursive lookup to an RT-2).
• Route type 2 (MAC route for the SBD IRB) with the same values as in Section 4.4.2, except:
  • ML = 48, M = M1, IPL = 0, Label = 10.

(2)

DGW1 imports the received routes from NVE1:

• DGW1 installs SN1/24 in the IP-VRF identified by the RT-5 Route Target.
  • The MAC contained in the EVPN Router's MAC Extended Community sent along with the RT-5 (M1) will be used as the Overlay Index for the recursive route resolution to the RT-2 carrying M1.

(3)

When DGW1 receives a packet from the WAN with destination IPx, where IPx belongs to SN1/24:

• A destination IP lookup is performed on the DGW1 IP-VRF table. The lookup yields SN1/24, which is associated with the Overlay Index M1. The forwarding information is derived from the RT-2 received for M1.
• The IP packet destined to IPx is encapsulated with: inner source MAC = M3, inner destination MAC = M1, outer source IP (source VTEP) = DGW1 IP, and outer destination IP (destination VTEP) = NVE1 IP.

(4)

When the packet arrives at NVE1:

• NVE1 will identify the IP-VRF for an IP lookup based on the label and the inner MAC DA.
• An IP lookup is performed in the routing context, where SN1 turns out to be a local subnet associated with BD-2. A subsequent lookup in the ARP table and the BD FIB will provide the forwarding information for the packet in BD-2.

[EVPNRouteTypes]

IANA, "EVPN Route Types",
<https://www.iana.org/assignments/evpn>.

[RFC2119]

S. Bradner, "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.

[RFC7432]

A. Sajassi, R. Aggarwal, N. Bitar, A. Isaac, J. Uttaro, J. Drake, and W. Henderickx, "BGP MPLS-Based Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 2015,
<https://www.rfc-editor.org/info/rfc7432>.

[RFC8174]

B. Leiba, "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017,
<https://www.rfc-editor.org/info/rfc8174>.

[RFC8365]

A. Sajassi, J. Drake, N. Bitar, R. Shekhar, J. Uttaro, and W. Henderickx, "A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, DOI 10.17487/RFC8365, March 2018,
<https://www.rfc-editor.org/info/rfc8365>.

[RFC9012]

K. Patel, G. Van de Velde, S. Sangli, and J. Scudder, "The BGP Tunnel Encapsulation Attribute", RFC 9012, DOI 10.17487/RFC9012, April 2021,
<https://www.rfc-editor.org/info/rfc9012>.

[RFC9135]

A Sajassi, S Salam, S Thoria, J Drake, and J Rabadan, "Integrated Routing and Bridging in Ethernet VPN (EVPN)", RFC 9135, DOI 10.17487/RFC9135, October 2021,
<https://www.rfc-editor.org/info/rfc9135>.

[IEEE-802.1Q]

IEEE, "IEEE Standard for Local and Metropolitan Area Networks -- Bridges and Bridged Networks", DOI 10.1109/IEEESTD.2018.8403927, IEEE Std 802.1Q, July 2018,
<https://standards.ieee.org/standard/802_1Q-2018.html>.

[RFC4364]

E. Rosen, and Y. Rekhter, "BGP/MPLS IP Virtual Private Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 2006,
<https://www.rfc-editor.org/info/rfc4364>.

[RFC5227]

S. Cheshire, "IPv4 Address Conflict Detection", RFC 5227, DOI 10.17487/RFC5227, July 2008,
<https://www.rfc-editor.org/info/rfc5227>.

[RFC5798]

S. Nadas, "Virtual Router Redundancy Protocol (VRRP) Version 3 for IPv4 and IPv6", RFC 5798, DOI 10.17487/RFC5798, March 2010,
<https://www.rfc-editor.org/info/rfc5798>.

[RFC7348]

M. Mahalingam, D. Dutt, K. Duda, P. Agarwal, L. Kreeger, T. Sridhar, M. Bursell, and C. Wright, "Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
<https://www.rfc-editor.org/info/rfc7348>.

[RFC7365]

M. Lasserre, F. Balus, T. Morin, N. Bitar, and Y. Rekhter, "Framework for Data Center (DC) Network Virtualization", RFC 7365, DOI 10.17487/RFC7365, October 2014,
<https://www.rfc-editor.org/info/rfc7365>.

[RFC7606]

E. Chen, J. Scudder, P. Mohapatra, and K. Patel, "Revised Error Handling for BGP UPDATE Messages", RFC 7606, DOI 10.17487/RFC7606, August 2015,
<https://www.rfc-editor.org/info/rfc7606>.

[RFC8926]

J. Gross, I. Ganga, and T. Sridhar, "Geneve: Generic Network Virtualization Encapsulation", RFC 8926, DOI 10.17487/RFC8926, November 2020,
<https://www.rfc-editor.org/info/rfc8926>.

• Senthil Sathappan
• Florin Balus
• Aldrin Isaac
• Senad Palislamovic
• Samir Thoria

Jorge Rabadan

Wim Henderickx

John Drake

Wen Lin

Ali Sajassi