Internet Engineering Task Force (IETF)                C. Villamizar, Ed.
Request for Comments: 7325                                         OCCNC
Category: Informational                                      K. Kompella
ISSN: 2070-1721                                         Juniper Networks
                                                               S. Amante
                                                              Apple Inc.
                                                                A. Malis
                                                                  Huawei
                                                            C. Pignataro
                                                                   Cisco
                                                             August 2014


        MPLS Forwarding Compliance and Performance Requirements

Abstract

   This document provides guidelines for implementers regarding MPLS
   forwarding and a basis for evaluations of forwarding implementations.
   Guidelines cover many aspects of MPLS forwarding.  Topics are
   highlighted where implementers might otherwise overlook practical
   requirements that are unstated or underemphasized, or that are
   optional for conformance to RFCs but often considered mandatory by
   providers.

Status of This Memo

   This document is not an Internet Standards Track specification; it is
   published for informational purposes.

   This document is a product of the Internet Engineering Task Force
   (IETF).  It represents the consensus of the IETF community.  It has
   received public review and has been approved for publication by the
   Internet Engineering Steering Group (IESG).  Not all documents
   approved by the IESG are a candidate for any level of Internet
   Standard; see Section 2 of RFC 5741.

   Information about the current status of this document, any errata,
   and how to provide feedback on it may be obtained at
   http://www.rfc-editor.org/info/rfc7325.

Copyright Notice

   Copyright (c) 2014 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents

   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction and Document Scope . . . . . . . . . . . . . . .   4
     1.1.  Abbreviations . . . . . . . . . . . . . . . . . . . . . .   4
     1.2.  Use of Requirements Language  . . . . . . . . . . . . . .   8
     1.3.  Apparent Misconceptions . . . . . . . . . . . . . . . . .   9
     1.4.  Target Audience . . . . . . . . . . . . . . . . . . . . .  10
   2.  Forwarding Issues . . . . . . . . . . . . . . . . . . . . . .  11
     2.1.  Forwarding Basics . . . . . . . . . . . . . . . . . . . .  11
       2.1.1.  MPLS Special-Purpose Labels . . . . . . . . . . . . .  12
       2.1.2.  MPLS Differentiated Services  . . . . . . . . . . . .  13
       2.1.3.  Time Synchronization  . . . . . . . . . . . . . . . .  14
       2.1.4.  Uses of Multiple Label Stack Entries  . . . . . . . .  14
       2.1.5.  MPLS Link Bundling  . . . . . . . . . . . . . . . . .  15
       2.1.6.  MPLS Hierarchy  . . . . . . . . . . . . . . . . . . .  16
       2.1.7.  MPLS Fast Reroute (FRR) . . . . . . . . . . . . . . .  16
       2.1.8.  Pseudowire Encapsulation  . . . . . . . . . . . . . .  17
         2.1.8.1.  Pseudowire Sequence Number  . . . . . . . . . . .  17
       2.1.9.  Layer 2 and Layer 3 VPN . . . . . . . . . . . . . . .  19
     2.2.  MPLS Multicast  . . . . . . . . . . . . . . . . . . . . .  20
     2.3.  Packet Rates  . . . . . . . . . . . . . . . . . . . . . .  21
     2.4.  MPLS Multipath Techniques . . . . . . . . . . . . . . . .  23
       2.4.1.  Pseudowire Control Word . . . . . . . . . . . . . . .  24
       2.4.2.  Large Microflows  . . . . . . . . . . . . . . . . . .  24
       2.4.3.  Pseudowire Flow Label . . . . . . . . . . . . . . . .  25
       2.4.4.  MPLS Entropy Label  . . . . . . . . . . . . . . . . .  25
       2.4.5.  Fields Used for Multipath Load Balance  . . . . . . .  25
         2.4.5.1.  MPLS Fields in Multipath  . . . . . . . . . . . .  26
         2.4.5.2.  IP Fields in Multipath  . . . . . . . . . . . . .  27
         2.4.5.3.  Fields Used in Flow Label . . . . . . . . . . . .  29
         2.4.5.4.  Fields Used in Entropy Label  . . . . . . . . . .  29
     2.5.  MPLS-TP and UHP . . . . . . . . . . . . . . . . . . . . .  30

     2.6.  Local Delivery of Packets . . . . . . . . . . . . . . . .  30
       2.6.1.  DoS Protection  . . . . . . . . . . . . . . . . . . .  31
       2.6.2.  MPLS OAM  . . . . . . . . . . . . . . . . . . . . . .  33
       2.6.3.  Pseudowire OAM  . . . . . . . . . . . . . . . . . . .  34
       2.6.4.  MPLS-TP OAM . . . . . . . . . . . . . . . . . . . . .  34
       2.6.5.  MPLS OAM and Layer 2 OAM Interworking . . . . . . . .  35
       2.6.6.  Extent of OAM Support by Hardware . . . . . . . . . .  36
       2.6.7.  Support for IPFIX in Hardware . . . . . . . . . . . .  37
     2.7.  Number and Size of Flows  . . . . . . . . . . . . . . . .  37
   3.  Questions for Suppliers . . . . . . . . . . . . . . . . . . .  38
     3.1.  Basic Compliance  . . . . . . . . . . . . . . . . . . . .  38
     3.2.  Basic Performance . . . . . . . . . . . . . . . . . . . .  40
     3.3.  Multipath Capabilities and Performance  . . . . . . . . .  41
     3.4.  Pseudowire Capabilities and Performance . . . . . . . . .  41
     3.5.  Entropy Label Support and Performance . . . . . . . . . .  42
     3.6.  DoS Protection  . . . . . . . . . . . . . . . . . . . . .  42
     3.7.  OAM Capabilities and Performance  . . . . . . . . . . . .  42
   4.  Forwarding Compliance and Performance Testing . . . . . . . .  43
     4.1.  Basic Compliance  . . . . . . . . . . . . . . . . . . . .  43
     4.2.  Basic Performance . . . . . . . . . . . . . . . . . . . .  44
     4.3.  Multipath Capabilities and Performance  . . . . . . . . .  45
     4.4.  Pseudowire Capabilities and Performance . . . . . . . . .  46
     4.5.  Entropy Label Support and Performance . . . . . . . . . .  46
     4.6.  DoS Protection  . . . . . . . . . . . . . . . . . . . . .  47
     4.7.  OAM Capabilities and Performance  . . . . . . . . . . . .  47
   5.  Security Considerations . . . . . . . . . . . . . . . . . . .  48
   6.  Organization of References Section  . . . . . . . . . . . . .  50
   7.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  50
     7.1.  Normative References  . . . . . . . . . . . . . . . . . .  50
     7.2.  Informative References  . . . . . . . . . . . . . . . . .  53
   Appendix A.  Acknowledgements . . . . . . . . . . . . . . . . . .  59

1.  Introduction and Document Scope

   The initial purpose of this document was to address concerns raised
   on the MPLS WG mailing list about shortcomings in implementations of
   MPLS forwarding.  Documenting existing misconceptions and potential
   pitfalls might potentially avoid repeating past mistakes.  The
   document has grown to address a broad set of forwarding requirements.

   The focus of this document is MPLS forwarding, base pseudowire
   forwarding, and MPLS Operations, Administration, and Maintenance
   (OAM).  The use of pseudowire Control Word and the use of pseudowire
   Sequence Number are discussed.  Specific pseudowire Attachment
   Circuit (AC) and Native Service Processing (NSP) are out of scope.
   Specific pseudowire applications, such as various forms of Virtual
   Private Network (VPN), are out of scope.

   MPLS support for multipath techniques is considered essential by many
   service providers and is useful for other high-capacity networks.  In
   order to obtain sufficient entropy from MPLS, traffic service
   providers and others find it essential for the MPLS implementation to
   interpret the MPLS payload as IPv4 or IPv6 based on the contents of
   the first nibble of payload.  The use of IP addresses, the IP
   protocol field, and UDP and TCP port number fields in multipath load
   balancing are considered within scope.  The use of any other IP
   protocol fields, such as tunneling protocols carried within IP, are
   out of scope.

   Implementation details are a local matter and are out of scope.  Most
   interfaces today operate at 1 Gb/s or greater.  It is assumed that
   all forwarding operations are implemented in specialized forwarding
   hardware rather than on a general-purpose processor.  This is often
   referred to as "fast path" and "slow path" processing.  Some
   recommendations are made regarding implementing control or
   management-plane functionality in specialized hardware or with
   limited assistance from specialized hardware.  This advice is based
   on expected control or management protocol loads and on the need for
   denial of service (DoS) protection.

1.1.  Abbreviations

   The following abbreviations are used.

   AC    Attachment Circuit ([RFC3985])

   ACH   Associated Channel Header (pseudowires)

   ACK   Acknowledgement (TCP flag and type of TCP packet)

   AIS   Alarm Indication Signal (MPLS-TP OAM)

   ATM   Asynchronous Transfer Mode (legacy switched circuits)

   BFD   Bidirectional Forwarding Detection

   BGP   Border Gateway Protocol

   CC-CV Continuity Check and Connectivity Verification

   CE    Customer Edge ([RFC4364])

   CPU   Central Processing Unit (computer or microprocessor)

   CT    Class Type ([RFC4124])

   CW    Control Word ([RFC4385])

   DCCP  Datagram Congestion Control Protocol

   DDoS  Distributed Denial of Service

   DM    Delay Measurement (MPLS-TP OAM)

   DSCP  Differentiated Services Code Point ([RFC2474])

   DWDM  Dense Wave Division Multiplexing

   DoS   Denial of Service

   E-LSP Explicitly TC-encoded-PSC LSP ([RFC5462])

   EBGP  External BGP

   ECMP  Equal-Cost Multipath

   ECN   Explicit Congestion Notification ([RFC3168] and [RFC5129])

   EL    Entropy Label ([RFC6790])

   ELI   Entropy Label Indicator ([RFC6790])

   EXP   Experimental (field in MPLS renamed to "TC" in [RFC5462])

   FEC   Forwarding Equivalence Classes ([RFC3031]); also Forward Error
         Correction in other context

   FR    Frame Relay (legacy switched circuits)

   FRR   Fast Reroute ([RFC4090])

   G-ACh Generic Associated Channel ([RFC5586])

   GAL   Generic Associated Channel Label ([RFC5586])

   GFP   Generic Framing Procedure (used in OTN)

   GMPLS Generalized MPLS ([RFC3471])

   GTSM  Generalized TTL Security Mechanism ([RFC5082])

   Gb/s  Gigabits per second (billion bits per second)

   IANA  Internet Assigned Numbers Authority

   ILM   Incoming Label Map ([RFC3031])

   IP    Internet Protocol

   IPVPN Internet Protocol VPN

   IPv4  Internet Protocol version 4

   IPv6  Internet Protocol version 6

   L-LSP Label-Only-Inferred-PSC LSP ([RFC3270])

   L2VPN Layer 2 VPN

   LDP   Label Distribution Protocol ([RFC5036])

   LER   Label Edge Router ([RFC3031])

   LM    Loss Measurement (MPLS-TP OAM)

   LSP   Label Switched Path ([RFC3031])

   LSR   Label Switching Router ([RFC3031])

   MP2MP Multipoint to Multipoint

   MPLS  Multiprotocol Label Switching ([RFC3031])

   MPLS-TP  MPLS Transport Profile ([RFC5317])

   Mb/s  Megabits per second (million bits per second)

   NSP   Native Service Processing ([RFC3985])

   NTP   Network Time Protocol

   OAM   Operations, Administration, and Maintenance ([RFC6291])

   OOB   Out-of-band (not carried within a data channel)

   OTN   Optical Transport Network

   P     Provider router ([RFC4364])

   P2MP  Point to Multipoint

   PE    Provider Edge router ([RFC4364])

   PHB   Per-Hop Behavior ([RFC2475])

   PHP   Penultimate Hop Popping ([RFC3443])

   POS   PPP over SONET

   PSC   This abbreviation has multiple interpretations.

         1.  Packet Switch Capable ([RFC3471]

         2.  PHB Scheduling Class ([RFC3270])

         3.  Protection State Coordination ([RFC6378])

   PTP   Precision Time Protocol

   PW    Pseudowire

   QoS   Quality of Service

   RA    Router Alert ([RFC3032])

   RDI   Remote Defect Indication (MPLS-TP OAM)

   RSVP-TE  RSVP Traffic Engineering

   RTP   Real-time Transport Protocol

   SCTP  Stream Control Transmission Protocol

   SDH   Synchronous Data Hierarchy (European SONET, a form of TDM)

   SONET Synchronous Optical Network (US SDH, a form of TDM)

   T-LDP Targeted LDP (LDP sessions over more than one hop)

   TC    Traffic Class ([RFC5462])

   TCP   Transmission Control Protocol

   TDM   Time-Division Multiplexing (legacy encapsulations)

   TOS   Type of Service (see [RFC2474])

   TTL   Time-to-live (a field in IP and MPLS headers)

   UDP   User Datagram Protocol

   UHP   Ultimate Hop Popping (opposite of PHP)

   VCCV  Virtual Circuit Connectivity Verification ([RFC5085])

   VLAN  Virtual Local Area Network (Ethernet)

   VOQ   Virtual Output Queuing (switch fabric design)

   VPN   Virtual Private Network

   WG    Working Group

1.2.  Use of Requirements Language

   This document is Informational.  The uppercase [RFC2119] key words
   "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", and "MAY" are used in
   this document in the following cases.

   1.  RFC 2119 keywords are used where requirements stated in this
       document are called for in referenced RFCs.  In most cases, the
       RFC containing the requirement is cited within the statement
       using an RFC 2119 keyword.

   2.  RFC 2119 keywords are used where explicitly noted that the
       keywords indicate that operator experiences indicate a
       requirement, but there are no existing RFC requirements.

   Advice provided by this document may be ignored by implementations.
   Similarly, implementations not claiming conformance to specific RFCs
   may ignore the requirements of those RFCs.  In both cases,
   implementers should consider the risk of doing so.

1.3.  Apparent Misconceptions

   In early generations of forwarding silicon (which might now be behind
   us), there apparently were some misconceptions about MPLS.  The
   following statements provide clarifications.

   1.  There are practical reasons to have more than one or two labels
       in an MPLS label stack.  Under some circumstances, the label
       stack can become quite deep.  See Section 2.1.

   2.  The label stack MUST be considered to be arbitrarily deep.
       Section 3.27.4 ("Hierarchy: LSP Tunnels within LSPs") of RFC 3031
       states "The label stack mechanism allows LSP tunneling to nest to
       any depth" [RFC3031].  If a bottom of the label stack cannot be
       found, but sufficient number of labels exist to forward, an LSR
       MUST forward the packet.  An LSR MUST NOT assume the packet is
       malformed unless the end of packet is found before the bottom of
       the stack.  See Section 2.1.

   3.  In networks where deep label stacks are encountered, they are not
       rare.  Full packet rate performance is required regardless of
       label stack depth, except where multiple pop operations are
       required.  See Section 2.1.

   4.  Research has shown that long bursts of short packets with 40-byte
       or 44-byte IP payload sizes in these bursts are quite common.
       This is due to TCP ACK compression [ACK-compression].  The
       following two sub-bullets constitute advice that reflects very
       common nonnegotiable requirements of providers.  Implementers may
       ignore this advice but should consider the risk of doing so.

       A.  A forwarding engine SHOULD, if practical, be able to sustain
           an arbitrarily long sequence of small packets arriving at
           full interface rate.

       B.  If indefinitely sustained full packet rate for small packets
           is not practical, a forwarding engine MUST be able to buffer
           a long sequence of small packets inbound to the on-chip
           decision engine and sustain full interface rate for some
           reasonable average packet rate.  Absent this small on-chip
           buffering, QoS-agnostic packet drops can occur.

       See Section 2.3.

   5.  The implementations and system designs MUST support pseudowire
       Control Word (CW) if MPLS-TP is supported or if ACH [RFC5586] is
       being used on a pseudowire.  The implementation and system
       designs SHOULD support pseudowire CW even if MPLS-TP and ACH

       [RFC5586] are not used, using instead CW and VCCV Type 1
       [RFC5085] to allow the use of multipath in the underlying network
       topology without impacting the PW traffic.  [RFC7079] does note
       that there are still some deployments where the CW is not always
       used.  It also notes that many service providers do enable the
       CW.  See Section 2.4.1 for more discussion on why deployments
       SHOULD enable the pseudowire CW.

   The following statements provide clarification regarding more recent
   requirements that are often missed.

   1.  The implementer and system designer SHOULD support adding a
       pseudowire Flow Label [RFC6391].  Deployments MAY enable this
       feature for appropriate pseudowire types.  See Section 2.4.3.

   2.  The implementer and system designer SHOULD support adding an MPLS
       Entropy Label [RFC6790].  Deployments MAY enable this feature.
       See Section 2.4.4.

   Non-IETF definitions of MPLS exist, and these should not be used as
   normative texts in place of the relevant IETF RFCs.  [RFC5704]
   documents incompatibilities between the IETF definition of MPLS and
   one such alternative MPLS definition, which led to significant issues
   in the resulting non-IETF specification.

1.4.  Target Audience

   This document is intended for multiple audiences: implementer
   (implementing MPLS forwarding in silicon or in software); systems
   designer (putting together a MPLS forwarding systems); deployer
   (running an MPLS network).  These guidelines are intended to serve
   the following purposes:

   1.  Explain what to do and what not to do when a deep label stack is
       encountered. (audience: implementer)

   2.  Highlight pitfalls to look for when implementing an MPLS
       forwarding chip. (audience: implementer)

   3.  Provide a checklist of features and performance specifications to
       request.  (audience: systems designer, deployer)

   4.  Provide a set of tests to perform.  (audience: systems designer,
       deployer).

   The implementer, systems designer, and deployer have a transitive
   supplier-customer relationship.  It is in the best interest of the
   supplier to review their product against their customer's checklist
   and secondary customer's checklist if applicable.

   This document identifies and explains many details and potential
   pitfalls of MPLS forwarding.  It is likely that the identified set of
   potential pitfalls will later prove to be an incomplete set.