Network Working Group                                         Y. Rekhter
Request for Comments: 1787        T.J. Watson Research Center, IBM Corp.
Category: Informational                                       April 1995


                  Routing in a Multi-provider Internet

Status of this Memo

   This memo provides information for the Internet community.  This memo
   does not specify an Internet standard of any kind.  Distribution of
   this memo is unlimited.

Abstract

   This document was prepared by the author on behalf of the Internet
   Architecture Board (IAB). It is offered by the IAB to stimulate
   discussion.

   Over the past few years the Internet has undergone significant
   changes.  Among them is the emergence of multiple Network Service
   Providers, where resources that provide Internet-wide IP connectivity
   (routers, links) are controlled by different organizations.  This
   document presents some of the issues related to network layer routing
   in a multi-provider Internet, and specifically to the unicast
   routing.

1. Network Service Providers vs Network Service Subscribers

   Within the current routing paradigm the service offered by a provider
   at the network layer (IP) is the set of destinations (hosts) that can
   be reached through the provider. Once a subscriber establishes direct
   connectivity to a provider, the subscriber can in principle reach all
   the destinations reachable through the provider. Since the value of
   the Internet-wide connectivity service offered by a provider
   increases with the number of destinations reachable through the
   provider, providers are motivated to interconnect with each other.

   In principle a provider need not offer the same service (in terms of
   the set of destinations) to all of its subscribers -- for some of the
   subscribers the provider may restrict the services to a subset of the
   destinations reachable through the provider. In fact, for certain
   types of subscribers constrained connectivity could be seen as part
   of the service offered by a provider.

   In a multi-provider environment individual providers may be driven by
   diverse and sometimes even conflicting goals and objectives. Some of
   the providers exist to provide connectivity to only a specific group

   of Network Service Subscribers. Other providers place no constraints
   on the subscribers that can subscribe to them, as long as the
   subscribers pay the fee charged by the providers. Some of the
   providers place certain constraints on the reselling of the
   connectivity services by organizations (e.g., other providers)
   attached to the providers. Some of the providers may be operated by
   companies that are subject to specific regulations (e.g.,  regulated
   monopoly), while other providers are completely unregulated.  The
   scope of geographical coverage among providers varies from a small
   region (e.g., county, town) to a country-wide, international, or even
   intercontinental.

   There is no centralized control over all the providers in the
   Internet.  The providers do not always coordinate their efforts with
   each other, and quite often are in competition with each other.

   Despite all the diversity among the providers, the Internet-wide IP
   connectivity is realized via Internet-wide distributed routing, which
   involves multiple providers, and thus implies certain degree of
   cooperation and coordination. Therefore, there is a need to balance
   the providers' goals and objectives against the public interest of
   Internet-wide connectivity and subscribers' choices. Further work is
   needed to understand how to reach the balance.

2. Routing Requirements

   Conceptually routing requirements can be classified into the
   following three categories: source preferences, destination
   preferences, and constraints on transit traffic. Source preferences
   allow an originator of a packet to exert control over the path to a
   destination.  Destination preferences allow a destination to exert
   control over the path from a source to the destination. Constraints
   on transit traffic allow a provider to control the traffic that can
   traverse through the resources (routers, links) controlled by the
   provider.

   From a conceptual point of view the requirements over the degree of
   control for source and destination preferences may vary from being
   able to just provide connectivity (regardless of the path), to being
   able to select immediate providers, to more complex scenarios, where
   at the other extreme a subscriber may want to have complete control
   over the path selection.

   From a conceptual point of view the requirements over the degree of
   control for transit traffic may vary from control based only on the
   direct physical connectivity (controlling the set of organizations
   directly connected to the provider), to being able to restrict
   traffic to a particular set of sources or destinations, or a

   combination of particular sources and destinations, or even take into
   account the paths to/from these sources and/or destinations.

   In view of a potentially wide variety of routing requirements, we
   need to get a better understanding on the relative practical
   importance of various routing requirements. In practice organizations
   usually don't formulate their routing requirements in a vacuum. For
   example, since the primary role of a provider is to provide services
   to a set of subscribers, the provider usually formulates its routing
   requirements based on the set of the routing requirements of the
   subscribers the provider is expected to serve.

   Support for various routing requirements should take into account the
   overhead and the scope of the overhead associated with those
   requirements. A situation where an organization can unilaterally
   impose routing information overhead on other organization (e.g., by
   requiring the other organization to maintain an additional routing
   information) should be viewed as undesirable. The cost of supporting
   a particular routing requirement should not be borne by organizations
   that do not benefit from supporting that requirement. Ideally the
   routing system should allow to shift the overhead associated with a
   particular routing requirement towards the entity that instigates the
   requirement (for example, there is a need to carefully balance the
   overhead associated with maintaining a state needed for multi-hop
   header compression vs carrying explicit forwarding information on a
   per packet basis).  Organizations with simple routing requirements
   shouldn't bear the same routing information overhead as organizations
   with complex routing requirements.

   A situation where the overhead associated with supporting a
   particular routing requirement has to be carried by every entity
   (e.g., router, host) within an organization that would like to impose
   the requirement could be viewed as undesirable. An organization
   should be able to instantiate its routing requirements in a more or
   less central fashion, for example by utilizing just some of the
   routers.

   Even if the scope of the routing information overhead is purely
   local, there is a need to perform a careful analysis of the tradeoff
   between the potential benefits and the cost associated with
   supporting various routing requirements.

3. Encapsulation

   The technique of encapsulation allows for the creation of a "virtual"
   IP overlay over an existing IP infrastructure. This has certain
   implications for the Internet routing system.

   In the presence of encapsulation, a provider may no longer be able to
   constrain its transit traffic to a particular set of ultimate sources
   and/or destinations, as a packet may be encapsulated by some router
   along the path, with the original source and/or destination addresses
   being "hidden" (via encapsulation) at the Network layer. Likewise,
   encapsulation may affect source and destination preferences, as a
   source (or a destination) may either (a) be unaware of the
   encapsulation, or (b) have little or no control over the encapsulated
   segment of a path.

   Further work is needed to understand the implications of the overlay
   capabilities created via encapsulation on the semantics of routing
   requirements, as well as the interaction among the routing
   requirements by the entities that form the overlay and the entities
   that form the underlying infrastructure.

4. Price Structure and its Impact on Routing

   Routing among providers, as well as between providers and subscribers
   may be influenced by the price structure employed by the providers,
   as well as the usage pattern of the subscribers. A provider can view
   routing as a mechanism that allows the provider to exert control over
   who can use the provider's services. A subscriber can view routing as
   a mechanism that allows the subscriber to exert control over the
   price it pays for the Internet connectivity.

   The need to exert control has to be carefully balanced against the
   cost of the routing mechanisms needed to provide such control. In a
   competitive market one could question the viability of a mechanism
   whose incremental cost would be greater than the saving recovered by
   the mechanism -- competitive pressure or alternate mechanisms are
   likely to push providers and subscribers towards choosing the
   cheapest mechanism.

5. Scalability

   One of the key requirements imposed on the Internet routing is its
   ability to scale. In addition to conventional metrics for scalability
   (e.g., memory, CPU, bandwidth), we need to take into account
   scalability with respect to the human resources required to operate
   the system. The need for deployment of CIDR already showed that a
   routing scheme that scales linearly with respect to the number of
   connected networks, or even to the number of connected organizations
   is unacceptable today, and is likely to be unacceptable in the long
   term. It is not clear whether routing that scales linearly with the
   number of providers is going to be acceptable in the long term.

   Scaling implies that the Internet routing system needs to have
   powerful mechanisms to provide routing information
   aggregation/abstraction.

   In the absence of Internet-wide coordination and in the presence of
   competition among the providers, the aggregation/abstraction
   mechanisms should minimize preconditions as well as limit the amount
   of required inter-provider coordination. Ideally the routing system
   should allow a provider to control the amount of its local resources
   needed to deal with the routing overhead based on considerations that
   are purely local to the provider.

   One of the side effects of the routing information
   aggregation/abstraction is that some of the routing information is
   going to be lost. This may impact route optimality and even the
   ability to find an existing route. The need for routing information
   aggregation/abstraction also implies certain homogeneity of the
   information to be aggregated/abstracted. This needs to be counter-
   balanced against the potential diversity of routing requirements.

   As a way to deal with the routing information loss due to
   aggregation/abstraction, we need to explore mechanisms that allow
   routing that is based on the on-demand acquisition of subsets of
   unaggregated information.

   The overhead associated with supporting specific routing requirements
   has a direct impact on the overall scalability of the Internet
   routing system. We need to get a better understanding of how various
   routing requirements impact scalability. When the impact is
   significant, and the requirements have practical importance we need
   to develop mechanisms that allow the impact to be reduced.

6. Hierarchical Routing

   Classless Inter-Domain Routing (CIDR) (RFC1518, RFC1519) that is used
   today for scalable Internet-wide routing is based on the technique of
   hierarchical routing. Essential to this technique is the assumption
   that Network layer addresses assigned to individual entities (e.g.,
   hosts, routers) reflect the position of these entities within the
   network topology -- addresses are said to be "topologically
   significant". With CIDR addresses assigned to most of the individual
   sites are expected to reflect providers the sites are connected to --
   CIDR uses "provider-based" addresses.

   One of the fundamental consequences of using hierarchical routing is
   that in order to preserve topological significance of network
   addresses, changes in the network topology may need to be accompanied
   by the corresponding changes in the addresses. Presence of multiple

   providers serving the same geographical area implies that a
   subscriber should be able to switch from one provider to another.
   Since such a switch implies changes in the Internet topology, it
   follows that to retain topological significance of the (provider-
   based) addresses within the subscriber, the subscriber has to change
   the addresses of all of its entities -- the process known as
   "renumbering". There are already tools to facilitate this process --
   Dynamic Host Configuration Protocol (DHCP).  However, DHCP is not yet
   widely deployed. Further work is needed to improve these tools, get
   them widely deployed, and to integrate them with Domain Name System
   (DNS).

   Multi-level hierarchical routing allows for recapturing additional
   routing information (routing entropy) due to the mismatch between
   addresses and topology at a particular level in the routing hierarchy
   at some higher level in the hierarchy (e.g., at an exchange point
   among providers).  This enables the routing system to contain the
   scope of entities impacted by the mismatch. Containing the scope of
   entities could be an important factor to facilitate graceful
   renumbering.  Further work is needed to develop appropriate
   deployment strategies to put these capabilities in place.

   It is important to emphasize that the requirement to maintain
   topologically significant addresses doesn't need to be applied
   indiscriminately to all the organizations connected to the Internet
   -- hierarchical routing requires that most, but not all addresses be
   topologically significant.  For a large organization it could be
   sufficient if the set of destinations within the organization can be
   represented within the Internet routing system as a small number of
   address prefixes, even if these address prefixes are independent of
   the providers that the organization uses to connect to the Internet
   ("provider-independent" addresses). The volume of routing information
   that a large organization would inject into the Internet routing
   system would be comparable to the (aggregated) routing information
   associated with a large number of small organizations.

   Existence of multiple providers allows a subscriber to be
   simultaneously connected to more than one provider (multi-homed
   subscribers). CIDR offers several alternatives for handling such
   cases. We need to gain more operational experience as well as better
   understand tradeoffs associated with the proposed alternatives.

   An alternative to CIDR address assignment is to assign addresses
   based purely on the geographical location. However, address
   assignment that reflects geographical location of an entity implies
   that either (a) the Internet topology needs to be made sufficiently
   congruent to the geography, or (b) addresses aren't going to be
   topologically significant. In the former case we need to understand

   the driving forces that would make the topology congruent to the
   geography. In the latter case techniques other than hierarchical
   routing need to be developed.

7. Routing Information Sharing

   While ensuring Internet-wide coordination may be more and more
   difficult, as the Internet continues to grow, stability and
   consistency of the Internet-wide routing could significantly benefit
   if the information about routing requirements of various
   organizations could be shared across organizational boundaries. Such
   information could be used in a wide variety of situations ranging
   from troubleshooting to detecting and eliminating conflicting routing
   requirements. The scale of the Internet implies that the information
   should be distributed. Work is currently underway to establish
   depositories of this information (Routing Registries), as well as to
   develop tools that analyze, as well as utilize this information.

8. Summary

   In this section we enumerate some of the issues that the IAB thinks
   should be brought to the attention of the Internet community.

   The following two tasks require the most immediate attention:

      - further work is needed to develop technologies that facilitate
        renumbering

      - further work is needed to investigate feasibility of routing
        information aggregation above the direct (immediate) provider
        level

   The following tasks are viewed as medium term:

      - further work is needed to get a better understanding on the
        relative practical importance of various routing requirements

      - further work is needed to understand of how various routing
        requirements impact scalability of the routing system

      - further work is needed to investigate alternatives to
        hierarchical routing

   Finally, the following tasks are viewed as long term:

      - further work is needed to understand and utilize the benefits of
        routing information sharing

      - further work is needed to understand the implications of virtual
        overlays created via encapsulation

      - further work is needed to understand how different price
        structures influence routing requirements

      - further work is needed to understand how to balance the
        providers' goals and objectives against the public interest of
        Internet-wide connectivity and subscribers' choices.

9. Conclusions

   This document presents some of the issues related to routing in a
   multi-provider Internet. There are no doubt routing-related areas
   that are not covered in this document. For instance, such areas as
   multicast routing, or routing in the presence of mobile hosts, or
   routing in the presence of a large shared media (e.g., ATM) aren't
   discussed here. Further work is needed to understand the implications
   of a multi-provider Internet on these areas.

   The impact of multi-provider Internet goes well beyond just routing,
   and percolates into such areas as network management,
   troubleshooting, and others. Further work is needed to assess the
   implications of multi-provider environment on these areas, as well as
   to understand the interaction among all these areas from a system-
   wide perspective.

10. Acknowledgments

   Many thanks to all the IAB members, and especially to Brian
   Carpenter, Robert Elz, Christian Huitema, Paul Mockapetris, and Lixia
   Zhang for their contributions to this document.

Security Considerations

   Security issues are not discussed in this memo.

Editor's Address

   Yakov Rekhter
   T.J. Watson Research Center IBM Corporation
   P.O. Box 704, Office H3-D40
   Yorktown Heights, NY 10598

   Phone:  +1 914 784 7361
   EMail:  yakov@watson.ibm.com