4.  Functional Guidelines

   Supporting functions and related interfaces over the peering point
   that enable the multicast transport of the application are listed in
   this section.  Critical information parameters that need to be
   exchanged in support of these functions are enumerated, along with
   guidelines as appropriate.  Specific interface functions for
   consideration are as follows.

4.1.  Network Interconnection Transport Guidelines

   The term "network interconnection transport" refers to the
   interconnection points between the two administrative domains.  The
   following is a representative set of attributes that the two
   administrative domains will need to agree on to support multicast
   delivery.

   o  Number of peering points.

   o  Peering point addresses and locations.

   o  Connection type - Dedicated for multicast delivery or shared with
      other services.

   o  Connection mode - Direct connectivity between the two ADs or via
      another ISP.

   o  Peering point protocol support - Multicast protocols that will be
      used for multicast delivery will need to be supported at these
      points.  Examples of such protocols include External BGP (EBGP)
      [RFC4760] peering via MP-BGP (Multiprotocol BGP) SAFI-2 [RFC4760].

   o  Bandwidth allocation - If shared with other services, then there
      needs to be a determination of the share of bandwidth reserved for
      multicast delivery.  See Section 4.1.1 below for more details.

   o  QoS requirements - Delay and/or latency specifications that need
      to be specified in an SLA.

   o  AD roles and responsibilities - The role played by each AD for
      provisioning and maintaining the set of peering points to support
      multicast delivery.

4.1.1.  Bandwidth Management

   Like IP unicast traffic, IP multicast traffic carried across
   non-controlled networks must comply with congestion control
   principles as described in [BCP41] and as explained in detail for UDP
   IP multicast in [BCP145].

   Non-controlled networks (such as the Internet) are networks where
   there is no policy for managing bandwidth other than best effort with
   a fair share of bandwidth under congestion.  As a simplified rule of
   thumb, complying with congestion control principles means reducing
   bandwidth under congestion in a way that is fair to competing
   (typically TCP) flows ("rate adaptive").

   In many instances, multicast content delivery evolves from
   intra-domain deployments where it is handled as a controlled network
   service and does not comply with congestion control principles.  It
   was given a reserved amount of bandwidth and admitted to the network
   so that congestion never occurs.  Therefore, the congestion control
   issue should be given specific attention when evolving to an
   inter-domain peering deployment.

   In the case where end-to-end IP multicast traffic passes across the
   network of two ADs (and their subsidiaries/customers), both ADs must
   agree on a consistent traffic-management policy.  If, for example,
   AD-1 sources non-congestion-aware IP multicast traffic and AD-2
   carries it as best-effort traffic across links shared with other
   Internet traffic (subject to congestion), this will not work: under
   congestion, some amount of that traffic will be dropped, often
   rendering the remaining packets as undecodable garbage clogging up
   the network in AD-2; because this traffic is not congestion aware,
   the loss does not reduce this rate.  Competing traffic will not get
   their fair share under congestion, and EUs will be frustrated by the
   extremely bad quality of both their IP multicast traffic and other
   (e.g., TCP) traffic.  Note that this is not an IP multicast
   technology issue but is solely a transport-layer / application-layer
   issue: the problem would just as likely happen if AD-1 were to send
   non-rate-adaptive unicast traffic -- for example, legacy IPTV
   video-on-demand traffic, which is typically also non-congestion
   aware.  Note that because rate adaptation in IP unicast video is
   commonplace today due to the availability of ABR (Adaptive Bitrate)
   video, it is very unlikely that this will happen in reality with IP
   unicast.

   While the rules for traffic management apply whether IP multicast is
   tunneled or not, the one feature that can make AMT tunnels more
   difficult is the unpredictability of bandwidth requirements across
   underlying links because of the way they can be used: with native IP

   multicast or GRE tunnels, the amount of bandwidth depends on the
   amount of content -- not the number of EUs -- and is therefore easier
   to plan for.  AMT tunnels terminating in the EU/G, on the other hand,
   scale with the number of EUs.  In the vicinity of the AMT relay, they
   can introduce a very large amount of replicated traffic, and it is
   not always feasible to provision enough bandwidth for all possible
   EUs to get the highest quality for all their content during peak
   utilization in such setups -- unless the AMT relays are very close to
   the EU edge.  Therefore, it is also recommended that IP multicast
   rate adaptation be used, even inside controlled networks, when using
   AMT tunnels directly to the EU/G.

   Note that rate-adaptive IP multicast traffic in general does not mean
   that the sender is reducing the bitrate but rather that the EUs that
   experience congestion are joining to a lower-bitrate (S,G) stream of
   the content, similar to ABR streaming over TCP.  Therefore, migration
   from a non-rate-adaptive bitrate to a rate-adaptive bitrate in IP
   multicast will also change the dynamic (S,G) join behavior in the
   network, resulting in potentially higher performance requirements for
   IP multicast protocols (IGMP/PIM), especially on the last hops where
   dynamic changes occur (including AMT gateways/relays): in non-rate-
   adaptive IP multicast, only "channel change" causes state change, but
   in rate-adaptive multicast, congestion also causes state change.

   Even though not fully specified in this document, peerings that rely
   on GRE/AMT tunnels may be across one or more transit ADs instead of
   an exclusive (non-shared, L1/L2) path.  Unless those transit ADs are
   explicitly contracted to provide other than "best effort" transit for
   the tunneled traffic, the tunneled IP multicast traffic must be
   rate adaptive in order to not violate BCP 41 across those
   transit ADs.

4.2.  Routing Aspects and Related Guidelines

   The main objective for multicast delivery routing is to ensure that
   the EU receives the multicast stream from the "most optimal" source
   [INF_ATIS_10], which typically:

   o  Maximizes the multicast portion of the transport and minimizes any
      unicast portion of the delivery, and

   o  Minimizes the overall combined route distance of the network(s).

   This routing objective applies to both native multicast and AMT; the
   actual methodology of the solution will be different for each.
   Regardless, the routing solution is expected to:

   o  Be scalable,

   o  Avoid or minimize new protocol development or modifications, and

   o  Be robust enough to achieve high reliability and to automatically
      adjust to changes and problems in the multicast infrastructure.

   For both native and AMT environments, having a source as close as
   possible to the EU network is most desirable; therefore, in some
   cases, an AD may prefer to have multiple sources near different
   peering points.  However, that is entirely an implementation issue.

4.2.1.  Native Multicast Routing Aspects

   Native multicast simply requires that the administrative domains
   coordinate and advertise the correct source address(es) at their
   network interconnection peering points (i.e., BRs).  An example of
   multicast delivery via a native multicast process across two
   administrative domains is as follows, assuming that the
   interconnecting peering points are also multicast enabled:

   o  Appropriate information is obtained by the EU client, who is a
      subscriber to AD-2 (see Use Case 3.1).  This information is in the
      form of metadata, and it contains instructions directing the EU
      client to launch an appropriate application if necessary, as well
      as additional information for the application about the source
      location and the group (or stream) ID in the form of (S,G) data.
      The "S" portion provides the name or IP address of the source of
      the multicast stream.  The metadata may also contain alternate
      delivery information, such as specifying the unicast address of
      the stream.

   o  The client uses the join message with (S,G) to join the multicast
      stream [RFC4604].  To facilitate this process, the two ADs need to
      do the following:

      *  Advertise the source ID(s) over the peering points.

      *  Exchange such relevant peering point information as capacity
         and utilization.

      *  Implement compatible multicast protocols to ensure proper
         multicast delivery across the peering points.

4.2.2.  GRE Tunnel over Interconnecting Peering Point

   If the interconnecting peering point is not multicast enabled and
   both ADs are multicast enabled, then a simple solution is to
   provision a GRE tunnel between the two ADs; see Use Case 3.2
   (Section 3.2).  The termination points of the tunnel will usually be
   a network engineering decision but generally will be between the BRs
   or even between the AD-2 BR and the AD-1 source (or source access
   router).  The GRE tunnel would allow end-to-end native multicast or
   AMT multicast to traverse the interface.  Coordination and
   advertisement of the source IP are still required.

   The two ADs need to follow the same process as the process described
   in Section 4.2.1 to facilitate multicast delivery across the peering
   points.

4.2.3.  Routing Aspects with AMT Tunnels

   Unlike native multicast (with or without GRE), an AMT multicast
   environment is more complex.  It presents a two-layered problem
   in that there are two criteria that should be simultaneously met:

   o  Find the closest AMT relay to the EU that also has multicast
      connectivity to the content source, and

   o  Minimize the AMT unicast tunnel distance.

   There are essentially two components in the AMT specification:

   AMT relays:  These serve the purpose of tunneling UDP multicast
      traffic to the receivers (i.e., endpoints).  The AMT relay will
      receive the traffic natively from the multicast media source and
      will replicate the stream on behalf of the downstream AMT
      gateways, encapsulating the multicast packets into unicast packets
      and sending them over the tunnel toward the AMT gateways.  In
      addition, the AMT relay may collect various usage and activity
      statistics.  This results in moving the replication point closer
      to the EU and cuts down on traffic across the network.  Thus, the
      linear costs of adding unicast subscribers can be avoided.
      However, unicast replication is still required for each requesting
      endpoint within the unicast-only network.

   AMT gateway:  The gateway will reside on an endpoint; this could be
      any type of IP host, such as a Personal Computer (PC), mobile
      phone, Set-Top Box (STB), or appliances.  The AMT gateway receives
      join and leave requests from the application via an Application
      Programming Interface (API).  In this manner, the gateway allows
      the endpoint to conduct itself as a true multicast endpoint.  The

      AMT gateway will encapsulate AMT messages into UDP packets and
      send them through a tunnel (across the unicast-only
      infrastructure) to the AMT relay.

   The simplest AMT use case (Section 3.3) involves peering points that
   are not multicast enabled between two multicast-enabled ADs.  An
   AMT tunnel is deployed between an AMT relay on the AD-1 side of the
   peering point and an AMT gateway on the AD-2 side of the peering
   point.  One advantage of this arrangement is that the tunnel is
   established on an as-needed basis and need not be a provisioned
   element.  The two ADs can coordinate and advertise special AMT relay
   anycast addresses with, and to, each other.  Alternately, they may
   decide to simply provision relay addresses, though this would not be
   an optimal solution in terms of scalability.

   Use Cases 3.4 and 3.5 describe AMT situations that are more
   complicated, as AD-2 is not multicast enabled in these two cases.
   For these cases, the EU device needs to be able to set up an AMT
   tunnel in the most optimal manner.  There are many methods by which
   relay selection can be done, including the use of DNS-based queries
   and static lookup tables [RFC7450].  The choice of the method is
   implementation dependent and is up to the network operators.
   Comparison of various methods is out of scope for this document and
   is left for further study.

   An illustrative example of a relay selection based on DNS queries as
   part of an anycast IP address process is described here for Use
   Cases 3.4 and 3.5 (Sections 3.4 and 3.5).  Using an anycast
   IP address for AMT relays allows all AMT gateways to find the
   "closest" AMT relay -- the nearest edge of the multicast topology of
   the source.  Note that this is strictly illustrative; the choice of
   the method is up to the network operators.  The basic process is as
   follows:

   o  Appropriate metadata is obtained by the EU client application.
      The metadata contains instructions directing the EU client to an
      ordered list of particular destinations to seek the requested
      stream and, for multicast, specifies the source location and the
      group (or stream) ID in the form of (S,G) data.  The "S" portion
      provides the URI (name or IP address) of the source of the
      multicast stream, and the "G" identifies the particular stream
      originated by that source.  The metadata may also contain
      alternate delivery information such as the address of the unicast
      form of the content to be used -- for example, if the multicast
      stream becomes unavailable.

   o  Using the information from the metadata and, possibly, information
      provisioned directly in the EU client, a DNS query is initiated in
      order to connect the EU client / AMT gateway to an AMT relay.

   o  Query results are obtained and may return an anycast address or a
      specific unicast address of a relay.  Multiple relays will
      typically exist.  The anycast address is a routable
      "pseudo-address" shared among the relays that can gain multicast
      access to the source.

   o  If a specific IP address unique to a relay was not obtained, the
      AMT gateway then sends a message (e.g., the discovery message) to
      the anycast address such that the network is making the routing
      choice of a particular relay, e.g., the relay that is closest to
      the EU.  Details are outside the scope of this document.  See
      [RFC4786].

   o  The contacted AMT relay then returns its specific unicast IP
      address (after which the anycast address is no longer required).
      Variations may exist as well.

   o  The AMT gateway uses that unicast IP address to initiate a
      three-way handshake with the AMT relay.

   o  The AMT gateway provides the (S,G) information to the AMT relay
      (embedded in AMT protocol messages).

   o  The AMT relay receives the (S,G) information and uses it to join
      the appropriate multicast stream, if it has not already subscribed
      to that stream.

   o  The AMT relay encapsulates the multicast stream into the tunnel
      between the relay and the gateway, providing the requested content
      to the EU.

4.2.4.  Public Peering Routing Aspects

   Figure 6 shows an example of a broadcast peering point.

              AD-1a            AD-1b
              BR                BR
               |                 |
             --+-+---------------+-+-- broadcast peering point LAN
                 |                 |
                 BR               BR
                AD-2a            AD-2b

                     Figure 6: Broadcast Peering Point

   A broadcast peering point is an L2 subnet connecting three or more
   ADs.  It is common in IXPs and usually consists of Ethernet
   switch(es) operated by the IXP connecting to BRs operated by the ADs.

   In an example setup domain, AD-2a peers with AD-1a and wants to
   receive IP multicast from it.  Likewise, AD-2b peers with AD-1b and
   wants to receive IP multicast from it.

   Assume that one or more IP multicast (S,G) traffic streams can be
   served by both AD-1a and AD-1b -- for example, because both AD-1a and
   AD-1b contact this content from the same content source.

   In this case, AD-2a and AD-2b can no longer control which upstream
   domain -- AD-1a or AD-1b -- will forward this (S,G) into the LAN.
   The AD-2a BR requests the (S,G) from the AD-1a BR, and the AD-2b BR
   requests the same (S,G) from the AD-1b BR.  To avoid duplicate
   packets, an (S,G) can be forwarded by only one router onto the LAN;
   PIM-SM / PIM-SSM detects requests for duplicate transmissions and
   resolves them via the so-called "assert" protocol operation, which
   results in only one BR forwarding the traffic.  Assume that this is
   the AD-1a BR.  AD-2b will then receive unexpected multicast traffic
   from a provider with whom it does not have a mutual agreement for
   that traffic.  Quality issues in EUs behind AD-2b caused by AD-1a
   will cause a lot of issues related to responsibility and
   troubleshooting.

   In light of these technical issues, we describe, via the following
   options, how IP multicast can be carried across broadcast peering
   point LANs:

   1.  IP multicast is tunneled across the LAN.  Any of the GRE/AMT
       tunneling solutions mentioned in this document are applicable.
       This is the one case where a GRE tunnel between the upstream BR
       (e.g., AD-1a) and downstream BR (e.g., AD-2a) is specifically
       recommended, as opposed to tunneling across uBRs (which are not
       the actual BRs).

   2.  The LAN has only one upstream AD that is sourcing IP multicast,
       and native IP multicast is used.  This is an efficient way to
       distribute the same IP multicast content to multiple downstream
       ADs.  Misbehaving downstream BRs can still disrupt the delivery
       of IP multicast from the upstream BR to other downstream BRs;
       therefore, strict rules must be followed to prohibit such a case.
       The downstream BRs must ensure that they will always consider
       only the upstream BR as a source for multicast traffic: e.g., no
       BGP SAFI-2 peerings between the downstream ADs across the peering
       point LAN, so that the upstream BR is the only possible next hop
       reachable across this LAN.  Also, routing policies can be

       configured to avoid falling back to using SAFI-1 (unicast) routes
       for IP multicast if unicast BGP peering is not limited in the
       same way.

   3.  The LAN has multiple upstream ADs, but they are federated and
       agree on a consistent policy for IP multicast traffic across the
       LAN.  One policy is that each possible source is only announced
       by one upstream BR.  Another policy is that sources are
       redundantly announced (the problematic case mentioned in the
       example in Figure 6 above), but the upstream domains also provide
       mutual operational insight to help with troubleshooting (outside
       the scope of this document).

4.3.  Back-Office Functions - Provisioning and Logging Guidelines

   "Back office" refers to the following:

   o  Servers and content-management systems that support the delivery
      of applications via multicast and interactions between ADs.

   o  Functionality associated with logging, reporting, ordering,
      provisioning, maintenance, service assurance, settlement, etc.

4.3.1.  Provisioning Guidelines

   Resources for basic connectivity between ADs' providers need to be
   provisioned as follows:

   o  Sufficient capacity must be provisioned to support multicast-based
      delivery across ADs.

   o  Sufficient capacity must be provisioned for connectivity between
      all supporting back offices of the ADs as appropriate.  This
      includes activating proper security treatment for these
      back-office connections (gateways, firewalls, etc.) as
      appropriate.

   Provisioning aspects related to multicast-based inter-domain delivery
   are as follows.

   The ability to receive a requested application via multicast is
   triggered via receipt of the necessary metadata.  Hence, this
   metadata must be provided to the EU regarding the multicast URL --
   and unicast fallback if applicable.  AD-2 must enable the delivery of
   this metadata to the EU and provision appropriate resources for this
   purpose.

   It is assumed that native multicast functionality is available across
   many ISP backbones, peering points, and access networks.  If,
   however, native multicast is not an option (Use Cases 3.4 and 3.5),
   then:

   o  The EU must have a multicast client to use AMT multicast obtained
      from either (1) the application source (per agreement with AD-1)
      or (2) AD-1 or AD-2 (if delegated by the application source).

   o  If provided by AD-1 or AD-2, then the EU could be redirected to a
      client download site.  (Note: This could be an application source
      site.)  If provided by the application source, then this source
      would have to coordinate with AD-1 to ensure that the proper
      client is provided (assuming multiple possible clients).

   o  Where AMT gateways support different application sets, all AD-2
      AMT relays need to be provisioned with all source and group
      addresses for streams it is allowed to join.

   o  DNS across each AD must be provisioned to enable a client gateway
      to locate the optimal AMT relay (i.e., longest multicast path and
      shortest unicast tunnel) with connectivity to the content's
      multicast source.

   Provisioning aspects related to operations and customer care are as
   follows.

   It is assumed that each AD provider will provision operations and
   customer care access to their own systems.

   AD-1's operations and customer care functions must be able to see
   enough of what is happening in AD-2's network or in the service
   provided by AD-2 to verify their mutual goals and operations, e.g.,
   to know how the EUs are being served.  This can be done in two ways:

   o  Automated interfaces are built between AD-1 and AD-2 such that
      operations and customer care continue using their own systems.
      This requires coordination between the two ADs, with appropriate
      provisioning of necessary resources.

   o  AD-1's operations and customer care personnel are provided direct
      access to AD-2's systems.  In this scenario, additional
      provisioning in these systems will be needed to provide necessary
      access.  The two ADs must agree on additional provisioning to
      support this option.

4.3.2.  Inter-domain Authentication Guidelines

   All interactions between pairs of ADs can be discovered and/or
   associated with the account(s) utilized for delivered applications.
   Supporting guidelines are as follows:

   o  A unique identifier is recommended to designate each master
      account.

   o  AD-2 is expected to set up "accounts" (a logical facility
      generally protected by credentials such as login passwords) for
      use by AD-1.  Multiple accounts, and multiple types or partitions
      of accounts, can apply, e.g., customer accounts, security
      accounts.

   The reason to specifically mention the need for AD-1 to initiate
   interactions with AD-2 (and use some account for that), as opposed to
   the opposite, is based on the recommended workflow initiated by
   customers (see Section 4.4): the customer contacts the content
   source, which is part of AD-1.  Consequently, if AD-1 sees the need
   to escalate the issue to AD-2, it will interact with AD-2 using the
   aforementioned guidelines.

4.3.3.  Log-Management Guidelines

   Successful delivery (in terms of user experience) of applications or
   content via multicast between pairs of interconnecting ADs can be
   improved through the ability to exchange appropriate logs for various
   workflows -- troubleshooting, accounting and billing, optimization of
   traffic and content transmission, optimization of content and
   application development, and so on.

   Specifically, AD-1 take over primary responsibility for customer
   experience on behalf of the content source, with support from AD-2 as
   needed.  The application/content owner is the only participant who
   has, and needs, full insight into the application level and can map
   the customer application experience to the network traffic flows --
   which, with the help of AD-2 or logs from AD-2, it can then analyze
   and interpret.

   The main difference between unicast delivery and multicast delivery
   is that the content source can infer a lot more about downstream
   network problems from a unicast stream than from a multicast stream:
   the multicast stream is not per EU, except after the last
   replication, which is in most cases not in AD-1.  Logs from the
   application, including the receiver side at the EU, can provide
   insight but cannot help to fully isolate network problems because of

   the IP multicast per-application operational state built across AD-1
   and AD-2 (aka the (S,G) state and any other operational-state
   features, such as Diffserv QoS).

   See Section 7 for more discussion regarding the privacy
   considerations of the model described here.

   Different types of logs are known to help support operations in AD-1
   when provided by AD-2.  This could be done as part of AD-1/AD-2
   contracts.  Note that except for implied multicast-specific elements,
   the options listed here are not unique or novel for IP multicast, but
   they are more important for services novel to the operators than for
   operationally well-established services (such as unicast).  We
   therefore detail them as follows:

   o  Usage information logs at an aggregate level.

   o  Usage failure instances at an aggregate level.

   o  Grouped or sequenced application access: performance, behavior,
      and failure at an aggregate level to support potential
      application-provider-driven strategies.  Examples of aggregate
      levels include grouped video clips, web pages, and software-
      download sets.

   o  Security logs, aggregated or summarized according to agreement
      (with additional detail potentially provided during security
      events, by agreement).

   o  Access logs (EU), when needed for troubleshooting.

   o  Application logs ("What is the application doing?"), when needed
      for shared troubleshooting.

   o  Syslogs (network management), when needed for shared
      troubleshooting.

   The two ADs may supply additional security logs to each other, as
   agreed upon in contract(s).  Examples include the following:

   o  Information related to general security-relevant activity, which
      may be of use from a protection or response perspective: types and
      counts of attacks detected, related source information, related
      target information, etc.

   o  Aggregated or summarized logs according to agreement (with
      additional detail potentially provided during security events, by
      agreement).

4.4.  Operations - Service Performance and Monitoring Guidelines

   "Service performance" refers to monitoring metrics related to
   multicast delivery via probes.  The focus is on the service provided
   by AD-2 to AD-1 on behalf of all multicast application sources
   (metrics may be specified for SLA use or otherwise).  Associated
   guidelines are as follows:

   o  Both ADs are expected to monitor, collect, and analyze service
      performance metrics for multicast applications.  AD-2 provides
      relevant performance information to AD-1; this enables AD-1 to
      create an end-to-end performance view on behalf of the multicast
      application source.

   o  Both ADs are expected to agree on the types of probes to be used
      to monitor multicast delivery performance.  For example, AD-2 may
      permit AD-1's probes to be utilized in the AD-2 multicast service
      footprint.  Alternately, AD-2 may deploy its own probes and relay
      performance information back to AD-1.

   "Service monitoring" generally refers to a service (as a whole)
   provided on behalf of a particular multicast application source
   provider.  It thus involves complaints from EUs when service problems
   occur.  EUs direct their complaints to the source provider; the
   source provider in turn submits these complaints to AD-1.  The
   responsibility for service delivery lies with AD-1; as such, AD-1
   will need to determine where the service problem is occurring -- in
   its own network or in AD-2.  It is expected that each AD will have
   tools to monitor multicast service status in its own network.

   o  Both ADs will determine how best to deploy multicast service
      monitoring tools.  Typically, each AD will deploy its own set of
      monitoring tools, in which case both ADs are expected to inform
      each other when multicast delivery problems are detected.

   o  AD-2 may experience some problems in its network.  For example,
      for the AMT use cases (Sections 3.3, 3.4, and 3.5), one or more
      AMT relays may be experiencing difficulties.  AD-2 may be able to
      fix the problem by rerouting the multicast streams via alternate
      AMT relays.  If the fix is not successful and multicast service
      delivery degrades, then AD-2 needs to report the issue to AD-1.

   o  When a problem notification is received from a multicast
      application source, AD-1 determines whether the cause of the
      problem is within its own network or within AD-2.  If the cause is
      within AD-2, then AD-1 supplies all necessary information to AD-2.
      Examples of supporting information include the following:

      *  Kind(s) of problem(s).

      *  Starting point and duration of problem(s).

      *  Conditions in which one or more problems occur.

      *  IP address blocks of affected users.

      *  ISPs of affected users.

      *  Type of access, e.g., mobile versus desktop.

      *  Network locations of affected EUs.

   o  Both ADs conduct some form of root-cause analysis for multicast
      service delivery problems.  Examples of various factors for
      consideration include:

      *  Verification that the service configuration matches the product
         features.

      *  Correlation and consolidation of the various customer problems
         and resource troubles into a single root-service problem.

      *  Prioritization of currently open service problems, giving
         consideration to problem impacts, SLAs, etc.

      *  Conducting service tests, including tests performed once or a
         series of tests over a period of time.

      *  Analysis of test results.

      *  Analysis of relevant network fault or performance data.

      *  Analysis of the problem information provided by the customer.

   o  Once the cause of the problem has been determined and the problem
      has been fixed, both ADs need to work jointly to verify and
      validate the success of the fix.

4.5.  Client Reliability Models / Service Assurance Guidelines

   There are multiple options for instituting reliability architectures.
   Most are at the application level.  Both ADs should work these
   options out per their contract or agreement and also with the
   multicast application source providers.

   Network reliability can also be enhanced by the two ADs if they
   provision alternate delivery mechanisms via unicast means.

4.6.  Application Accounting Guidelines

   Application-level accounting needs to be handled differently in the
   application than in IP unicast, because the source side does not
   directly deliver packets to individual receivers.  Instead, this
   needs to be signaled back by the receiver to the source.

   For network transport diagnostics, AD-1 and AD-2 should have
   mechanisms in place to ensure proper accounting for the volume of
   bytes delivered through the peering point and, separately, the number
   of bytes delivered to EUs.