4.  Background

   When "Framework for IP Performance Metrics" [RFC2330] was published
   in 1998, sound Bulk Transport Capacity (BTC) measurement was known to
   be well beyond our capabilities.  Even when "A Framework for Defining
   Empirical Bulk Transfer Capacity Metrics" [RFC3148] was published, we
   knew that we didn't really understand the problem.  Now, in
   hindsight, we understand why assessing BTC is such a difficult
   problem:

   o  TCP is a control system with circular dependencies -- everything
      affects performance, including components that are explicitly not
      part of the test (for example, the host processing power is not
      in-scope of path performance tests).

   o  Congestion control is a dynamic equilibrium process, similar to
      processes observed in chemistry and other fields.  The network and
      transport protocols find an operating point that balances opposing
      forces: the transport protocol pushing harder (raising the data
      rate and/or window) while the network pushes back (raising packet
      loss ratio, RTT, and/or ECN CE marks).  By design, TCP congestion
      control keeps raising the data rate until the network gives some
      indication that its capacity has been exceeded by dropping packets
      or adding ECN CE marks.  If a TCP sender accurately fills a path
      to its IP capacity (e.g., the bottleneck is 100% utilized), then
      packet losses and ECN CE marks are mostly determined by the TCP
      sender and how aggressively it seeks additional capacity; they are
      not determined by the network itself, because the network must
      send exactly the signals that TCP needs to set its rate.

   o  TCP's ability to compensate for network impairments (such as loss,
      delay, and delay variation, outside of those caused by TCP itself)
      is directly proportional to the number of send-ACK round-trip
      exchanges per second (i.e., inversely proportional to the RTT).
      As a consequence, an impaired subpath may pass a short RTT local
      test even though it fails when the subpath is extended by an
      effectively perfect network to some larger RTT.

   o  TCP has an extreme form of the Observer Effect (colloquially known
      as the "Heisenberg Effect").  Measurement and cross traffic
      interact in unknown and ill-defined ways.  The situation is
      actually worse than the traditional physics problem where you can
      at least estimate bounds on the relative momentum of the
      measurement and measured particles.  In general, for network
      measurement, you cannot determine even the order of magnitude of
      the effect.  It is possible to construct measurement scenarios
      where the measurement traffic starves real user traffic, yielding
      an overly inflated measurement.  The inverse is also possible: the
      user traffic can fill the network, such that the measurement
      traffic detects only minimal available capacity.  In general, you
      cannot determine which scenario might be in effect, so you cannot
      gauge the relative magnitude of the uncertainty introduced by
      interactions with other network traffic.

   o  As a consequence of the properties listed above, it is difficult,
      if not impossible, for two independent implementations (hardware
      or software) of TCP congestion control to produce equivalent
      performance results [RFC6576] under the same network conditions.

   These properties are a consequence of the dynamic equilibrium
   behavior intrinsic to how all throughput-maximizing protocols
   interact with the Internet.  These protocols rely on control systems
   based on estimated network metrics to regulate the quantity of data
   to send into the network.  The packet-sending characteristics in turn
   alter the network properties estimated by the control system metrics,
   such that there are circular dependencies between every transmission
   characteristic and every estimated metric.  Since some of these
   dependencies are nonlinear, the entire system is nonlinear, and any
   change anywhere causes a difficult-to-predict response in network
   metrics.  As a consequence, Bulk Transport Capacity metrics have not
   fulfilled the analytic framework envisioned in [RFC2330].

   Model-Based Metrics overcome these problems by making the measurement
   system open loop: the packet transfer statistics (akin to the network
   estimators) do not affect the traffic or traffic patterns (bursts),
   which are computed on the basis of the Target Transport Performance.
   A path or subpath meeting the Target Transfer Performance

   requirements would exhibit packet transfer statistics and estimated
   metrics that would not cause the control system to slow the traffic
   below the Target Data Rate.

4.1.  TCP Properties

   TCP and other self-clocked protocols (e.g., the Stream Control
   Transmission Protocol (SCTP)) carry the vast majority of all Internet
   data.  Their dominant bulk data transport behavior is to have an
   approximately fixed quantity of data and acknowledgments (ACKs)
   circulating in the network.  The data receiver reports arriving data
   by returning ACKs to the data sender, and the data sender typically
   responds by sending approximately the same quantity of data back into
   the network.  The total quantity of data plus the data represented by
   ACKs circulating in the network is referred to as the "window".  The
   mandatory congestion control algorithms incrementally adjust the
   window by sending slightly more or less data in response to each ACK.
   The fundamentally important property of this system is that it is
   self-clocked: the data transmissions are a reflection of the ACKs
   that were delivered by the network, and the ACKs are a reflection of
   the data arriving from the network.

   A number of protocol features cause bursts of data, even in idealized
   networks that can be modeled as simple queuing systems.

   During slowstart, the IP rate is doubled on each RTT by sending twice
   as much data as was delivered to the receiver during the prior RTT.
   Each returning ACK causes the sender to transmit twice the data the
   ACK reported arriving at the receiver.  For slowstart to be able to
   fill the pipe, the network must be able to tolerate slowstart bursts
   up to the full pipe size inflated by the anticipated window reduction
   on the first loss or ECN CE mark.  For example, with classic Reno
   congestion control, an optimal slowstart has to end with a burst that
   is twice the bottleneck rate for one RTT in duration.  This burst
   causes a queue that is equal to the pipe size (i.e., the window is
   twice the pipe size), so when the window is halved in response to the
   first packet loss, the new window will be the pipe size.

   Note that if the bottleneck IP rate is less than half of the capacity
   of the front path (which is almost always the case), the slowstart
   bursts will not by themselves cause significant queues anywhere else
   along the front path; they primarily exercise the queue at the
   dominant bottleneck.

   Several common efficiency algorithms also cause bursts.  The self-
   clock is typically applied to groups of packets: the receiver's
   delayed ACK algorithm generally sends only one ACK per two data
   segments.  Furthermore, modern senders use TCP segmentation offload

   (TSO) to reduce CPU overhead.  The sender's software stack builds
   super-sized TCP segments that the TSO hardware splits into MTU-sized
   segments on the wire.  The net effect of TSO, delayed ACK, and other
   efficiency algorithms is to send bursts of segments at full sender
   interface rate.

   Note that these efficiency algorithms are almost always in effect,
   including during slowstart, such that slowstart typically has a two-
   level burst structure.  Section 6.1 describes slowstart in more
   detail.

   Additional sources of bursts include TCP's initial window [RFC6928],
   application pauses, channel allocation mechanisms, and network
   devices that schedule ACKs.  Appendix B describes these last two
   items.  If the application pauses (e.g., stops reading or writing
   data) for some fraction of an RTT, many TCP implementations catch up
   to their earlier window size by sending a burst of data at the full
   sender interface rate.  To fill a network with a realistic
   application, the network has to be able to tolerate sender interface
   rate bursts large enough to restore the prior window following
   application pauses.

   Although the sender interface rate bursts are typically smaller than
   the last burst of a slowstart, they are at a higher IP rate so they
   potentially exercise queues at arbitrary points along the front path
   from the data sender up to and including the queue at the dominant
   bottleneck.  It is known that these bursts can hurt network
   performance, especially in conjunction with other queue pressure;
   however, we are not aware of any models for estimating the impact or
   prescribing limits on the size or frequency of sender rate bursts.

   In conclusion, to verify that a path can meet a Target Transport
   Performance, it is necessary to independently confirm that the path
   can tolerate bursts at the scales that can be caused by the above
   mechanisms.  Three cases are believed to be sufficient:

   o  Two-level slowstart bursts sufficient to get connections started
      properly.

   o  Ubiquitous sender interface rate bursts caused by efficiency
      algorithms.  We assume four packet bursts to be the most common
      case, since it matches the effects of delayed ACK during
      slowstart.  These bursts should be assumed not to significantly
      affect packet transfer statistics.

   o  Infrequent sender interface rate bursts that are the maximum of
      the full target_window_size and the initial window size (10
      segments in [RFC6928]).  The target_run_length may be derated for
      these large fast bursts.

   If a subpath can meet the required packet loss ratio for bursts at
   all of these scales, then it has sufficient buffering at all
   potential bottlenecks to tolerate any of the bursts that are likely
   introduced by TCP or other transport protocols.

4.2.  Diagnostic Approach

   A complete path is expected to be able to attain a specified Bulk
   Transport Capacity if the path's RTT is equal to or smaller than the
   Target RTT, the path's MTU is equal to or larger than the Target MTU,
   and all of the following conditions are met:

   1.  The IP capacity is above the Target Data Rate by a sufficient
       margin to cover all TCP/IP overheads.  This can be confirmed by
       the tests described in Section 8.1 or any number of IP capacity
       tests adapted to implement MBM.

   2.  The observed packet transfer statistics are better than required
       by a suitable TCP performance model (e.g., fewer packet losses or
       ECN CE marks).  See Section 8.1 or any number of low- or fixed-
       rate packet loss tests outside of MBM.

   3.  There is sufficient buffering at the dominant bottleneck to
       absorb a slowstart burst large enough to get the flow out of
       slowstart at a suitable window size.  See Section 8.3.

   4.  There is sufficient buffering in the front path to absorb and
       smooth sender interface rate bursts at all scales that are likely
       to be generated by the application, any channel arbitration in
       the ACK path, or any other mechanisms.  See Section 8.4.

   5.  When there is a slowly rising standing queue at the bottleneck,
       then the onset of packet loss has to be at an appropriate point
       (in time or in queue depth) and has to be progressive, for
       example, by use of Active Queue Management [RFC7567].  See
       Section 8.2.

   6.  When there is a standing queue at a bottleneck for a shared media
       subpath (e.g., a half-duplex link), there must be a suitable
       bound on the interaction between ACKs and data, for example, due
       to the channel arbitration mechanism.  See Section 8.2.4.

   Note that conditions 1 through 4 require capacity tests for
   validation and thus may need to be monitored on an ongoing basis.
   Conditions 5 and 6 require engineering tests, which are best
   performed in controlled environments (e.g., bench tests).  They won't
   generally fail due to load but may fail in the field (e.g., due to
   configuration errors, etc.) and thus should be spot checked.

   A tool that can perform many of the tests is available from
   [MBMSource].

4.3.  New Requirements Relative to RFC 2330

   Model-Based Metrics are designed to fulfill some additional
   requirements that were not recognized at the time RFC 2330 [RFC2330]
   was published.  These missing requirements may have significantly
   contributed to policy difficulties in the IP measurement space.  Some
   additional requirements are:

   o  IP metrics must be actionable by the ISP -- they have to be
      interpreted in terms of behaviors or properties at the IP or lower
      layers that an ISP can test, repair, and verify.

   o  Metrics should be spatially composable, such that measures of
      concatenated paths should be predictable from subpaths.

   o  Metrics must be vantage point invariant over a significant range
      of measurement point choices, including off-path measurement
      points.  The only requirements for Measurement Point (MP)
      selection should be that the RTT between the MPs is below some
      reasonable bound and that the effects of the "test leads"
      connecting MPs to the subpath under test can be calibrated out of
      the measurements.  The latter might be accomplished if the test
      leads are effectively ideal or their properties can be deducted
      from the measurements between the MPs.  While many tests require
      that the test leads have at least as much IP capacity as the
      subpath under test, some do not, for example, the Background
      Packet Transfer Statistics Tests described in Section 8.1.3.

   o  Metric measurements should be repeatable by multiple parties with
      no specialized access to MPs or diagnostic infrastructure.  It
      should be possible for different parties to make the same
      measurement and observe the same results.  In particular, it is
      important that both a consumer (or the consumer's delegate) and
      ISP be able to perform the same measurement and get the same
      result.  Note that vantage independence is key to meeting this
      requirement.

5.  Common Models and Parameters

5.1.  Target End-to-End Parameters

   The target end-to-end parameters are the Target Data Rate, Target
   RTT, and Target MTU as defined in Section 3.  These parameters are
   determined by the needs of the application or the ultimate end user
   and the complete Internet path over which the application is expected
   to operate.  The target parameters are in units that make sense to
   layers above the TCP layer: payload bytes delivered to the
   application.  They exclude overheads associated with TCP and IP
   headers, retransmits and other protocols (e.g., DNS).  Note that
   IP-based network services include TCP headers and retransmissions as
   part of delivered payload; this difference (header_overhead) is
   recognized in calculations below.

   Other end-to-end parameters defined in Section 3 include the
   effective bottleneck data rate, the sender interface data rate, and
   the TCP and IP header sizes.

   The target_data_rate must be smaller than all subpath IP capacities
   by enough headroom to carry the transport protocol overhead,
   explicitly including retransmissions and an allowance for
   fluctuations in TCP's actual data rate.  Specifying a
   target_data_rate with insufficient headroom is likely to result in
   brittle measurements that have little predictive value.

   Note that the target parameters can be specified for a hypothetical
   path (for example, to construct TIDS designed for bench testing in
   the absence of a real application) or for a live in situ test of
   production infrastructure.

   The number of concurrent connections is explicitly not a parameter in
   this model.  If a subpath requires multiple connections in order to
   meet the specified performance, that must be stated explicitly, and
   the procedure described in Section 6.4 applies.

5.2.  Common Model Calculations

   The Target Transport Performance is used to derive the
   target_window_size and the reference target_run_length.

   The target_window_size is the average window size in packets needed
   to meet the target_rate, for the specified target_RTT and target_MTU.
   To calculate target_window_size:

   target_window_size = ceiling(target_rate * target_RTT / (target_MTU -
   header_overhead))

   The target_run_length is an estimate of the minimum required number
   of unmarked packets that must be delivered between losses or ECN CE
   marks, as computed by a mathematical model of TCP congestion control.
   The derivation here is parallel to the derivation in [MSMO97] and, by
   design, is quite conservative.

   The reference target_run_length is derived as follows.  Assume the
   subpath_IP_capacity is infinitesimally larger than the
   target_data_rate plus the required header_overhead.  Then,
   target_window_size also predicts the onset of queuing.  A larger
   window will cause a standing queue at the bottleneck.

   Assume the transport protocol is using standard Reno-style Additive
   Increase Multiplicative Decrease (AIMD) congestion control [RFC5681]
   (but not Appropriate Byte Counting [RFC3465]) and the receiver is
   using standard delayed ACKs.  Reno increases the window by one packet
   every pipe size worth of ACKs.  With delayed ACKs, this takes two
   RTTs per increase.  To exactly fill the pipe, the spacing of losses
   must be no closer than when the peak of the AIMD sawtooth reached
   exactly twice the target_window_size.  Otherwise, the multiplicative
   window reduction triggered by the loss would cause the network to be
   underfilled.  Per [MSMO97] the number of packets between losses must
   be the area under the AIMD sawtooth.  They must be no more frequent
   than every 1 in ((3/2)*target_window_size)*(2*target_window_size)
   packets, which simplifies to:

   target_run_length = 3*(target_window_size^2)

   Note that this calculation is very conservative and is based on a
   number of assumptions that may not apply.  Appendix A discusses these
   assumptions and provides some alternative models.  If a different
   model is used, an FSTIDS must document the actual method for
   computing target_run_length and the ratio between alternate
   target_run_length and the reference target_run_length calculated
   above, along with a discussion of the rationale for the underlying
   assumptions.

   Most of the individual parameters for the tests in Section 8 are
   derived from target_window_size and target_run_length.

5.3.  Parameter Derating

   Since some aspects of the models are very conservative, the MBM
   framework permits some latitude in derating test parameters.  Rather
   than trying to formalize more complicated models, we permit some test
   parameters to be relaxed as long as they meet some additional
   procedural constraints:

   o  The FSTIDS must document and justify the actual method used to
      compute the derated metric parameters.

   o  The validation procedures described in Section 10 must be used to
      demonstrate the feasibility of meeting the Target Transport
      Performance with infrastructure that just barely passes the
      derated tests.

   o  The validation process for an FSTIDS itself must be documented in
      such a way that other researchers can duplicate the validation
      experiments.

   Except as noted, all tests below assume no derating.  Tests for which
   there is not currently a well-established model for the required
   parameters explicitly include derating as a way to indicate
   flexibility in the parameters.

5.4.  Test Preconditions

   Many tests have preconditions that are required to assure their
   validity.  Examples include the presence or non-presence of cross
   traffic on specific subpaths; negotiating ECN; and a test stream
   preamble of appropriate length to achieve stable access to network
   resources in the presence of reactive network elements (as defined in
   Section 1.1 of [RFC7312]).  If preconditions are not properly
   satisfied for some reason, the tests should be considered to be
   inconclusive.  In general, it is useful to preserve diagnostic
   information as to why the preconditions were not met and any test
   data that was collected even if it is not useful for the intended
   test.  Such diagnostic information and partial test data may be
   useful for improving the test or test procedures themselves.

   It is important to preserve the record that a test was scheduled;
   otherwise, precondition enforcement mechanisms can introduce sampling
   bias.  For example, canceling tests due to cross traffic on
   subscriber access links might introduce sampling bias in tests of the
   rest of the network by reducing the number of tests during peak
   network load.

   Test preconditions and failure actions must be specified in an
   FSTIDS.

6.  Generating Test Streams

   Many important properties of Model-Based Metrics, such as vantage
   independence, are a consequence of using test streams that have
   temporal structures that mimic TCP or other transport protocols
   running over a complete path.  As described in Section 4.1, self-

   clocked protocols naturally have burst structures related to the RTT
   and pipe size of the complete path.  These bursts naturally get
   larger (contain more packets) as either the Target RTT or Target Data
   Rate get larger or the Target MTU gets smaller.  An implication of
   these relationships is that test streams generated by running self-
   clocked protocols over short subpaths may not adequately exercise the
   queuing at any bottleneck to determine if the subpath can support the
   full Target Transport Performance over the complete path.

   Failing to authentically mimic TCP's temporal structure is part of
   the reason why simple performance tools such as iPerf, netperf, nc,
   etc., have the reputation for yielding false pass results over short
   test paths, even when a subpath has a flaw.

   The definitions in Section 3 are sufficient for most test streams.
   We describe the slowstart and standing queue test streams in more
   detail.

   In conventional measurement practice, stochastic processes are used
   to eliminate many unintended correlations and sample biases.
   However, MBM tests are designed to explicitly mimic temporal
   correlations caused by network or protocol elements themselves.  Some
   portions of these systems, such as traffic arrival (e.g., test
   scheduling), are naturally stochastic.  Other behaviors, such as
   back-to-back packet transmissions, are dominated by implementation-
   specific deterministic effects.  Although these behaviors always
   contain non-deterministic elements and might be modeled
   stochastically, these details typically do not contribute
   significantly to the overall system behavior.  Furthermore, it is
   known that real protocols are subject to failures caused by network
   property estimators suffering from bias due to correlation in their
   own traffic.  For example, TCP's RTT estimator used to determine the
   Retransmit Timeout (RTO), can be fooled by periodic cross traffic or
   start-stop applications.  For these reasons, many details of the test
   streams are specified deterministically.

   It may prove useful to introduce fine-grained noise sources into the
   models used for generating test streams in an update of Model-Based
   Metrics, but the complexity is not warranted at the time this
   document was written.

6.1.  Mimicking Slowstart

   TCP slowstart has a two-level burst structure as shown in Figure 2.
   The fine time structure is caused by efficiency algorithms that
   deliberately batch work (CPU, channel allocation, etc.) to better
   amortize certain network and host overheads.  ACKs passing through
   the return path typically cause the sender to transmit small bursts

   of data at the full sender interface rate.  For example, TCP
   Segmentation Offload (TSO) and Delayed Acknowledgment both contribute
   to this effect.  During slowstart, these bursts are at the same
   headway as the returning ACKs but are typically twice as large (e.g.,
   have twice as much data) as the ACK reported was delivered to the
   receiver.  Due to variations in delayed ACK and algorithms such as
   Appropriate Byte Counting [RFC3465], different pairs of senders and
   receivers produce slightly different burst patterns.  Without loss of
   generality, we assume each ACK causes four packet sender interface
   rate bursts at an average headway equal to the ACK headway; this
   corresponds to sending at an average rate equal to twice the
   effective bottleneck IP rate.  Each slowstart burst consists of a
   series of four packet sender interface rate bursts such that the
   total number of packets is the current window size (as of the last
   packet in the burst).

   The coarse time structure is due to each RTT being a reflection of
   the prior RTT.  For real transport protocols, each slowstart burst is
   twice as large (twice the window) as the previous burst but is spread
   out in time by the network bottleneck, such that each successive RTT
   exhibits the same effective bottleneck IP rate.  The slowstart phase
   ends on the first lost packet or ECN mark, which is intended to
   happen after successive slowstart bursts merge in time: the next
   burst starts before the bottleneck queue is fully drained and the
   prior burst is complete.

   For the diagnostic tests described below, we preserve the fine time
   structure but manipulate the coarse structure of the slowstart bursts
   (burst size and headway) to measure the ability of the dominant
   bottleneck to absorb and smooth slowstart bursts.

   Note that a stream of repeated slowstart bursts has three different
   average rates, depending on the averaging time interval.  At the
   finest timescale (a few packet times at the sender interface), the
   peak of the average IP rate is the same as the sender interface rate;
   at a medium timescale (a few ACK times at the dominant bottleneck),
   the peak of the average IP rate is twice the implied bottleneck IP
   capacity; and at timescales longer than the target_RTT and when the
   burst size is equal to the target_window_size, the average rate is
   equal to the target_data_rate.  This pattern corresponds to repeating
   the last RTT of TCP slowstart when delayed ACK and sender-side byte
   counting are present but without the limits specified in Appropriate
   Byte Counting [RFC3465].

   time ==>    ( - equals one packet)

   Fine time structure of the packet stream:

   ----  ----  ----  ----  ----

   |<>| sender interface rate bursts (typically 3 or 4 packets)
   |<===>| burst headway (from the ACK headway)

   \____repeating sender______/
          rate bursts

   Coarse (RTT-level) time structure of the packet stream:

   ----  ----  ----  ----  ----                     ----  ---- ...

   |<========================>| slowstart burst size (from the window)
   |<==============================================>| slowstart headway
                                                       (from the RTT)
   \__________________________/                     \_________ ...
       one slowstart burst                     Repeated slowstart bursts

               Figure 2: Multiple Levels of Slowstart Bursts

6.2.  Constant Window Pseudo CBR

   Pseudo constant bit rate (CBR) is implemented by running a standard
   self-clocked protocol such as TCP with a fixed window size.  If that
   window size is test_window, the data rate will be slightly above the
   target_rate.

   Since the test_window is constrained to be an integer number of
   packets, for small RTTs or low data rates, there may not be
   sufficiently precise control over the data rate.  Rounding the
   test_window up (as defined above) is likely to result in data rates
   that are higher than the target rate, but reducing the window by one
   packet may result in data rates that are too small.  Also, cross
   traffic potentially raises the RTT, implicitly reducing the rate.
   Cross traffic that raises the RTT nearly always makes the test more
   strenuous (i.e., more demanding for the network path).

   Note that Constant Window Pseudo CBR (and Scanned Window Pseudo CBR
   in the next section) both rely on a self-clock that is at least
   partially derived from the properties of the subnet under test.  This
   introduces the possibility that the subnet under test exhibits
   behaviors such as extreme RTT fluctuations that prevent these
   algorithms from accurately controlling data rates.

   An FSTIDS specifying a Constant Window Pseudo CBR test must
   explicitly indicate under what conditions errors in the data rate
   cause tests to be inconclusive.  Conventional paced measurement
   traffic may be more appropriate for these environments.

6.3.  Scanned Window Pseudo CBR

   Scanned Window Pseudo CBR is similar to the Constant Window Pseudo
   CBR described above, except the window is scanned across a range of
   sizes designed to include two key events: the onset of queuing and
   the onset of packet loss or ECN CE marks.  The window is scanned by
   incrementing it by one packet every 2*target_window_size delivered
   packets.  This mimics the additive increase phase of standard Reno
   TCP congestion avoidance when delayed ACKs are in effect.  Normally,
   the window increases are separated by intervals slightly longer than
   twice the target_RTT.

   There are two ways to implement this test: 1) applying a window clamp
   to standard congestion control in a standard protocol such as TCP and
   2) stiffening a non-standard transport protocol.  When standard
   congestion control is in effect, any losses or ECN CE marks cause the
   transport to revert to a window smaller than the clamp, such that the
   scanning clamp loses control of the window size.  The NPAD (Network
   Path and Application Diagnostics) pathdiag tool is an example of this
   class of algorithms [Pathdiag].

   Alternatively, a non-standard congestion control algorithm can
   respond to losses by transmitting extra data, such that it maintains
   the specified window size independent of losses or ECN CE marks.
   Such a stiffened transport explicitly violates mandatory Internet
   congestion control [RFC5681] and is not suitable for in situ testing.
   It is only appropriate for engineering testing under laboratory
   conditions.  The Windowed Ping tool implements such a test [WPING].
   This tool has been updated (see [mpingSource]).

   The test procedures in Section 8.2 describe how to the partition the
   scans into regions and how to interpret the results.

6.4.  Concurrent or Channelized Testing

   The procedures described in this document are only directly
   applicable to single-stream measurement, e.g., one TCP connection or
   measurement stream.  In an ideal world, we would disallow all
   performance claims based on multiple concurrent streams, but this is
   not practical due to at least two issues.  First, many very high-rate
   link technologies are channelized and at last partially pin the flow-
   to-channel mapping to minimize packet reordering within flows.

   Second, TCP itself has scaling limits.  Although the former problem
   might be overcome through different design decisions, the latter
   problem is more deeply rooted.

   All congestion control algorithms that are philosophically aligned
   with [RFC5681] (e.g., claim some level of TCP compatibility,
   friendliness, or fairness) have scaling limits; that is, as a long
   fat network (LFN) with a fixed RTT and MTU gets faster, these
   congestion control algorithms get less accurate and, as a
   consequence, have difficulty filling the network [CCscaling].  These
   properties are a consequence of the original Reno AIMD congestion
   control design and the requirement in [RFC5681] that all transport
   protocols have similar responses to congestion.

   There are a number of reasons to want to specify performance in terms
   of multiple concurrent flows; however, this approach is not
   recommended for data rates below several megabits per second, which
   can be attained with run lengths under 10000 packets on many paths.
   Since the required run length is proportional to the square of the
   data rate, at higher rates, the run lengths can be unreasonably
   large, and multiple flows might be the only feasible approach.

   If multiple flows are deemed necessary to meet aggregate performance
   targets, then this must be stated both in the design of the TIDS and
   in any claims about network performance.  The IP diagnostic tests
   must be performed concurrently with the specified number of
   connections.  For the tests that use bursty test streams, the bursts
   should be synchronized across streams unless there is a priori
   knowledge that the applications have some explicit mechanism to
   stagger their own bursts.  In the absence of an explicit mechanism to
   stagger bursts, many network and application artifacts will sometimes
   implicitly synchronize bursts.  A test that does not control burst
   synchronization may be prone to false pass results for some
   applications.

7.  Interpreting the Results

7.1.  Test Outcomes

   To perform an exhaustive test of a complete network path, each test
   of the TIDS is applied to each subpath of the complete path.  If any
   subpath fails any test, then a standard transport protocol running
   over the complete path can also be expected to fail to attain the
   Target Transport Performance under some conditions.

   In addition to passing or failing, a test can be deemed to be
   inconclusive for a number of reasons.  Proper instrumentation and
   treatment of inconclusive outcomes is critical to the accuracy and

   robustness of Model-Based Metrics.  Tests can be inconclusive if the
   precomputed traffic pattern or data rates were not accurately
   generated; the measurement results were not statistically
   significant; the required preconditions for the test were not met; or
   other causes.  See Section 5.4.

   For example, consider a test that implements Constant Window Pseudo
   CBR (Section 6.2) by adding rate controls and detailed IP packet
   transfer instrumentation to TCP (e.g., using the extended performance
   statistics for TCP as described in [RFC4898]).  TCP includes built-in
   control systems that might interfere with the sending data rate.  If
   such a test meets the required packet transfer statistics (e.g., run
   length) while failing to attain the specified data rate, it must be
   treated as an inconclusive result, because we cannot a priori
   determine if the reduced data rate was caused by a TCP problem or a
   network problem or if the reduced data rate had a material effect on
   the observed packet transfer statistics.

   Note that for capacity tests, if the observed packet transfer
   statistics meet the statistical criteria for failing (based on
   acceptance of hypothesis H1 in Section 7.2), the test can be
   considered to have failed because it doesn't really matter that the
   test didn't attain the required data rate.

   The important new properties of MBM, such as vantage independence,
   are a direct consequence of opening the control loops in the
   protocols, such that the test stream does not depend on network
   conditions or IP packets received.  Any mechanism that introduces
   feedback between the path's measurements and the test stream
   generation is at risk of introducing nonlinearities that spoil these
   properties.  Any exceptional event that indicates that such feedback
   has happened should cause the test to be considered inconclusive.

   Inconclusive tests may be caused by situations in which a test
   outcome is ambiguous because of network limitations or an unknown
   limitation on the IP diagnostic test itself, which may have been
   caused by some uncontrolled feedback from the network.

   Note that procedures that attempt to search the target parameter
   space to find the limits on a parameter such as target_data_rate are
   at risk of breaking the location-independent properties of Model-
   Based Metrics if any part of the boundary between passing,
   inconclusive, or failing results is sensitive to RTT (which is
   normally the case).  For example, the maximum data rate for a
   marginal link (e.g., exhibiting excess errors) is likely to be
   sensitive to the test_path_RTT.  The maximum observed data rate over
   the test path has very little value for predicting the maximum rate
   over a different path.

   One of the goals for evolving TIDS designs will be to keep sharpening
   the distinctions between inconclusive, passing, and failing tests.
   The criteria for inconclusive, passing, and failing tests must be
   explicitly stated for every test in the TIDS or FSTIDS.

   One of the goals for evolving the testing process, procedures, tools,
   and measurement point selection should be to minimize the number of
   inconclusive tests.

   It may be useful to keep raw packet transfer statistics and ancillary
   metrics [RFC3148] for deeper study of the behavior of the network
   path and to measure the tools themselves.  Raw packet transfer
   statistics can help to drive tool evolution.  Under some conditions,
   it might be possible to re-evaluate the raw data for satisfying
   alternate Target Transport Performance.  However, it is important to
   guard against sampling bias and other implicit feedback that can
   cause false results and exhibit measurement point vantage
   sensitivity.  Simply applying different delivery criteria based on a
   different Target Transport Performance is insufficient if the test
   traffic patterns (bursts, etc.) do not match the alternate Target
   Transport Performance.

7.2.  Statistical Criteria for Estimating run_length

   When evaluating the observed run_length, we need to determine
   appropriate packet stream sizes and acceptable error levels for
   efficient measurement.  In practice, can we compare the empirically
   estimated packet loss and ECN CE marking ratios with the targets as
   the sample size grows?  How large a sample is needed to say that the
   measurements of packet transfer indicate a particular run length is
   present?

   The generalized measurement can be described as recursive testing:
   send packets (individually or in patterns) and observe the packet
   transfer performance (packet loss ratio, other metric, or any marking
   we define).

   As each packet is sent and measured, we have an ongoing estimate of
   the performance in terms of the ratio of packet loss or ECN CE marks
   to total packets (i.e., an empirical probability).  We continue to
   send until conditions support a conclusion or a maximum sending limit
   has been reached.

   We have a target_mark_probability, one mark per target_run_length,
   where a "mark" is defined as a lost packet, a packet with ECN CE
   mark, or other signal.  This constitutes the null hypothesis:

   H0:  no more than one mark in target_run_length =
      3*(target_window_size)^2 packets

   We can stop sending packets if ongoing measurements support accepting
   H0 with the specified Type I error = alpha (= 0.05, for example).

   We also have an alternative hypothesis to evaluate: is performance
   significantly lower than the target_mark_probability?  Based on
   analysis of typical values and practical limits on measurement
   duration, we choose four times the H0 probability:

   H1:  one or more marks in (target_run_length/4) packets

   and we can stop sending packets if measurements support rejecting H0
   with the specified Type II error = beta (= 0.05, for example), thus
   preferring the alternate hypothesis H1.

   H0 and H1 constitute the success and failure outcomes described
   elsewhere in this document; while the ongoing measurements do not
   support either hypothesis, the current status of measurements is
   inconclusive.

   The problem above is formulated to match the Sequential Probability
   Ratio Test (SPRT) [Wald45] [Montgomery90].  Note that as originally
   framed, the events under consideration were all manufacturing
   defects.  In networking, ECN CE marks and lost packets are not
   defects but signals, indicating that the transport protocol should
   slow down.

   The Sequential Probability Ratio Test also starts with a pair of
   hypotheses specified as above:

   H0:  p0 = one defect in target_run_length

   H1:  p1 = one defect in target_run_length/4

   As packets are sent and measurements collected, the tester evaluates
   the cumulative defect count against two boundaries representing H0
   Acceptance or Rejection (and acceptance of H1):

   Acceptance line:  Xa = -h1 + s*n

   Rejection line:  Xr = h2 + s*n

   where n increases linearly for each packet sent and

   h1 =  { log((1-alpha)/beta) }/k

   h2 =  { log((1-beta)/alpha) }/k

   k  =  log{ (p1(1-p0)) / (p0(1-p1)) }

   s  =  [ log{ (1-p0)/(1-p1) } ]/k

   for p0 and p1 as defined in the null and alternative hypotheses
   statements above, and alpha and beta as the Type I and Type II
   errors.

   The SPRT specifies simple stopping rules:

   o  Xa < defect_count(n) < Xr: continue testing

   o  defect_count(n) <= Xa: Accept H0

   o  defect_count(n) >= Xr: Accept H1

   The calculations above are implemented in the R-tool for Statistical
   Analysis [Rtool], in the add-on package for Cross-Validation via
   Sequential Testing (CVST) [CVST].

   Using the equations above, we can calculate the minimum number of
   packets (n) needed to accept H0 when x defects are observed.  For
   example, when x = 0:

   Xa = 0  = -h1 + s*n

   and  n = h1 / s

   Note that the derivations in [Wald45] and [Montgomery90] differ.
   Montgomery's simplified derivation of SPRT may assume a Bernoulli
   processes, where the packet loss probabilities are independent and
   identically distributed, making the SPRT more accessible.  Wald's
   seminal paper showed that this assumption is not necessary.  It helps
   to remember that the goal of SPRT is not to estimate the value of the
   packet loss rate but only whether or not the packet loss ratio is
   likely (1) low enough (when we accept the H0 null hypothesis),
   yielding success or (2) too high (when we accept the H1 alternate
   hypothesis), yielding failure.

7.3.  Reordering Tolerance

   All tests must be instrumented for packet-level reordering [RFC4737].
   However, there is no consensus for how much reordering should be
   acceptable.  Over the last two decades, the general trend has been to

   make protocols and applications more tolerant to reordering (for
   example, see [RFC5827]), in response to the gradual increase in
   reordering in the network.  This increase has been due to the
   deployment of technologies such as multithreaded routing lookups and
   Equal-Cost Multipath (ECMP) routing.  These techniques increase
   parallelism in the network and are critical to enabling overall
   Internet growth to exceed Moore's Law.

   With transport retransmission strategies, there are fundamental
   trade-offs among reordering tolerance, how quickly losses can be
   repaired, and overhead from spurious retransmissions.  In advance of
   new retransmission strategies, we propose the following strawman:
   transport protocols should be able to adapt to reordering as long as
   the reordering extent is not more than the maximum of one quarter
   window or 1 ms, whichever is larger.  (These values come from
   experience prototyping Early Retransmit [RFC5827] and related
   algorithms.  They agree with the values being proposed for "RACK: a
   time-based fast loss detection algorithm" [RACK].)  Within this limit
   on reorder extent, there should be no bound on reordering density.

   By implication, recording that is less than these bounds should not
   be treated as a network impairment.  However, [RFC4737] still
   applies: reordering should be instrumented, and the maximum
   reordering that can be properly characterized by the test (because of
   the bound on history buffers) should be recorded with the measurement
   results.

   Reordering tolerance and diagnostic limitations, such as the size of
   the history buffer used to diagnose packets that are way out of
   order, must be specified in an FSTIDS.