In this test case, the minimum bottleneck-link capacity between the two endpoints varies over time. This test is designed to measure the responsiveness of the candidate algorithm. This test tries to address the requirements in [
RFC 8836], which requires the algorithm to adapt the flow(s) and provide lower end-to-end latency when there exists:
-
an intermediate bottleneck
-
change in available capacity (e.g., due to interface change, routing change, abrupt arrival/departure of background non-adaptive traffic)
-
maximum media bit rate is greater than link capacity. In this case, when the application tries to ramp up to its maximum bit rate, since the link capacity is limited to a lower value, the congestion control scheme is expected to stabilize the sending bit rate close to the available bottleneck capacity.
It should be noted that the exact variation in available capacity due to any of the above depends on the underlying technologies. Hence, we describe a set of known factors, which may be extended to devise a more specific test case targeting certain behaviors in a certain network environment.
-
Expected behavior:
-
The candidate algorithm is expected to detect the path capacity constraint, converge to the bottleneck link's capacity, and adapt the flow to avoid unwanted media rate oscillation when the sending bit rate is approaching the bottleneck link's capacity. Such oscillations might occur when the media flow(s) attempts to reach its maximum bit rate but overshoots the usage of the available bottleneck capacity, then to rectify, it reduces the bit rate and starts to ramp up again.
-
Evaluation metrics:
-
As described in Section 4.1.
-
Testbed topology:
-
One media source S1 is connected to the corresponding R1. The media traffic is transported over the forward path and corresponding feedback/control traffic is transported over the backward path.
Forward -->
+---+ +-----+ +-----+ +---+
|S1 |=======| A |------------------------------>| B |=======|R1 |
+---+ | |<------------------------------| | +---+
+-----+ +-----+
<-- Backward
-
Testbed attributes:
-
-
Test duration:
-
100 s
-
Path characteristics:
-
as described in Section 4.2
-
Application-related:
-
-
Media Traffic:
-
-
Media type:
-
Video
-
Media direction:
-
forward
-
Number of media sources:
-
one (1)
-
Media timeline:
-
-
Start time:
-
0 s
-
End time:
-
99 s
-
Media type:
-
Audio
-
Media direction:
-
forward
-
Number of media sources:
-
one (1)
-
Media timeline:
-
-
Start time:
-
0 s
-
End time:
-
99 s
-
Competing traffic:
-
-
Number of sources:
-
zero (0)
-
Test-specific information:
-
-
One-way propagation delay:
-
[50 ms, 100 ms]. On the forward path direction.
This test uses bottleneck path capacity variation as listed in Table 1.
When using background non-adaptive UDP traffic to induce a time-varying bottleneck, the physical path capacity remains at 4 Mbps, and the UDP traffic source rate changes over time as (4 - (Y x r)), where r is the Reference bottleneck capacity in Mbps, and Y is the path capacity ratio specified in Table 1.
Variation pattern index |
Path direction |
Start time |
Path capacity ratio |
One |
Forward |
0 s |
1.0 |
Two |
Forward |
40 s |
2.5 |
Three |
Forward |
60 s |
0.6 |
Four |
Forward |
80 s |
1.0 |
Table 1: Path Capacity Variation Pattern for the Forward Direction
This test case is similar to
Section 5.1. However, this test will also consider persistent network load due to competing traffic.
-
Expected behavior:
-
The candidate algorithm is expected to detect the variation in available capacity and adapt the media stream(s) accordingly. The flows stabilize around their maximum bit rate as the maximum link capacity is large enough to accommodate the flows. When the available capacity drops, the flows adapt by decreasing their sending bit rate, and when congestion disappears, the flows are again expected to ramp up.
-
Evaluation metrics:
-
As described in Section 4.1.
-
Testbed topology:
-
Two (2) media sources S1 and S2 are connected to their corresponding destinations R1 and R2. The media traffic is transported over the forward path and corresponding feedback/control traffic is transported over the backward path.
+---+ +---+
|S1 |===== \ / =======|R1 |
+---+ \\ Forward --> // +---+
\\ //
+-----+ +-----+
| A |------------------------------>| B |
| |<------------------------------| |
+-----+ +-----+
// \\
// <-- Backward \\
+---+ // \\ +---+
|S2 |====== / \ ======|R2 |
+---+ +---+
-
Testbed attributes:
-
Testbed attributes are similar to those described in Section 5.1, except for the test-specific capacity variation setup.
-
Test-specific information:
-
This test uses path capacity variation as listed in Table 2 with a corresponding end time of 125 seconds. The reference bottleneck capacity is 2 Mbps. When using background non-adaptive UDP traffic to induce time-varying bottleneck for congestion-controlled media flows, the physical path capacity is 4 Mbps, and the UDP traffic source rate changes over time as (4 - (Y x r)), where r is the Reference bottleneck capacity in Mbps, and Y is the path capacity ratio specified in Table 2.
Variation pattern index |
Path direction |
Start time |
Path capacity ratio |
One |
Forward |
0 s |
2.0 |
Two |
Forward |
25 s |
1.0 |
Three |
Forward |
50 s |
1.75 |
Four |
Forward |
75 s |
0.5 |
Five |
Forward |
100 s |
1.0 |
Table 2: Path Capacity Variation Pattern for the Forward Direction
Real-time interactive media uses RTP; hence it is assumed that RTCP, RTP header extension, or such would be used by the congestion control algorithm in the back channel. Due to the asymmetric nature of the link between communicating peers, it is possible for a participating peer to not receive such feedback information due to an impaired or congested back channel (even when the forward channel might not be impaired). This test case is designed to observe the candidate congestion control behavior in such an event.
-
Expected behavior:
-
It is expected that the candidate algorithms are able to cope with the lack of feedback information and to adapt to minimize the performance degradation of media flows in the forward channel.
It should be noted that for this test case, logs are compared with the reference case, i.e., when the backward channel has no impairments.
-
Evaluation metrics:
-
As described in Section 4.1.
-
Testbed topology:
-
One (1) media source S1 is connected to corresponding R1, but both endpoints are additionally receiving and sending data, respectively. The media traffic (S1->R1) is transported over the forward path, and the corresponding feedback/control traffic is transported over the backward path. Likewise, media traffic (S2->R2) is transported over the backward path, and the corresponding feedback/control traffic is transported over the forward path.
+---+ +---+
|S1 |====== \ Forward --> / =======|R1 |
+---+ \\ // +---+
\\ //
+-----+ +-----+
| A |------------------------------>| B |
| |<------------------------------| |
+-----+ +-----+
// \\
// <-- Backward \\
+---+ // \\ +---+
|R2 |===== / \ ======|S2 |
+---+ +---+
-
Testbed attributes:
-
-
Test duration:
-
100 s
-
Path characteristics:
-
-
Reference bottleneck capacity:
-
1 Mbps
-
Application-related:
-
-
Media source:
-
-
Media type:
-
Video
-
Media direction:
-
forward and backward
-
Number of media sources:
-
two (2)
-
Media timeline:
-
-
Start time:
-
0 s
-
End time:
-
99 s
-
Media type:
-
Audio
-
Media direction:
-
forward and backward
-
Number of media sources:
-
two (2)
-
Media timeline:
-
-
Start time:
-
0 s
-
End time:
-
99 s
-
Competing traffic:
-
-
Number of sources:
-
zero (0)
-
Test-specific information:
-
This test uses path capacity variations to create a congested feedback link. Table 3 lists the variation patterns applied to the forward path, and Table 4 lists the variation patterns applied to the backward path. When using background non-adaptive UDP traffic to induce a time-varying bottleneck for congestion-controlled media flows, the physical path capacity is 4 Mbps for both directions, and the UDP traffic source rate changes over time as (4-x) Mbps in each direction, where x is the bottleneck capacity specified in Table 4.
Variation pattern index |
Path direction |
Start time |
Path capacity ratio |
One |
Forward |
0 s |
2.0 |
Two |
Forward |
20 s |
1.0 |
Three |
Forward |
40 s |
0.5 |
Four |
Forward |
60 s |
2.0 |
Table 3: Path Capacity Variation Pattern for the Forward Direction
Variation pattern index |
Path direction |
Start time |
Path capacity ratio |
One |
Backward |
0 s |
2.0 |
Two |
Backward |
35 s |
0.8 |
Three |
Backward |
70 s |
2.0 |
Table 4: Path Capacity Variation Pattern for the Backward Direction
In this test case, more than one media flow share the bottleneck link, and each of them uses the same congestion control algorithm. This is a typical scenario where a real-time interactive application sends more than one media flow to the same destination, and these flows are multiplexed over the same port. In such a scenario, it is likely that the flows will be routed via the same path and need to share the available bandwidth amongst themselves. For the sake of simplicity, it is assumed that there are no other competing traffic sources in the bottleneck link and that there is sufficient capacity to accommodate all the flows individually. While this appears to be a variant of the test case defined in
Section 5.2, it focuses on the capacity-sharing aspect of the candidate algorithm. The previous test case, on the other hand, measures adaptability, stability, and responsiveness of the candidate algorithm.
-
Expected behavior:
-
It is expected that the competing flows will converge to an optimum bit rate to accommodate all the flows with minimum possible latency and loss. Specifically, the test introduces three media flows at different time instances. When the second flow appears, there should still be room to accommodate another flow on the bottleneck link. Lastly, when the third flow appears, the bottleneck link should be saturated.
-
Evaluation metrics:
-
As described in Section 4.1.
-
Testbed topology:
-
Three media sources S1, S2, and S3 are connected to R1, R2, and R3, respectively. The media traffic is transported over the forward path, and the corresponding feedback/control traffic is transported over the backward path.
+---+ +---+
|S1 |===== \ Forward --> / =======|R1 |
+---+ \\ // +---+
\\ //
+---+ +-----+ +-----+ +---+
|S2 |=======| A |------------------------------>| B |=======|R2 |
+---+ | |<------------------------------| | +---+
+-----+ +-----+
// <-- Backward \\
+---+ // \\ +---+
|S3 |===== / \ ======|R3 |
+---+ +---+
-
Testbed attributes:
-
-
Test duration:
-
120 s
-
Path characteristics:
-
-
Reference bottleneck capacity:
-
3.5 Mbps
-
Path capacity ratio:
-
1.0
-
Application-related:
-
-
Media Source:
-
-
Media type:
-
Video
-
Media direction:
-
forward
-
Number of media sources:
-
three (3)
-
Media timeline:
-
New media flows are added sequentially, at short time intervals. See the test-specific setup below.
-
Media type:
-
Audio
-
Media direction:
-
forward
-
Number of media sources:
-
three (3)
-
Media timeline:
-
New media flows are added sequentially, at short time intervals. See the test-specific setup below.
-
Competing traffic:
-
-
Number of sources:
-
zero (0)
-
Test-specific information:
-
Table 5 defines the media timeline for both media types.
Flow ID |
Media type |
Start time |
End time |
1 |
Video |
0 s |
119 s |
2 |
Video |
20 s |
119 s |
3 |
Video |
40 s |
119 s |
4 |
Audio |
0 s |
119 s |
5 |
Audio |
20 s |
119 s |
6 |
Audio |
40 s |
119 s |
Table 5: Media Timelines for Video and Audio Media Sources
In this test case, multiple media flows share the bottleneck link, but the one-way propagation delay for each flow is different. For the sake of simplicity, it is assumed that there are no other competing traffic sources in the bottleneck link and that there is sufficient capacity to accommodate all the flows. While this appears to be a variant of test case 5.2 (
Section 5.2), it focuses on the capacity-sharing aspect of the candidate algorithm under different RTTs.
-
Expected behavior:
-
It is expected that the competing flows will converge to bit rates to accommodate all the flows with minimum possible latency and loss. The effectiveness of the algorithm depends on how fast and fairly the competing flows converge to their steady states irrespective of the RTT observed.
-
Evaluation metrics:
-
As described in Section 4.1.
-
Testbed topology:
-
Five (5) media sources S1..S5 are connected to their corresponding media sinks R1..R5. The media traffic is transported over the forward path, and the corresponding feedback/control traffic is transported over the backward path. The topology is the same as in Section 5.4.
-
Testbed attributes:
-
-
Test duration:
-
300 s
-
Path characteristics:
-
-
Reference bottleneck capacity:
-
4 Mbps
-
Path capacity ratio:
-
1.0
-
One-way propagation delay for each flow:
-
10 ms for S1-R1, 25 ms for S2-R2, 50 ms for S3-R3, 100 ms for S4-R4, and 150 ms S5-R5.
-
Application-related:
-
-
Media source:
-
-
Media type:
-
Video
-
Media direction:
-
forward
-
Number of media sources:
-
five (5)
-
Media timeline:
-
New media flows are added sequentially, at short time intervals. See the test-specific setup below.
-
Media type:
-
Audio
-
Media direction:
-
forward
-
Number of media sources:
-
five (5)
-
Media timeline:
-
New media flows are added sequentially, at short time intervals. See the test-specific setup below.
-
Competing traffic:
-
-
Number of sources:
-
zero (0)
-
Test-specific information:
-
Table 6 defines the media timeline for both media types.
Flow ID |
Media type |
Start time |
End time |
1 |
Video |
0 s |
299 s |
2 |
Video |
10 s |
299 s |
3 |
Video |
20 s |
299 s |
4 |
Video |
30 s |
299 s |
5 |
Video |
40 s |
299 s |
6 |
Audio |
0 s |
299 s |
7 |
Audio |
10 s |
299 s |
8 |
Audio |
20 s |
299 s |
9 |
Audio |
30 s |
299 s |
10 |
Audio |
40 s |
299 s |
Table 6: Media Timeline for Video and Audio Media Sources
In this test case, one or more media flows share the bottleneck link with at least one long-lived TCP flow. Long-lived TCP flows download data throughout the session and are expected to have infinite amount of data to send and receive. This is a scenario where a multimedia application coexists with a large file download. The test case measures the adaptivity of the candidate algorithm to competing traffic. It addresses requirement 3 in
Section 2 of
RFC 8836.
-
Expected behavior:
-
Depending on the convergence observed in test cases 5.1 and 5.2, the candidate algorithm may be able to avoid congestion collapse. In the worst case, the media stream will fall to the minimum media bit rate.
-
Evaluation metrics:
-
Includes the following metrics in addition to those described in Section 4.1:
-
Flow level:
-
TCP throughput
-
Loss for the TCP flow
-
Testbed topology:
-
One (1) media source S1 is connected to the corresponding media sink, R1. In addition, there is a long-lived TCP flow sharing the same bottleneck link. The media traffic is transported over the forward path, and the corresponding feedback/control traffic is transported over the backward path. The TCP traffic goes over the forward path from S_tcp with acknowledgment packets going over the backward path from R_tcp.
+--+ +--+
|S1|===== \ Forward --> / =====|R1|
+--+ \\ // +--+
\\ //
+-----+ +-----+
| A |---------------------------->| B |
| |<----------------------------| |
+-----+ +-----+
// <-- Backward \\
+-----+ // \\ +-----+
|S_tcp|=== / \ ===|R_tcp|
+-----+ +-----+
-
Testbed attributes:
-
-
Test duration:
-
120 s
-
Path characteristics:
-
-
Reference bottleneck capacity:
-
2 Mbps
-
Path capacity ratio:
-
1.0
-
Bottleneck queue size:
-
[300 ms, 1000 ms]
-
Application-related:
-
-
Media source:
-
-
Media type:
-
Video
-
Media direction:
-
forward
-
Number of media sources:
-
one (1)
-
Media timeline:
-
-
Start time:
-
5 s
-
End time:
-
119 s
-
Media type:
-
Audio
-
Media direction:
-
forward
-
Number of media sources:
-
one (1)
-
Media timeline:
-
-
Start time:
-
5 s
-
End time:
-
119 s
Additionally, implementers are encouraged to run the experiment with multiple media sources.
-
Competing traffic:
-
-
Number and types of sources:
-
one (1) and long-lived TCP
-
Traffic direction:
-
forward
-
Congestion control:
-
default TCP congestion control [RFC 5681]. Implementers are also encouraged to run the experiment with alternative TCP congestion control algorithms.
-
Traffic timeline:
-
-
Start time:
-
0 s
-
End time:
-
119 s
-
Test-specific information:
-
none
In this test case, one or more congestion-controlled media flows share the bottleneck link with multiple short-lived TCP flows. Short-lived TCP flows resemble the on/off pattern observed in web traffic, wherein clients (for example, browsers) connect to a server and download a resource (typically a web page, few images, text files, etc.) using several TCP connections. This scenario shows the performance of a multimedia application when several browser windows are active. The test case measures the adaptivity of the candidate algorithm to competing web traffic, and it addresses requirement 1.E in
Section 2 of
RFC 8836.
Depending on the number of short TCP flows, the cross traffic either appears as a short burst flow or resembles a long-lived TCP flow. The intention of this test is to observe the impact of a short-term burst on the behavior of the candidate algorithm.
-
Expected behavior:
-
The candidate algorithm is expected to avoid flow starvation during the presence of short and bursty competing TCP flows, streaming at least at the minimum media bit rate. After competing TCP flows terminate, the media streams are expected to be robust enough to eventually recover to previous steady state behavior, and at the very least, avoid persistent starvation.
-
Evaluation metrics:
-
Includes the following metrics in addition to those described in Section 4.1:
-
Flow level:
-
Variation in the sending rate of the TCP flow
-
TCP throughput
-
Testbed topology:
-
The topology described here is the same as the one described in Figure 6.
-
Testbed attributes:
-
-
Test duration:
-
300 s
-
Path characteristics:
-
-
Reference bottleneck capacity:
-
2.0 Mbps
-
Path capacity ratio:
-
1.0
-
Application-related:
-
-
Media source:
-
-
Media type:
-
Video
-
Media direction:
-
forward
-
Number of media sources:
-
two (2)
-
Media timeline:
-
-
Start time:
-
5 s
-
End time:
-
299 s
-
Media type:
-
Audio
-
Media direction:
-
forward
-
Number of media sources:
-
two (2)
-
Media timeline:
-
-
Start time:
-
5 s
-
End time:
-
299 s
-
Competing traffic:
-
-
Number and types of sources:
-
ten (10), short-lived TCP flows.
-
Traffic direction:
-
forward
-
Congestion algorithm:
-
default TCP congestion control [RFC 5681]. Implementers are also encouraged to run the experiment with an alternative TCP congestion control algorithm.
-
Traffic timeline:
-
Each short TCP flow is modeled as a sequence of file downloads interleaved with idle periods. Not all short TCP flows start at the same time, two of them start in the ON state, while rest of the eight flows start in an OFF state. For a description of the short TCP flow model, see test-specific information below.
-
Test-specific information:
-
-
Short TCP traffic model:
-
The short TCP model to be used in this test is described in [RFC 8868].
In this test case, more than one real-time interactive media flow share the link bandwidth, and all flows reach to a steady state by utilizing the link capacity in an optimum way. At this stage, one of the media flows is paused for a moment. This event will result in more available bandwidth for the rest of the flows as they are on a shared link. When the paused media flow resumes, it no longer has the same bandwidth share on the link. It has to make its way through the other existing flows in the link to achieve a fair share of the link capacity. This test case is important specially for real-time interactive media, which consists of more than one media flows and can pause/resume media flows at any point of time during the session. This test case directly addresses requirement 5 in
Section 2 of
RFC 8836. One can think of it as a variation of the test case defined in
Section 5.4. However, it is different as the candidate algorithms can use different strategies to increase efficiency, for example, in terms of fairness, convergence time, oscillation reduction, etc., by capitalizing on the fact that they have previous information of the link.
-
Expected behavior:
-
During the period where the third stream is paused, the two remaining flows are expected to increase their rates and reach the maximum media bit rate. When the third stream resumes, all three flows are expected to converge to the same original fair share of rates prior to the media pause/resume event.
-
Evaluation metrics:
-
Includes the following metrics in addition to those described in Section 4.1:
-
Flow level:
-
Variation in sending bit rate and throughput. Mainly observing the frequency and magnitude of oscillations.
-
Testbed topology:
-
Same as the test case defined in Section 5.4.
-
Testbed attributes:
-
The general description of the testbed parameters are the same as Section 5.4 with changes in the test-specific setup as below:
Other test-specific setup:
-
Media flow timeline:
-
-
Flow ID:
-
one (1)
-
Start time:
-
0 s
-
Flow duration:
-
119 s
-
Pause time:
-
not required
-
Resume time:
-
not required
-
Media flow timeline:
-
-
Flow ID:
-
two (2)
-
Start time:
-
0 s
-
Flow duration:
-
119 s
-
Pause time:
-
at 40 s
-
Resume time:
-
at 60 s
-
Media flow timeline:
-
-
Flow ID:
-
three (3)
-
Start time:
-
0 s
-
Flow duration:
-
119 s
-
Pause time:
-
not required
-
Resume time:
-
not required