RFC 5559
Pre-Congestion Notification (PCN) Architecture
4. Detailed Functional Architecture
This section is intended to provide a systematic summary of the new functional architecture in the PCN-domain. First, it describes functions needed at the three specific types of PCN-node; these are data plane functions and are in addition to the normal router functions for PCN-nodes. Then, it describes the further functionality needed for both flow admission control and flow termination; these are signalling and decision-making functions, and there are various possibilities for where the functions are physically located. The section is split into: 1. functions needed at PCN-interior-nodes 2. functions needed at PCN-ingress-nodes 3. functions needed at PCN-egress-nodes 4. other functions needed for flow admission control 5. other functions needed for flow termination control Note: Probing is covered in the Appendix. The section then discusses some other detailed topics: 1. addressing 2. tunnelling 3. fault handling
4.1. PCN-Interior-Node Functions
Each link of the PCN-domain is configured with the following functionality: o Behaviour aggregate classification - determine whether or not an incoming packet is a PCN-packet. o PCN-meter - measure the "amount of PCN-traffic". The measurement is made on the overall PCN-traffic, not per flow. Algorithms determine whether to indicate to the PCN-marking functionality that packets should be PCN-marked. o PCN-mark - as triggered by indications from the PCN-meter functionality; if necessary, PCN-mark packets with the appropriate encoding. o Drop - if the queue overflows, then naturally packets are dropped. In addition, the link may be configured with a maximum rate for PCN-traffic (below the physical link rate), above which PCN- packets are dropped. The functions are defined in [Eardley09] and the baseline encoding in [Moncaster09-1] (extended encodings are to be defined in other documents). +---------+ Result +->|Threshold|-------+ | | Meter | | | +---------+ V +----------+ +- - - - -+ | +------+ | BA | | | | | | Marked Packet =>|Classifier|==>| Dropper |==?===============>|Marker|==> Packet Stream | | | | | | | Stream +----------+ +- - - - -+ | +------+ | +---------+ ^ | | Excess | | +->| Traffic |-------+ | Meter | Result +---------+ Figure 4: Schematic of PCN-interior-node functionality.4.2. PCN-Ingress-Node Functions
Each ingress link of the PCN-domain is configured with the following functionality:
o Packet classification - determine whether an incoming packet is
part of a previously admitted flow by using a filter spec (eg,
DSCP, source and destination addresses, port numbers, and
protocol).
o Police - police, by dropping any packets received with a DSCP
indicating PCN transport that do not belong to an admitted flow.
(A prospective PCN-flow that is rejected could be blocked or
admitted into a lower-priority behaviour aggregate.) Similarly,
police packets that are part of a previously admitted flow, to
check that the flow keeps to the agreed rate or flowspec (eg, see
[RFC1633] for a microflow and its NSIS equivalent).
o PCN-colour - set the DSCP and ECN fields appropriately for the
PCN-domain, for example, as in [Moncaster09-1].
o Meter - some approaches to flow termination require the PCN-
ingress-node to measure the (aggregate) rate of PCN-traffic
towards a particular PCN-egress-node.
The first two are policing functions, needed to make sure that PCN-
packets admitted into the PCN-domain belong to a flow that has been
admitted and to ensure that the flow keeps to the flowspec agreed
(eg, doesn't exceed an agreed maximum rate and is inelastic traffic).
Installing the filter spec will typically be done by the signalling
protocol, as will re-installing the filter, for example, after a re-
route that changes the PCN-ingress-node (see [Briscoe06] for an
example using RSVP). PCN-colouring allows the rest of the PCN-domain
to recognise PCN-packets.
4.3. PCN-Egress-Node Functions
Each egress link of the PCN-domain is configured with the following
functionality:
o Packet classify - determine which PCN-ingress-node a PCN-packet
has come from.
o Meter - "measure PCN-traffic" or "monitor PCN-marks".
o PCN-colour - for PCN-packets, set the DSCP and ECN fields to the
appropriate values for use outside the PCN-domain.
The metering functionality, of course, depends on whether it is
targeted at admission control or flow termination. Alternatives
involve the PCN-egress-node "measuring", as an aggregate (ie, not per
flow), all PCN-packets from a particular PCN-ingress-node, or
"monitoring" the PCN-traffic and reacting to one (or several) PCN-
marked packets. For PCN-colouring, [Moncaster09-1] specifies that the PCN-egress-node resets the ECN field to 00; other encodings may define different behaviour.4.4. Admission Control Functions
As well as the functions covered above, other specific admission control functions need to be performed (others might be possible): o Make decision about admission - based on the output of the PCN- egress-node's meter function. In the case where it "measures PCN- traffic", the measured traffic on the ingress-egress-aggregate is compared with some reference level. In the case where it "monitors PCN-marks", the decision is based on whether or not one (or several) packets are PCN-marked (eg, the RSVP PATH message). In either case, the admission decision also takes account of policy and application-layer requirements [RFC2753]. o Communicate decision about admission - signal the decision to the node making the admission control request (which may be outside the PCN-domain) and to the policer (PCN-ingress-node function) for enforcement of the decision. There are various possibilities for how the functionality could be distributed (we assume the operator will configure which is used): o The decision is made at the PCN-egress-node and the decision (admit or block) is signalled to the PCN-ingress-node. o The decision is recommended by the PCN-egress-node (admit or block), but the decision is definitively made by the PCN-ingress- node. The rationale is that the PCN-egress-node naturally has the necessary information about the amount of PCN-marks on the ingress-egress-aggregate, whereas the PCN-ingress-node is the policy enforcement point [RFC2753] that polices incoming traffic to ensure it is part of an admitted PCN-flow. o The decision is made at the PCN-ingress-node, which requires that the PCN-egress-node signals PCN-feedback-information to the PCN- ingress-node. For example, it could signal the current fraction of PCN-traffic that is PCN-marked. o The decision is made at a centralised node (see Appendix). Note: Admission control functionality is not performed by normal PCN- interior-nodes.
4.5. Flow Termination Functions
As well as the functions covered above, other specific termination control functions need to be performed (others might be possible): o PCN-meter at PCN-egress-node - similarly to flow admission, there are two types of possibilities: to "measure PCN-traffic" on the ingress-egress-aggregate, or to "monitor PCN-marks" and react to one (or several) PCN-marks. o (if required) PCN-meter at PCN-ingress-node - make "measurements of PCN-traffic" being sent towards a particular PCN-egress-node; again, this is done for the ingress-egress-aggregate and not per flow. o (if required) Communicate PCN-feedback-information to the node that makes the flow termination decision - for example, as in [Briscoe06], communicate the PCN-egress-node's measurements to the PCN-ingress-node. o Make decision about flow termination - use the information from the PCN-meter(s) to decide which PCN-flow or PCN-flows to terminate. The decision takes account of policy and application- layer requirements [RFC2753]. o Communicate decision about flow termination - signal the decision to the node that is able to terminate the flow (which may be outside the PCN-domain) and to the policer (PCN-ingress-node function) for enforcement of the decision. There are various possibilities for how the functionality could be distributed, similar to those discussed above in Section 4.4. Note: Flow termination functionality is not performed by normal PCN- interior-nodes.4.6. Addressing
PCN-nodes may need to know the address of other PCN-nodes. Note that PCN-interior-nodes don't need to know the address of other PCN-nodes (except their next-hop neighbours for routing purposes). At a minimum, the PCN-egress-node needs to know the address of the PCN-ingress-node associated with a flow so that the PCN-ingress-node can be informed of the admission decision (and any flow termination decision) and enforce it through policing. There are various
possibilities for how the PCN-egress-node can do this, ie, associate
the received packet to the correct ingress-egress-aggregate. It is
not the intention of this document to mandate a particular mechanism.
o The addressing information can be gathered from signalling -- for
example, through the regular processing of an RSVP PATH message,
as the PCN-ingress-node is the previous RSVP hop (PHOP)
([Lefaucheur06]). Another option is that the PCN-ingress-node
could signal its address to the PCN-egress-node.
o Always tunnel PCN-traffic across the PCN-domain. Then the PCN-
ingress-node's address is simply the source address of the outer
packet header. The PCN-ingress-node needs to learn the address of
the PCN-egress-node, either by manual configuration or by one of
the automated tunnel endpoint discovery mechanisms (such as
signalling or probing over the data route, interrogating routing,
or using a centralised broker).
4.7. Tunnelling
Tunnels may originate and/or terminate within a PCN-domain (eg, IP
over IP, IP over MPLS). It is important that the PCN-marking of any
packet can potentially influence PCN's flow admission control and
termination -- it shouldn't matter whether the packet happens to be
tunnelled at the PCN-node that PCN-marks the packet, or indeed
whether it's decapsulated or encapsulated by a subsequent PCN-node.
This suggests that the "uniform conceptual model" described in
[RFC2983] should be re-applied in the PCN context. In line with both
this and the approach of [RFC4303] and [Briscoe09], the following
rule is applied if encapsulation is done within the PCN-domain:
o Any PCN-marking is copied into the outer header.
Note: A tunnel will not provide this behaviour if it complies with
[RFC3168] tunnelling in either mode, but it will if it complies with
[RFC4301] IPsec tunnelling.
Similarly, in line with the "uniform conceptual model" of [RFC2983],
with the "full-functionality option" of [RFC3168], and with
[RFC4301], the following rule is applied if decapsulation is done
within the PCN-domain:
o If the outer header's marking state is more severe, then it is
copied onto the inner header.
Note that the order of increasing severity is: not PCN-marked,
threshold-marked, and excess-traffic-marked.
An operator may wish to tunnel PCN-traffic from PCN-ingress-nodes to PCN-egress-nodes. The PCN-marks shouldn't be visible outside the PCN-domain, which can be achieved by the PCN-egress-node doing the PCN-colouring function (Section 4.3) after all the other (PCN and tunnelling) functions. The potential reasons for doing such tunnelling are: the PCN-egress-node then automatically knows the address of the relevant PCN-ingress-node for a flow, and, even if ECMP (Equal Cost Multi-Path) is running, all PCN-packets on a particular ingress-egress-aggregate follow the same path (for more on ECMP, see Section 6.4). But such tunnelling also has drawbacks, for example, the additional overhead in terms of bandwidth and processing as well as the cost of setting up a mesh of tunnels between PCN- boundary-nodes (there is an N^2 scaling issue). Potential issues arise for a "partially PCN-capable tunnel", ie, where only one tunnel endpoint is in the PCN-domain: 1. The tunnel originates outside a PCN-domain and ends inside it. If the packet arrives at the tunnel ingress with the same encoding as used within the PCN-domain to indicate PCN-marking, then this could lead the PCN-egress-node to falsely measure pre- congestion. 2. The tunnel originates inside a PCN-domain and ends outside it. If the packet arrives at the tunnel ingress already PCN-marked, then it will still have the same encoding when it's decapsulated, which could potentially confuse nodes beyond the tunnel egress. In line with the solution for partially capable Diffserv tunnels in [RFC2983], the following rules are applied: o For case (1), the tunnel egress node clears any PCN-marking on the inner header. This rule is applied before the "copy on decapsulation" rule above. o For case (2), the tunnel ingress node clears any PCN-marking on the inner header. This rule is applied after the "copy on encapsulation" rule above. Note that the above implies that one has to know, or determine, the characteristics of the other end of the tunnel as part of establishing it. Tunnelling constraints were a major factor in the choice of the baseline encoding. As explained in [Moncaster09-1], with current tunnelling endpoints, only the 11 codepoint of the ECN field survives decapsulation, and hence the baseline encoding only uses the 11 codepoint to indicate PCN-marking. Extended encoding schemes need to
explain their interactions with (or assumptions about) tunnelling. A lengthy discussion of all the issues associated with layered encapsulation of congestion notification (for ECN as well as PCN) is in [Briscoe09].4.8. Fault Handling
If a PCN-interior-node (or one of its links) fails, then lower-layer protection mechanisms or the regular IP routing protocol will eventually re-route around it. If the new route can carry all the admitted traffic, flows will gracefully continue. If instead this causes early warning of pre-congestion on the new route, then admission control based on Pre-Congestion Notification will ensure that new flows will not be admitted until enough existing flows have departed. Re-routing may result in heavy (pre-)congestion, which will cause the flow termination mechanism to kick in. If a PCN-boundary-node fails, then we would like the regular QoS signalling protocol to be responsible for taking appropriate action. As an example, [Briscoe09] considers what happens if RSVP is the QoS signalling protocol.5. Operations and Management
This section considers operations and management issues, under the FCAPS headings: Faults, Configuration, Accounting, Performance, and Security. Provisioning is discussed with performance.5.1. Fault Operations and Management
Fault Operations and Management is about preventing faults, telling the management system (or manual operator) that the system has recovered (or not) from a failure, and about maintaining information to aid fault diagnosis. Admission blocking and, particularly, flow termination mechanisms should rarely be needed in practice. It would be unfortunate if they didn't work after an option had been accidentally disabled. Therefore, it will be necessary to regularly test that the live system works as intended (devising a meaningful test is left as an exercise for the operator). Section 4 describes how the PCN architecture has been designed to ensure admitted flows continue gracefully after recovering automatically from link or node failures. The need to record and monitor re-routing events affecting signalling is unchanged by the
addition of PCN to a Diffserv domain. Similarly, re-routing events within the PCN-domain will be recorded and monitored just as they would be without PCN. PCN-marking does make it possible to record "near-misses". For instance, at the PCN-egress-node a "reporting threshold" could be set to monitor how often -- and for how long -- the system comes close to triggering flow blocking without actually doing so. Similarly, bursts of flow termination marking could be recorded even if they are not sufficiently sustained to trigger flow termination. Such statistics could be correlated with per-queue counts of marking volume (Section 5.2) to upgrade resources in danger of causing service degradation or to trigger manual tracing of intermittent incipient errors that would otherwise have gone unnoticed. Finally, of course, many faults are caused by failings in the management process ("human error"): a wrongly configured address in a node, a wrong address given in a signalling protocol, a wrongly configured parameter in a queueing algorithm, a node set into a different mode from other nodes, and so on. Generally, a clean design with few configurable options ensures this class of faults can be traced more easily and prevented more often. Sound management practice at run-time also helps. For instance, a management system should be used that constrains configuration changes within system rules (eg, preventing an option setting inconsistent with other nodes), configuration options should be recorded in an offline database, and regular automatic consistency checks between live systems and the database should be performed. PCN adds nothing specific to this class of problems.5.2. Configuration Operations and Management
Threshold-metering and -marking and excess-traffic-metering and -marking are standardised in [Eardley09]. However, more diversity in PCN-boundary-node behaviours is expected, in order to interface with diverse industry architectures. It may be possible to have different PCN-boundary-node behaviours for different ingress-egress-aggregates within the same PCN-domain. PCN-metering behaviour is enabled on either the egress or the ingress interfaces of PCN-nodes. A consistent choice must be made across the PCN-domain to ensure that the PCN mechanisms protect all links.
PCN configuration control variables fall into the following categories: o system options (enabling or disabling behaviours) o parameters (setting levels, addresses, etc.) One possibility is that all configurable variables sit within an SNMP (Simple Network Management Protocol) management framework [RFC3411], being structured within a defined management information base (MIB) on each node, and being remotely readable and settable via a suitably secure management protocol (such as SNMPv3). Some configuration options and parameters have to be set once to "globally" control the whole PCN-domain. Where possible, these are identified below. This may affect operational complexity and the chances of interoperability problems between equipment from different vendors. It may be possible for an operator to configure some PCN-interior- nodes so that they don't run the PCN mechanisms, if it knows that these links will never become (pre-)congested.5.2.1. System Options
On PCN-interior-nodes there will be very few system options: o Whether two PCN-markings (threshold-marked and excess-traffic- marked) are enabled or only one. Typically, all nodes throughout a PCN-domain will be configured the same in this respect. However, exceptions could be made. For example, if most PCN-nodes used both markings but some legacy hardware was incapable of running two algorithms, an operator might be willing to configure these legacy nodes solely for excess-traffic-marking to enable flow termination as a back-stop. It would be sensible to place such nodes where they could be provisioned with a greater leeway over expected traffic levels. o In the case where only one PCN-marking is enabled, all nodes must be configured to generate PCN-marks from the same meter (ie, either the threshold meter or the excess-traffic meter). PCN-boundary-nodes (ingress and egress) will have more system options: o Which of admission and flow termination are enabled. If any PCN- interior-node is configured to generate a marking, all PCN- boundary-nodes must be able to interpret that marking (which
includes understanding, in a PCN-domain that uses only one type of
PCN-marking, whether they are generated by PCN-interior-nodes'
threshold meters or their excess-traffic meters). Therefore, all
PCN-boundary-nodes must be configured the same in this respect.
o Where flow admission and termination decisions are made: at PCN-
ingress-nodes or at PCN-egress-nodes (or at a centralised node,
see Appendix). Theoretically, this configuration choice could be
negotiated for each pair of PCN-boundary-nodes, but we cannot
imagine why such complexity would be required, except perhaps in
future inter-domain scenarios.
o How PCN-markings are translated into admission control and flow
termination decisions (see Sections 3.1 and 3.2).
PCN-egress-nodes will have further system options:
o How the mapping should be established between each packet and its
aggregate (eg, by MPLS label and by IP packet filter spec) and how
to take account of ECMP.
o If an equipment vendor provides a choice, there may be options for
selecting which smoothing algorithm to use for measurements.
5.2.2. Parameters
Like any Diffserv domain, every node within a PCN-domain will need to
be configured with the DSCP(s) used to identify PCN-packets. On each
interior link, the main configuration parameters are the PCN-
threshold-rate and PCN-excess-rate. A larger PCN-threshold-rate
enables more PCN-traffic to be admitted on a link, hence improving
capacity utilisation. A PCN-excess-rate set further above the PCN-
threshold-rate allows greater increases in traffic (whether due to
natural fluctuations or some unexpected event) before any flows are
terminated, ie, minimises the chances of unnecessarily triggering the
termination mechanism. For instance, an operator may want to design
their network so that it can cope with a failure of any single PCN-
node without terminating any flows.
Setting these rates on the first deployment of PCN will be very
similar to the traditional process for sizing an admission-controlled
network, depending on: the operator's requirements for minimising
flow blocking (grade of service), the expected PCN-traffic load on
each link and its statistical characteristics (the traffic matrix),
contingency for re-routing the PCN-traffic matrix in the event of
single or multiple failures, and the expected load from other classes
relative to link capacities [Menth09-1]. But, once a domain is in
operation, a PCN design goal is to be able to determine growth in
these configured rates much more simply, by monitoring PCN-marking rates from actual rather than expected traffic (see Section 5.4 on Performance and Provisioning). Operators may also wish to configure a rate greater than the PCN- excess-rate that is the absolute maximum rate that a link allows for PCN-traffic. This may simply be the physical link rate, but some operators may wish to configure a logical limit to prevent starvation of other traffic classes during any brief period after PCN-traffic exceeds the PCN-excess-rate but before flow termination brings it back below this rate. Threshold-metering requires a threshold token bucket depth to be configured, excess-traffic-metering requires a value for the MTU (maximum size of a PCN-packet on the link), and both require setting a maximum size of their token buckets. It is preferable to have rules that set defaults for these parameters but to then allow operators to change them -- for instance, if average traffic characteristics change over time. The PCN-egress-node may allow configuration of: o how it smooths metering of PCN-markings (eg, EWMA parameters) Whichever node makes admission and flow termination decisions will contain algorithms for converting PCN-marking levels into admission or flow termination decisions. These will also require configurable parameters, for instance: o An admission control algorithm that is based on the fraction of marked packets will at least require a marking threshold setting above which it denies admission to new flows. o Flow termination algorithms will probably require a parameter to delay termination of any flows until it is more certain that an anomalous event is not transient. o A parameter to control the trade-off between how quickly excess flows are terminated and over-termination. One particular approach [Charny07-2] would require a global parameter to be defined on all PCN-nodes, but would only need one PCN-marking rate to be configured on each link. The global parameter is a scaling factor between admission and termination (the rate of PCN- traffic on a link up to which flows are admitted vs. the rate above which flows are terminated). [Charny07-2] discusses in full the impact of this particular approach on the operation of PCN.
5.3. Accounting Operations and Management
Accounting is only done at trust boundaries so it is out of scope of this document, which is confined to intra-domain issues. Use of PCN internal to a domain makes no difference to the flow signalling events crossing trust boundaries outside the PCN-domain, which are typically used for accounting.5.4. Performance and Provisioning Operations and Management
Monitoring of performance factors measurable from *outside* the PCN- domain will be no different with PCN than with any other packet- based, flow admission control system, both at the flow level (blocking probability, etc.) and the packet level (jitter [RFC3393], [Y.1541], loss rate [RFC4656], mean opinion score [P.800], etc.). The difference is that PCN is intentionally designed to indicate *internally* which exact resource(s) are the cause of performance problems and by how much. Even better, PCN indicates which resources will probably cause problems if they are not upgraded soon. This can be achieved by the management system monitoring the total amount (in bytes) of PCN- marking generated by each queue over a period. Given possible long provisioning lead times, pre-congestion volume is the best metric to reveal whether sufficient persistent demand has occurred to warrant an upgrade because, even before utilisation becomes problematic, the statistical variability of traffic will cause occasional bursts of pre-congestion. This "early warning system" decouples the process of adding customers from the provisioning process. This should cut the time to add a customer when compared against admission control that is provided over native Diffserv [RFC2998] because it saves having to verify the capacity-planning process before adding each customer. Alternatively, before triggering an upgrade, the long-term pre- congestion volume on each link can be used to balance traffic load across the PCN-domain by adjusting the link weights of the routing system. When an upgrade to a link's configured PCN-rates is required, it may also be necessary to upgrade the physical capacity available to other classes. However, there will usually be sufficient physical capacity for the upgrade to go ahead as a simple configuration change. Alternatively, [Songhurst06] describes an adaptive rather than preconfigured system, where the configured PCN- threshold-rate is replaced with a high and low water mark and the marking algorithm automatically optimises how physical capacity is shared, using the relative loads from PCN and other traffic classes.
All the above processes require just three extra counters associated with each PCN queue: threshold-markings, excess-traffic-markings, and drops. Every time a PCN-packet is marked or dropped, its size in bytes should be added to the appropriate counter. Then the management system can read the counters at any time and subtract a previous reading to establish the incremental volume of each type of (pre-)congestion. Readings should be taken frequently so that anomalous events (eg, re-routes) can be distinguished from regular fluctuating demand, if required.5.5. Security Operations and Management
Security Operations and Management is about using secure operational practices as well as being able to track security breaches or near- misses at run-time. PCN adds few specifics to the general good practice required in this field [RFC4778]. The correct functions of the system should be monitored (Section 5.4) in multiple independent ways and correlated to detect possible security breaches. Persistent (pre-)congestion marking should raise an alarm (both on the node doing the marking and on the PCN-egress-node metering it). Similarly, persistently poor external QoS metrics (such as jitter or mean opinion score) should raise an alarm. The following are examples of symptoms that may be the result of innocent faults, rather than attacks; however, until diagnosed, they should be logged and should trigger a security alarm: o Anomalous patterns of non-conforming incoming signals and packets rejected at the PCN-ingress-nodes (eg, packets already marked PCN- capable or traffic persistently starving token bucket policers). o PCN-capable packets arriving at a PCN-egress-node with no associated state for mapping them to a valid ingress-egress- aggregate. o A PCN-ingress-node receiving feedback signals that are about the pre-congestion level on a non-existent aggregate or that are inconsistent with other signals (eg, unexpected sequence numbers, inconsistent addressing, conflicting reports of the pre-congestion level, etc.). o Pre-congestion marking arriving at a PCN-egress-node with (pre-)congestion markings focused on particular flows, rather than randomly distributed throughout the aggregate.
6. Applicability of PCN
6.1. Benefits
The key benefits of the PCN mechanisms are that they are simple, scalable, and robust, because: o Per-flow state is only required at the PCN-ingress-nodes ("stateless core"). This is required for policing purposes (to prevent non-admitted PCN-traffic from entering the PCN-domain) and so on. It is not generally required that other network entities are aware of individual flows (although they may be in particular deployment scenarios). o Admission control is resilient: with PCN, QoS is decoupled from the routing system. Hence, in general, admitted flows can survive capacity, routing, or topology changes without additional signalling. The PCN-admissible-rate on each link can be chosen to be small enough that admitted traffic can still be carried after a re-routing in most failure cases [Menth09-1]. This is an important feature, as QoS violations in core networks due to link failures are more likely than QoS violations due to increased traffic volume [Iyer03]. o The PCN-metering behaviours only operate on the overall PCN- traffic on the link, not per flow. o The information of these measurements is signalled to the PCN- egress-nodes by the PCN-marks in the packet headers, ie, "in- band". No additional signalling protocol is required for transporting the PCN-marks. Therefore, no secure binding is required between data packets and separate congestion messages. o The PCN-egress-nodes make separate measurements, operating on the aggregate PCN-traffic from each PCN-ingress-node, ie, not per flow. Similarly, signalling by the PCN-egress-node of PCN- feedback-information (which is used for flow admission and termination decisions) is at the granularity of the ingress- egress-aggregate. An alternative approach is that the PCN-egress- nodes monitor the PCN-traffic and signal PCN-feedback-information (which is used for flow admission and termination decisions) at the granularity of one (or a few) PCN-marks. o The admitted PCN-load is controlled dynamically. Therefore, it adapts as the traffic matrix changes. It also adapts if the network topology changes (eg, after a link failure). Hence, an operator can be less conservative when deploying network capacity and less accurate in their prediction of the PCN-traffic matrix.
o The termination mechanism complements admission control. It
allows the network to recover from sudden unexpected surges of
PCN-traffic on some links, thus restoring QoS to the remaining
flows. Such scenarios are expected to be rare but not impossible.
They can be caused by large network failures that redirect lots of
admitted PCN-traffic to other links or by the malfunction of
measurement-based admission control in the presence of admitted
flows that send for a while with an atypically low rate and then
increase their rates in a correlated way.
o Flow termination can also enable an operator to be less
conservative when deploying network capacity. It is an
alternative to running links at low utilisation in order to
protect against link or node failures. This is especially the
case with SRLGs (shared risk link groups), which are links that
share a resource, such as a fibre, whose failure affects all links
in that group [RFC4216]). Fully protecting traffic against a
single SRLG failure requires low utilisation (~10%) of the link
bandwidth on some links before failure [Charny08].
o The PCN-supportable-rate may be set below the maximum rate that
PCN-traffic can be transmitted on a link in order to trigger the
termination of some PCN-flows before loss (or excessive delay) of
PCN-packets occurs, or to keep the maximum PCN-load on a link
below a level configured by the operator.
o Provisioning of the network is decoupled from the process of
adding new customers. By contrast, with the Diffserv architecture
[RFC2475], operators rely on subscription-time Service Level
Agreements, which statically define the parameters of the traffic
that will be accepted from a customer. This way, the operator has
to verify that provision is sufficient each time a new customer is
added to check that the Service Level Agreement can be fulfilled.
A PCN-domain doesn't need such traffic conditioning.
6.2. Deployment Scenarios
Operators of networks will want to use the PCN mechanisms in various
arrangements depending, for instance, on how they are performing
admission control outside the PCN-domain (users after all are
concerned about QoS end-to-end), what their particular goals and
assumptions are, how many PCN encoding states are available, and so
on.
A PCN-domain may have three encoding states (or pedantically, an
operator may choose to use up three encoding states for PCN): not
PCN-marked, threshold-marked, and excess-traffic-marked. This way,
both PCN admission control and flow termination can be supported. As
illustrated in Figure 1, admission control accepts new flows until the PCN-traffic rate on the bottleneck link rises above the PCN- threshold-rate, whilst, if necessary, the flow termination mechanism terminates flows down to the PCN-excess-rate on the bottleneck link. On the other hand, a PCN-domain may have two encoding states (as in [Moncaster09-1]) (or pedantically, an operator may choose to use up two encoding states for PCN): not PCN-marked and PCN-marked. This way, there are three possibilities, as discussed in the following paragraphs (see also Section 3.3). First, an operator could just use PCN's admission control, solving heavy congestion (caused by re-routing) by "just waiting" -- as sessions end, PCN-traffic naturally reduces; meanwhile, the admission control mechanism will prevent admission of new flows that use the affected links. So, the PCN-domain will naturally return to normal operation, but with reduced capacity. The drawback of this approach would be that, until sufficient sessions have ended to relieve the congestion, all PCN-flows as well as lower-priority services will be adversely affected. Second, an operator could just rely on statically provisioned capacity per PCN-ingress-node (regardless of the PCN-egress-node of a flow) for admission control, as is typical in the hose model of the Diffserv architecture [Kumar01]. Such traffic-conditioning agreements can lead to focused overload: many flows happen to focus on a particular link and then all flows through the congested link fail catastrophically. PCN's flow termination mechanism could then be used to counteract such a problem. Third, both admission control and flow termination can be triggered from the single type of PCN-marking; the main downside here is that admission control is less accurate [Charny07-2]. This possibility is illustrated in Figure 3. Within the PCN-domain, there is some flexibility about how the decision-making functionality is distributed. These possibilities are outlined in Section 4.4 and are also discussed elsewhere, such as in [Menth09-2]. The flow admission and termination decisions need to be enforced through per-flow policing by the PCN-ingress-nodes. If there are several PCN-domains on the end-to-end path, then each needs to police at its PCN-ingress-nodes. One exception is if the operator runs both the access network (not a PCN-domain) and the core network (a PCN- domain); per-flow policing could be devolved to the access network
and not be done at the PCN-ingress-node. Note that, to aid readability, the rest of this document assumes that policing is done by the PCN-ingress-nodes. PCN admission control has to fit with the overall approach to admission control. For instance, [Briscoe06] describes the case where RSVP signalling runs end-to-end. The PCN-domain is a single RSVP hop, ie, only the PCN-boundary-nodes process RSVP messages, with RSVP messages processed on each hop outside the PCN-domain, as in IntServ over Diffserv [RFC2998]. It would also be possible for the RSVP signalling to be originated and/or terminated by proxies, with application-layer signalling between the end user and the proxy (eg, SIP signalling with a home hub). A similar example would use NSIS (Next Steps in Signalling) [RFC3726] instead of RSVP. It is possible that a user wants its inelastic traffic to use the PCN mechanisms but also react to ECN markings outside the PCN-domain [Sarker08]. Two possible ways to do this are to tunnel all PCN- packets across the PCN-domain, so that the ECN marks are carried transparently across the PCN-domain, or to use an encoding like [Moncaster09-2]. Tunnelling is discussed further in Section 4.7. Some further possible deployment models are outlined in the Appendix.6.3. Assumptions and Constraints on Scope
The scope of this document is restricted by the following assumptions: 1. These components are deployed in a single Diffserv domain, within which all PCN-nodes are PCN-enabled and are trusted for truthful PCN-marking and transport. 2. All flows handled by these mechanisms are inelastic and constrained to a known peak rate through policing or shaping. 3. The number of PCN-flows across any potential bottleneck link is sufficiently large that stateless, statistical mechanisms can be effective. To put it another way, the aggregate bit rate of PCN- traffic across any potential bottleneck link needs to be sufficiently large, relative to the maximum additional bit rate added by one flow. This is the basic assumption of measurement- based admission control.
4. PCN-flows may have different precedence, but the applicability of
the PCN mechanisms for emergency use (911, GETS (Government
Telecommunications Service), WPS (Wireless Priority Service),
MLPP (Multilevel Precedence and Premption), etc.) is out of
scope.
6.3.1. Assumption 1: Trust and Support of PCN - Controlled Environment
It is assumed that the PCN-domain is a controlled environment, ie,
all the nodes in a PCN-domain run PCN and are trusted. There are
several reasons for this assumption:
o The PCN-domain has to be encircled by a ring of PCN-boundary-
nodes; otherwise, traffic could enter a PCN-BA without being
subject to admission control, which would potentially degrade the
QoS of existing PCN-flows.
o Similarly, a PCN-boundary-node has to trust that all the PCN-nodes
mark PCN-traffic consistently. A node not performing PCN-marking
wouldn't be able to send an alert when it suffered pre-congestion,
which potentially would lead to too many PCN-flows being admitted
(or too few being terminated). Worse, a rogue node could perform
various attacks, as discussed in Section 7.
One way of assuring the above two points are in effect is to have the
entire PCN-domain run by a single operator. Another way is to have
several operators that trust each other in their handling of PCN-
traffic.
Note: All PCN-nodes need to be trustworthy. However, if it is known
that an interface cannot become pre-congested, then it is not
strictly necessary for it to be capable of PCN-marking, but this must
be known even in unusual circumstances, eg, after the failure of some
links.
6.3.2. Assumption 2: Real-Time Applications
It is assumed that any variation of source bit rate is independent of
the level of pre-congestion. We assume that PCN-packets come from
real-time applications generating inelastic traffic, ie, sending
packets at the rate the codec produces them, regardless of the
availability of capacity [RFC4594]. Examples of such real-time
applications include voice and video requiring low delay, jitter, and
packet loss, the Controlled Load Service [RFC2211], and the Telephony
service class [RFC4594]. This assumption is to help focus the effort
where it looks like PCN would be most useful, ie, the sorts of
applications where per-flow QoS is a known requirement. In other words, we focus on PCN providing a benefit to inelastic traffic (PCN may or may not provide a benefit to other types of traffic). As a consequence, it is assumed that PCN-metering and PCN-marking is being applied to traffic scheduled with an expedited forwarding per- hop behaviour [RFC3246] or with a per-hop behaviour with similar characteristics.6.3.3. Assumption 3: Many Flows and Additional Load
It is assumed that there are many PCN-flows on any bottleneck link in the PCN-domain (or, to put it another way, the aggregate bit rate of PCN-traffic across any potential bottleneck link is sufficiently large, relative to the maximum additional bit rate added by one PCN- flow). Measurement-based admission control assumes that the present is a reasonable prediction of the future: the network conditions are measured at the time of a new flow request, but the actual network performance must be acceptable during the call some time later. One issue is that if there are only a few variable rate flows, then the aggregate traffic level may vary a lot, perhaps enough to cause some packets to get dropped. If there are many flows, then the aggregate traffic level should be statistically smoothed. How many flows is enough depends on a number of factors, such as the variation in each flow's rate, the total rate of PCN-traffic, and the size of the "safety margin" between the traffic level at which we start admission-marking and at which packets are dropped or significantly delayed. No explicit assumptions are made about how many PCN-flows are in each ingress-egress-aggregate. Performance-evaluation work may clarify whether it is necessary to make any additional assumptions on aggregation at the ingress-egress-aggregate level.6.3.4. Assumption 4: Emergency Use Out of Scope
PCN-flows may have different precedence, but the applicability of the PCN mechanisms for emergency use (911, GETS, WPS, MLPP, etc.) is out of scope for this document.6.4. Challenges
Prior work on PCN and similar mechanisms has led to a number of considerations about PCN's design goals (things PCN should be good at) and some issues that have been hard to solve in a fully satisfactory manner. Taken as a whole, PCN represents a list of
trade-offs (it is unlikely that they can all be 100% achieved) and perhaps a list of evaluation criteria to help an operator (or the IETF) decide between options. The following are open issues. They are mainly taken from [Briscoe06], which also describes some possible solutions. Note that some may be considered unimportant in general or in specific deployment scenarios, or by some operators. Note: Potential solutions are out of scope for this document. o ECMP (Equal Cost Multi-Path) Routing: The level of pre-congestion is measured on a specific ingress-egress-aggregate. However, if the PCN-domain runs ECMP, then traffic on this ingress-egress- aggregate may follow several different paths -- some of the paths could be pre-congested whilst others are not. There are three potential problems: 1. over-admission: a new flow is admitted (because the pre- congestion level measured by the PCN-egress-node is sufficiently diluted by unmarked packets from non-congested paths that a new flow is admitted), but its packets travel through a pre-congested PCN-node. 2. under-admission: a new flow is blocked (because the pre- congestion level measured by the PCN-egress-node is sufficiently increased by PCN-marked packets from pre- congested paths that a new flow is blocked), but its packets travel along an uncongested path. 3. ineffective termination: a flow is terminated but its path doesn't travel through the (pre-)congested router(s). Since flow termination is a "last resort", which protects the network should over-admission occur, this problem is probably more important to solve than the other two. o ECMP and Signalling: It is possible that, in a PCN-domain running ECMP, the signalling packets (eg, RSVP, NSIS) follow a different path than the data packets, which could matter if the signalling packets are used as probes. Whether this is an issue depends on which fields the ECMP algorithm uses; if the ECMP algorithm is restricted to the source and destination IP addresses, then it will not be an issue. ECMP and signalling interactions are a specific instance of a general issue for non-traditional routing combined with resource management along a path [Hancock02].
o Tunnelling: There are scenarios where tunnelling makes it
difficult to determine the path in the PCN-domain. The problem,
its impact, and the potential solutions are similar to those for
ECMP.
o Scenarios with only one tunnel endpoint in the PCN-domain: Such
scenarios may make it harder for the PCN-egress-node to gather
from the signalling messages (eg, RSVP, NSIS) the identity of the
PCN-ingress-node.
o Bi-Directional Sessions: Many applications have bi-directional
sessions -- hence, there are two microflows that should be
admitted (or terminated) as a pair -- for instance, a bi-
directional voice call only makes sense if microflows in both
directions are admitted. However, the PCN mechanisms concern
admission and termination of a single flow, and coordination of
the decision for both flows is a matter for the signalling
protocol and out of scope for PCN. One possible example would use
SIP pre-conditions. However, there are others.
o Global Coordination: PCN makes its admission decision based on
PCN-markings on a particular ingress-egress-aggregate. Decisions
about flows through a different ingress-egress-aggregate are made
independently. However, one can imagine network topologies and
traffic matrices where, from a global perspective, it would be
better to make a coordinated decision across all the ingress-
egress-aggregates for the whole PCN-domain. For example, to block
(or even terminate) flows on one ingress-egress-aggregate so that
more important flows through a different ingress-egress-aggregate
could be admitted. The problem may well be relatively
insignificant.
o Aggregate Traffic Characteristics: Even when the number of flows
is stable, the traffic level through the PCN-domain will vary
because the sources vary their traffic rates. PCN works best when
there is not too much variability in the total traffic level at a
PCN-node's interface (ie, in the aggregate traffic from all
sources). Too much variation means that a node may (at one
moment) not be doing any PCN-marking and then (at another moment)
drop packets because it is overloaded. This makes it hard to tune
the admission control scheme to stop admitting new flows at the
right time. Therefore, the problem is more likely with fewer,
burstier flows.
o Flash crowds and Speed of Reaction: PCN is a measurement-based
mechanism and so there is an inherent delay between packet marking
by PCN-interior-nodes and any admission control reaction at PCN-
boundary-nodes. For example, if a big burst of admission requests
potentially occurs in a very short space of time (eg, prompted by
a televote), they could all get admitted before enough PCN-marks
are seen to block new flows. In other words, any additional load
offered within the reaction time of the mechanism must not move
the PCN-domain directly from a no congestion state to overload.
This "vulnerability period" may have an impact at the signalling
level, for instance, QoS requests should be rate-limited to bound
the number of requests able to arrive within the vulnerability
period.
o Silent at Start: After a successful admission request, the source
may wait some time before sending data (eg, waiting for the called
party to answer). Then the risk is that, in some circumstances,
PCN's measurements underestimate what the pre-congestion level
will be when the source does start sending data.
(page 40 continued on part 3)
