RFC 3168
The Addition of Explicit Congestion Notification (ECN) to IP
Pages: 63
Proposed Standard
→ Errata
Obsoletes: 2481
Updates: 2003 2474 2401 0793
Updated by: 4301 6040 8311
Proposed Standard
→ Errata
Obsoletes: 2481
Updates: 2003 2474 2401 0793
Updated by: 4301 6040 8311
Part 3 of 3 – Pages 38 to 63
12. Summary of changes required in IP and TCP
This document specified two bits in the IP header to be used for ECN. The not-ECT codepoint indicates that the transport protocol will ignore the CE codepoint. This is the default value for the ECN codepoint. The ECT codepoints indicate that the transport protocol is willing and able to participate in ECN. The router sets the CE codepoint to indicate congestion to the end nodes. The CE codepoint in a packet header MUST NOT be reset by a router. TCP requires three changes for ECN, a setup phase and two new flags in the TCP header. The ECN-Echo flag is used by the data receiver to inform the data sender of a received CE packet. The Congestion Window Reduced (CWR) flag is used by the data sender to inform the data receiver that the congestion window has been reduced.
When ECN (Explicit Congestion Notification) is used, it is required that congestion indications generated within an IP tunnel not be lost at the tunnel egress. We specified a minor modification to the IP protocol's handling of the ECN field during encapsulation and de- capsulation to allow flows that will undergo IP tunneling to use ECN. Two options for ECN in tunnels were specified: 1) A limited-functionality option that does not use ECN inside the IP tunnel, by setting the ECN field in the outer header to not-ECT, and not altering the inner header at the time of decapsulation. 2) The full-functionality option, which sets the ECN field in the outer header to either not-ECT or to one of the ECT codepoints, depending on the ECN field in the inner header. At decapsulation, if the CE codepoint is set in the outer header, and the inner header is set to one of the ECT codepoints, then the CE codepoint is copied to the inner header. For IPsec tunnels, this document also defines an optional IPsec Security Association (SA) attribute that enables negotiation of ECN usage within IPsec tunnels and an optional field in the Security Association Database to indicate whether ECN is permitted in tunnel mode on a SA. The required changes to IPsec tunnels for ECN usage modify RFC 2401 [RFC2401], which defines the IPsec architecture and specifies some aspects of its implementation. The new IPsec SA attribute is in addition to those already defined in Section 4.5 of [RFC2407]. This document obsoletes RFC 2481, "A Proposal to add Explicit Congestion Notification (ECN) to IP", which defined ECN as an Experimental Protocol for the Internet Community. The rest of this section describes the relationship between this document and its predecessor. RFC 2481 included a brief discussion of the use of ECN with encapsulated packets, and noted that for the IPsec specifications at the time (January 1999), flows could not safely use ECN if they were to traverse IPsec tunnels. RFC 2481 also described the changes that could be made to IPsec tunnel specifications to made them compatible with ECN. This document also incorporates work that was done after RFC 2481. First was to describe the changes to IPsec tunnels in detail, and extensively discuss the security implications of ECN (now included as Sections 18 and 19 of this document). Second was to extend the discussion of IPsec tunnels to include all IP tunnels. Because older IP tunnels are not compatible with a flow's use of ECN, the
deployment of ECN in the Internet will create strong pressure for older IP tunnels to be updated to an ECN-compatible version, using either the limited-functionality or the full-functionality option. This document does not address the issue of including ECN in non-IP tunnels such as MPLS, GRE, L2TP, or PPTP. An earlier preliminary document about adding ECN support to MPLS was not advanced. A third new piece of work after RFC2481 was to describe the ECN procedure with retransmitted data packets, that an ECT codepoint should not be set on retransmitted data packets. The motivation for this additional specification is to eliminate a possible avenue for denial-of-service attacks on an existing TCP connection. Some prior deployments of ECN-capable TCP might not conform to the (new) requirement not to set an ECT codepoint on retransmitted packets; we do not believe this will cause significant problems in practice. This document also expands slightly on the specification of the use of SYN packets for the negotiation of ECN. While some prior deployments of ECN-capable TCP might not conform to the requirements specified in this document, we do not believe that this will lead to any performance or compatibility problems for TCP connections with a combination of TCP implementations at the endpoints. This document also includes the specification of the ECT(1) codepoint, which may be used by TCP as part of the implementation of an ECN nonce.13. Conclusions
Given the current effort to implement AQM, we believe this is the right time to deploy congestion avoidance mechanisms that do not depend on packet drops alone. With the increased deployment of applications and transports sensitive to the delay and loss of a single packet (e.g., realtime traffic, short web transfers), depending on packet loss as a normal congestion notification mechanism appears to be insufficient (or at the very least, non- optimal). We examined the consequence of modifications of the ECN field within the network, analyzing all the opportunities for an adversary to change the ECN field. In many cases, the change to the ECN field is no worse than dropping a packet. However, we noted that some changes have the more serious consequence of subverting end-to-end congestion control. However, we point out that even then the potential damage is limited, and is similar to the threat posed by end-systems intentionally failing to cooperate with end-to-end congestion control.
14. Acknowledgements
Many people have made contributions to this work and this document, including many that we have not managed to directly acknowledge in this document. In addition, we would like to thank Kenjiro Cho for the proposal for the TCP mechanism for negotiating ECN-Capability, Kevin Fall for the proposal of the CWR bit, Steve Blake for material on IPv4 Header Checksum Recalculation, Jamal Hadi-Salim for discussions of ECN issues, and Steve Bellovin, Jim Bound, Brian Carpenter, Paul Ferguson, Stephen Kent, Greg Minshall, and Vern Paxson for discussions of security issues. We also thank the Internet End-to-End Research Group for ongoing discussions of these issues. Email discussions with a number of people, including Dax Kelson, Alexey Kuznetsov, Jamal Hadi-Salim, and Venkat Venkatsubra, have addressed the issues raised by non-conformant equipment in the Internet that does not respond to TCP SYN packets with the ECE and CWR flags set. We thank Mark Handley, Jitentra Padhye, and others for discussions on the TCP initialization procedures. The discussion of ECN and IP tunnel considerations draws heavily on related discussions and documents from the Differentiated Services Working Group. We thank Tabassum Bint Haque from Dhaka, Bangladesh, for feedback on IP tunnels. We thank Derrell Piper and Kero Tivinen for proposing modifications to RFC 2407 that improve the usability of negotiating the ECN Tunnel SA attribute. We thank David Wetherall, David Ely, and Neil Spring for the proposal for the ECN nonce. We also thank Stefan Savage for discussions on this issue. We thank Bob Briscoe and Jon Crowcroft for raising the issue of fragmentation in IP, on alternate semantics for the fourth ECN codepoint, and several other topics. We thank Richard Wendland for feedback on several issues in the document. We also thank the IESG, and in particular the Transport Area Directors over the years, for their feedback and their work towards the standardization of ECN.15. References
[AH] Kent, S. and R. Atkinson, "IP Authentication Header", RFC 2402, November 1998. [ECN] "The ECN Web Page", URL "http://www.aciri.org/floyd/ecn.html". Reference for informational purposes only.
[ESP] Kent, S. and R. Atkinson, "IP Encapsulating Security Payload", RFC 2406, November 1998. [FIXES] ECN-under-Linux Unofficial Vendor Support Page, URL "http://gtf.org/garzik/ecn/". Reference for informational purposes only. [FJ93] Floyd, S., and Jacobson, V., "Random Early Detection gateways for Congestion Avoidance", IEEE/ACM Transactions on Networking, V.1 N.4, August 1993, p. 397-413. [Floyd94] Floyd, S., "TCP and Explicit Congestion Notification", ACM Computer Communication Review, V. 24 N. 5, October 1994, p. 10-23. [Floyd98] Floyd, S., "The ECN Validation Test in the NS Simulator", URL "http://www-mash.cs.berkeley.edu/ns/", test tcl/test/test-all- ecn. Reference for informational purposes only. [FF99] Floyd, S., and Fall, K., "Promoting the Use of End-to- End Congestion Control in the Internet", IEEE/ACM Transactions on Networking, August 1999. [FRED] Lin, D., and Morris, R., "Dynamics of Random Early Detection", SIGCOMM '97, September 1997. [GRE] Hanks, S., Li, T., Farinacci, D. and P. Traina, "Generic Routing Encapsulation (GRE)", RFC 1701, October 1994. [Jacobson88] V. Jacobson, "Congestion Avoidance and Control", Proc. ACM SIGCOMM '88, pp. 314-329. [Jacobson90] V. Jacobson, "Modified TCP Congestion Avoidance Algorithm", Message to end2end-interest mailing list, April 1990. URL "ftp://ftp.ee.lbl.gov/email/vanj.90apr30.txt". [K98] Krishnan, H., "Analyzing Explicit Congestion Notification (ECN) benefits for TCP", Master's thesis, UCLA, 1998. Citation for acknowledgement purposes only. [L2TP] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, G. and B. Palter, "Layer Two Tunneling Protocol "L2TP"", RFC 2661, August 1999.
[MJV96] S. McCanne, V. Jacobson, and M. Vetterli, "Receiver- driven Layered Multicast", SIGCOMM '96, August 1996, pp. 117-130. [MPLS] Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M. and J. McManus, Requirements for Traffic Engineering Over MPLS, RFC 2702, September 1999. [PPTP] Hamzeh, K., Pall, G., Verthein, W., Taarud, J., Little, W. and G. Zorn, "Point-to-Point Tunneling Protocol (PPTP)", RFC 2637, July 1999. [RFC791] Postel, J., "Internet Protocol", STD 5, RFC 791, September 1981. [RFC793] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981. [RFC1141] Mallory, T. and A. Kullberg, "Incremental Updating of the Internet Checksum", RFC 1141, January 1990. [RFC1349] Almquist, P., "Type of Service in the Internet Protocol Suite", RFC 1349, July 1992. [RFC1455] Eastlake, D., "Physical Link Security Type of Service", RFC 1455, May 1993. [RFC1701] Hanks, S., Li, T., Farinacci, D. and P. Traina, "Generic Routing Encapsulation (GRE)", RFC 1701, October 1994. [RFC1702] Hanks, S., Li, T., Farinacci, D. and P. Traina, "Generic Routing Encapsulation over IPv4 networks", RFC 1702, October 1994. [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, October 1996. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC2309] Braden, B., et al., "Recommendations on Queue Management and Congestion Avoidance in the Internet", RFC 2309, April 1998. [RFC2401] Kent, S. and R. Atkinson, Security Architecture for the Internet Protocol, RFC 2401, November 1998.
[RFC2407] Piper, D., "The Internet IP Security Domain of Interpretation for ISAKMP", RFC 2407, November 1998. [RFC2408] Maughan, D., Schertler, M., Schneider, M. and J. Turner, "Internet Security Association and Key Management Protocol (ISAKMP)", RFC 2409, November 1998. [RFC2409] Harkins D. and D. Carrel, "The Internet Key Exchange (IKE)", RFC 2409, November 1998. [RFC2474] Nichols, K., Blake, S., Baker, F. and D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, December 1998. [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z. and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, December 1998. [RFC2481] Ramakrishnan K. and S. Floyd, "A Proposal to add Explicit Congestion Notification (ECN) to IP", RFC 2481, January 1999. [RFC2581] Alman, M., Paxson, V. and W. Stevens, "TCP Congestion Control", RFC 2581, April 1999. [RFC2884] Hadi Salim, J. and U. Ahmed, "Performance Evaluation of Explicit Congestion Notification (ECN) in IP Networks", RFC 2884, July 2000. [RFC2983] Black, D., "Differentiated Services and Tunnels", RFC2983, October 2000. [RFC2780] Bradner S. and V. Paxson, "IANA Allocation Guidelines For Values In the Internet Protocol and Related Headers", BCP 37, RFC 2780, March 2000. [RJ90] K. K. Ramakrishnan and Raj Jain, "A Binary Feedback Scheme for Congestion Avoidance in Computer Networks", ACM Transactions on Computer Systems, Vol.8, No.2, pp. 158-181, May 1990. [SCWA99] Stefan Savage, Neal Cardwell, David Wetherall, and Tom Anderson, TCP Congestion Control with a Misbehaving Receiver, ACM Computer Communications Review, October 1999.
[TBIT] Jitendra Padhye and Sally Floyd, "Identifying the TCP Behavior of Web Servers", ICSI TR-01-002, February 2001. URL "http://www.aciri.org/tbit/".16. Security Considerations
Security considerations have been discussed in Sections 7, 8, 18, and 19.17. IPv4 Header Checksum Recalculation
IPv4 header checksum recalculation is an issue with some high-end router architectures using an output-buffered switch, since most if not all of the header manipulation is performed on the input side of the switch, while the ECN decision would need to be made local to the output buffer. This is not an issue for IPv6, since there is no IPv6 header checksum. The IPv4 TOS octet is the last byte of a 16-bit half-word. RFC 1141 [RFC1141] discusses the incremental updating of the IPv4 checksum after the TTL field is decremented. The incremental updating of the IPv4 checksum after the CE codepoint was set would work as follows: Let HC be the original header checksum for an ECT(0) packet, and let HC' be the new header checksum after the CE bit has been set. That is, the ECN field has changed from '10' to '11'. Then for header checksums calculated with one's complement subtraction, HC' would be recalculated as follows: HC' = { HC - 1 HC > 1 { 0x0000 HC = 1 For header checksums calculated on two's complement machines, HC' would be recalculated as follows after the CE bit was set: HC' = { HC - 1 HC > 0 { 0xFFFE HC = 0 A similar incremental updating of the IPv4 checksum can be carried out when the ECN field is changed from ECT(1) to CE, that is, from ' 01' to '11'.18. Possible Changes to the ECN Field in the Network
This section discusses in detail possible changes to the ECN field in the network, such as falsely reporting congestion, disabling ECN- Capability for an individual packet, erasing the ECN congestion indication, or falsely indicating ECN-Capability.
18.1. Possible Changes to the IP Header
18.1.1. Erasing the Congestion Indication
First, we consider the changes that a router could make that would result in effectively erasing the congestion indication after it had been set by a router upstream. The convention followed is: ECN codepoint of received packet -> ECN codepoint of packet transmitted. Replacing the CE codepoint with the ECT(0) or ECT(1) codepoint effectively erases the congestion indication. However, with the use of two ECT codepoints, a router erasing the CE codepoint has no way to know whether the original ECT codepoint was ECT(0) or ECT(1). Thus, it is possible for the transport protocol to deploy mechanisms to detect such erasures of the CE codepoint. The consequence of the erasure of the CE codepoint for the upstream router is that there is a potential for congestion to build for a time, because the congestion indication does not reach the source. However, the packet would be received and acknowledged. The potential effect of erasing the congestion indication is complex, and is discussed in depth in Section 19 below. Note that the effect of erasing the congestion indication is different from dropping a packet in the network. When a data packet is dropped, the drop is detected by the TCP sender, and interpreted as an indication of congestion. Similarly, if a sufficient number of consecutive acknowledgement packets are dropped, causing the cumulative acknowledgement field not to be advanced at the sender, the sender is limited by the congestion window from sending additional packets, and ultimately the retransmit timer expires. In contrast, a systematic erasure of the CE bit by a downstream router can have the effect of causing a queue buildup at an upstream router, including the possible loss of packets due to buffer overflow. There is a potential of unfairness in that another flow that goes through the congested router could react to the CE bit set while the flow that has the CE bit erased could see better performance. The limitations on this potential unfairness are discussed in more detail in Section 19 below. The last of the three changes is to replace the CE codepoint with the not-ECT codepoint, thus erasing the congestion indication and disabling ECN-Capability at the same time. The `erasure' of the congestion indication is only effective if the packet does not end up being marked or dropped again by a downstream router. If the CE codepoint is replaced by an ECT codepoint, the
packet remains ECN-Capable, and could be either marked or dropped by a downstream router as an indication of congestion. If the CE codepoint is replaced by the not-ECT codepoint, the packet is no longer ECN-capable, and can therefore be dropped but not marked by a downstream router as an indication of congestion.18.1.2. Falsely Reporting Congestion
This change is to set the CE codepoint when an ECT codepoint was already set, even though there was no congestion. This change does not affect the treatment of that packet along the rest of the path. In particular, a router does not examine the CE codepoint in deciding whether to drop or mark an arriving packet. However, this could result in the application unnecessarily invoking end-to-end congestion control, and reducing its arrival rate. By itself, this is no worse (for the application or for the network) than if the tampering router had actually dropped the packet.18.1.3. Disabling ECN-Capability
This change is to turn off the ECT codepoint of a packet. This means that if the packet later encounters congestion (e.g., by arriving to a RED queue with a moderate average queue size), it will be dropped instead of being marked. By itself, this is no worse (for the application) than if the tampering router had actually dropped the packet. The saving grace in this particular case is that there is no congested router upstream expecting a reaction from setting the CE bit.18.1.4. Falsely Indicating ECN-Capability
This change would incorrectly label a packet as ECN-Capable. The packet may have been sent either by an ECN-Capable transport or a transport that is not ECN-Capable. If the packet later encounters moderate congestion at an ECN-Capable router, the router could set the CE codepoint instead of dropping the packet. If the transport protocol in fact is not ECN-Capable, then the transport will never receive this indication of congestion, and will not reduce its sending rate in response. The potential consequences of falsely indicating ECN-capability are discussed further in Section 19 below. If the packet never later encounters congestion at an ECN-Capable router, then the first of these two changes would have no effect, other than possibly interfering with the use of the ECN nonce by the transport protocol. The last change, however, would have the effect
of giving false reports of congestion to a monitoring device along the path. If the transport protocol is ECN-Capable, then this change could also have an effect at the transport level, by combining falsely indicating ECN-Capability with falsely reporting congestion. For an ECN-capable transport, this would cause the transport to unnecessarily react to congestion. In this particular case, the router that is incorrectly changing the ECN field could have dropped the packet. Thus for this case of an ECN-capable transport, the consequence of this change to the ECN field is no worse than dropping the packet.18.2. Information carried in the Transport Header
For TCP, an ECN-capable TCP receiver informs its TCP peer that it is ECN-capable at the TCP level, conveying this information in the TCP header at the time the connection is setup. This document does not consider potential dangers introduced by changes in the transport header within the network. We note that when IPsec is used, the transport header is protected both in tunnel and transport modes [ESP, AH]. Another issue concerns TCP packets with a spoofed IP source address carrying invalid ECN information in the transport header. For completeness, we examine here some possible ways that a node spoofing the IP source address of another node could use the two ECN flags in the TCP header to launch a denial-of-service attack. However, these attacks would require an ability for the attacker to use valid TCP sequence numbers, and any attacker with this ability and with the ability to spoof IP source addresses could damage the TCP connection without using the ECN flags. Therefore, ECN does not add any new vulnerabilities in this respect. An acknowledgement packet with a spoofed IP source address of the TCP data receiver could include the ECE bit set. If accepted by the TCP data sender as a valid packet, this spoofed acknowledgement packet could result in the TCP data sender unnecessarily halving its congestion window. However, to be accepted by the data sender, such a spoofed acknowledgement packet would have to have the correct 32- bit sequence number as well as a valid acknowledgement number. An attacker that could successfully send such a spoofed acknowledgement packet could also send a spoofed RST packet, or do other equally damaging operations to the TCP connection. Packets with a spoofed IP source address of the TCP data sender could include the CWR bit set. Again, to be accepted, such a packet would have to have a valid sequence number. In addition, such a spoofed packet would have a limited performance impact. Spoofing a data packet with the CWR bit set could result in the TCP data receiver
sending fewer ECE packets than it would otherwise, if the data receiver was sending ECE packets when it received the spoofed CWR packet.18.3. Split Paths
In some cases, a malicious or broken router might have access to only a subset of the packets from a flow. The question is as follows: can this router, by altering the ECN field in this subset of the packets, do more damage to that flow than if it had simply dropped that set of packets? We will classify the packets in the flow as A packets and B packets, and assume that the adversary only has access to A packets. Assume that the adversary is subverting end-to-end congestion control along the path traveled by A packets only, by either falsely indicating ECN-Capability upstream of the point where congestion occurs, or erasing the congestion indication downstream. Consider also that there exists a monitoring device that sees both the A and B packets, and will "punish" both the A and B packets if the total flow is determined not to be properly responding to indications of congestion. Another key characteristic that we believe is likely to be true is that the monitoring device, before `punishing' the A&B flow, will first drop packets instead of setting the CE codepoint, and will drop arriving packets of that flow that already have the CE codepoint set. If the end nodes are in fact using end-to-end congestion control, they will see all of the indications of congestion seen by the monitoring device, and will begin to respond to these indications of congestion. Thus, the monitoring device is successful in providing the indications to the flow at an early stage. It is true that the adversary that has access only to the A packets might, by subverting ECN-based congestion control, be able to deny the benefits of ECN to the other packets in the A&B aggregate. While this is unfortunate, this is not a reason to disable ECN. A variant of falsely reporting congestion occurs when there are two adversaries along a path, where the first adversary falsely reports congestion, and the second adversary `erases' those reports. (Unlike packet drops, ECN congestion reports can be `reversed' later in the network by a malicious or broken router. However, the use of the ECN nonce could help the transport to detect this behavior.) While this would be transparent to the end node, it is possible that a monitoring device between the first and second adversaries would see the false indications of congestion. Keep in mind our recommendation in this document, that before `punishing' a flow for not responding appropriately to congestion, the router will first switch to dropping
rather than marking as an indication of congestion, for that flow. When this includes dropping arriving packets from that flow that have the CE codepoint set, this ensures that these indications of congestion are being seen by the end nodes. Thus, there is no additional harm that we are able to postulate as a result of multiple conflicting adversaries.19. Implications of Subverting End-to-End Congestion Control
This section focuses on the potential repercussions of subverting end-to-end congestion control by either falsely indicating ECN- Capability, or by erasing the congestion indication in ECN (the CE codepoint). Subverting end-to-end congestion control by either of these two methods can have consequences both for the application and for the network. We discuss these separately below. The first method to subvert end-to-end congestion control, that of falsely indicating ECN-Capability, effectively subverts end-to-end congestion control only if the packet later encounters congestion that results in the setting of the CE codepoint. In this case, the transport protocol (which may not be ECN-capable) does not receive the indication of congestion from these downstream congested routers. The second method to subvert end-to-end congestion control, `erasing' the CE codepoint in a packet, effectively subverts end-to-end congestion control only when the CE codepoint in the packet was set earlier by a congested router. In this case, the transport protocol does not receive the indication of congestion from the upstream congested routers. Either of these two methods of subverting end-to-end congestion control can potentially introduce more damage to the network (and possibly to the flow itself) than if the adversary had simply dropped packets from that flow. However, as we discuss later in this section and in Section 7, this potential damage is limited.19.1. Implications for the Network and for Competing Flows
The CE codepoint of the ECN field is only used by routers as an indication of congestion during periods of *moderate* congestion. ECN-capable routers should drop rather than mark packets during heavy congestion even if the router's queue is not yet full. For example, for routers using active queue management based on RED, the router should drop rather than mark packets that arrive while the average queue sizes exceed the RED queue's maximum threshold.
One consequence for the network of subverting end-to-end congestion control is that flows that do not receive the congestion indications from the network might increase their sending rate until they drive the network into heavier congestion. Then, the congested router could begin to drop rather than mark arriving packets. For flows that are not isolated by some form of per-flow scheduling or other per-flow mechanisms, but are instead aggregated with other flows in a single queue in an undifferentiated fashion, this packet-dropping at the congested router would apply to all flows that share that queue. Thus, the consequences would be to increase the level of congestion in the network. In some cases, the increase in the level of congestion will lead to a substantial buffer buildup at the congested queue that will be sufficient to drive the congested queue from the packet-marking to the packet-dropping regime. This transition could occur either because of buffer overflow, or because of the active queue management policy described above that drops packets when the average queue is above RED's maximum threshold. At this point, all flows, including the subverted flow, will begin to see packet drops instead of packet marks, and a malicious or broken router will no longer be able to ` erase' these indications of congestion in the network. If the end nodes are deploying appropriate end-to-end congestion control, then the subverted flow will reduce its arrival rate in response to congestion. When the level of congestion is sufficiently reduced, the congested queue can return from the packet-dropping regime to the packet-marking regime. The steady-state pattern could be one of the congested queue oscillating between these two regimes. In other cases, the consequences of subverting end-to-end congestion control will not be severe enough to drive the congested link into sufficiently-heavy congestion that packets are dropped instead of being marked. In this case, the implications for competing flows in the network will be a slightly-increased rate of packet marking or dropping, and a corresponding decrease in the bandwidth available to those flows. This can be a stable state if the arrival rate of the subverted flow is sufficiently small, relative to the link bandwidth, that the average queue size at the congested router remains under control. In particular, the subverted flow could have a limited bandwidth demand on the link at this router, while still getting more than its "fair" share of the link. This limited demand could be due to a limited demand from the data source; a limitation from the TCP advertised window; a lower-bandwidth access pipe; or other factors. Thus the subversion of ECN-based congestion control can still lead to unfairness, which we believe is appropriate to note here.
The threat to the network posed by the subversion of ECN-based congestion control in the network is essentially the same as the threat posed by an end-system that intentionally fails to cooperate with end-to-end congestion control. The deployment of mechanisms in routers to address this threat is an open research question, and is discussed further in Section 10. Let us take the example described in Section 18.1.1, where the CE codepoint that was set in a packet is erased: {'11' -> '10' or '11' -> '01'}. The consequence for the congested upstream router that set the CE codepoint is that this congestion indication does not reach the end nodes for that flow. The source (even one which is completely cooperative and not malicious) is thus allowed to continue to increase its sending rate (if it is a TCP flow, by increasing its congestion window). The flow potentially achieves better throughput than the other flows that also share the congested router, especially if there are no policing mechanisms or per-flow queuing mechanisms at that router. Consider the behavior of the other flows, especially if they are cooperative: that is, the flows that do not experience subverted end-to-end congestion control. They are likely to reduce their load (e.g., by reducing their window size) on the congested router, thus benefiting our subverted flow. This results in unfairness. As we discussed above, this unfairness could either be transient (because the congested queue is driven into the packet- marking regime), oscillatory (because the congested queue oscillates between the packet marking and the packet dropping regime), or more moderate but a persistent stable state (because the congested queue is never driven to the packet dropping regime). The results would be similar if the subverted flow was intentionally avoiding end-to-end congestion control. One difference is that a flow that is intentionally avoiding end-to-end congestion control at the end nodes can avoid end-to-end congestion control even when the congested queue is in packet-dropping mode, by refusing to reduce its sending rate in response to packet drops in the network. Thus the problems for the network from the subversion of ECN-based congestion control are less severe than the problems caused by the intentional avoidance of end-to-end congestion control in the end nodes. It is also the case that it is considerably more difficult to control the behavior of the end nodes than it is to control the behavior of the infrastructure itself. This is not to say that the problems for the network posed by the network's subversion of ECN-based congestion control are small; just that they are dwarfed by the problems for the network posed by the subversion of either ECN-based or other currently known packet-based congestion control mechanisms by the end nodes.
19.2. Implications for the Subverted Flow
When a source indicates that it is ECN-capable, there is an expectation that the routers in the network that are capable of participating in ECN will use the CE codepoint for indication of congestion. There is the potential benefit of using ECN in reducing the amount of packet loss (in addition to the reduced queuing delays because of active queue management policies). When the packet flows through an IPsec tunnel where the nodes that the tunneled packets traverse are untrusted in some way, the expectation is that IPsec will protect the flow from subversion that results in undesirable consequences. In many cases, a subverted flow will benefit from the subversion of end-to-end congestion control for that flow in the network, by receiving more bandwidth than it would have otherwise, relative to competing non-subverted flows. If the congested queue reaches the packet-dropping stage, then the subversion of end-to-end congestion control might or might not be of overall benefit to the subverted flow, depending on that flow's relative tradeoffs between throughput, loss, and delay. One form of subverting end-to-end congestion control is to falsely indicate ECN-capability by setting the ECT codepoint. This has the consequence of downstream congested routers setting the CE codepoint in vain. However, as described in Section 9.1.2, if an ECT codepoint is changed in an IP tunnel, this can be detected at the egress point of the tunnel, as long as the inner header was not changed within the tunnel. The second form of subverting end-to-end congestion control is to erase the congestion indication by erasing the CE codepoint. In this case, it is the upstream congested routers that set the CE codepoint in vain. If an ECT codepoint is erased within an IP tunnel, then this can be detected at the egress point of the tunnel, as long as the inner header was not changed within the tunnel. If the CE codepoint is set upstream of the IP tunnel, then any erasure of the outer header's CE codepoint within the tunnel will have no effect because the inner header preserves the set value of the CE codepoint. However, if the CE codepoint is set within the tunnel, and erased either within or downstream of the tunnel, this is not necessarily detected at the egress point of the tunnel. With this subversion of end-to-end congestion control, an end-system transport does not respond to the congestion indication. Along with the increased unfairness for the non-subverted flows described in the
previous section, the congested router's queue could continue to build, resulting in packet loss at the congested router - which is a means for indicating congestion to the transport in any case. In the interim, the flow might experience higher queuing delays, possibly along with an increased bandwidth relative to other non-subverted flows. But transports do not inherently make assumptions of consistently experiencing carefully managed queuing in the path. We believe that these forms of subverting end-to-end congestion control are no worse for the subverted flow than if the adversary had simply dropped the packets of that flow itself.19.3. Non-ECN-Based Methods of Subverting End-to-end Congestion Control
We have shown that, in many cases, a malicious or broken router that is able to change the bits in the ECN field can do no more damage than if it had simply dropped the packet in question. However, this is not true in all cases, in particular in the cases where the broken router subverted end-to-end congestion control by either falsely indicating ECN-Capability or by erasing the ECN congestion indication (in the CE codepoint). While there are many ways that a router can harm a flow by dropping packets, a router cannot subvert end-to-end congestion control by dropping packets. As an example, a router cannot subvert TCP congestion control by dropping data packets, acknowledgement packets, or control packets. Even though packet-dropping cannot be used to subvert end-to-end congestion control, there *are* non-ECN-based methods for subverting end-to-end congestion control that a broken or malicious router could use. For example, a broken router could duplicate data packets, thus effectively negating the effects of end-to-end congestion control along some portion of the path. (For a router that duplicated packets within an IPsec tunnel, the security administrator can cause the duplicate packets to be discarded by configuring anti-replay protection for the tunnel.) This duplication of packets within the network would have similar implications for the network and for the subverted flow as those described in Sections 18.1.1 and 18.1.4 above.20. The Motivation for the ECT Codepoints.
20.1. The Motivation for an ECT Codepoint.
The need for an ECT codepoint is motivated by the fact that ECN will be deployed incrementally in an Internet where some transport protocols and routers understand ECN and some do not. With an ECT codepoint, the router can drop packets from flows that are not ECN- capable, but can *instead* set the CE codepoint in packets that *are*
ECN-capable. Because an ECT codepoint allows an end node to have the CE codepoint set in a packet *instead* of having the packet dropped, an end node might have some incentive to deploy ECN. If there was no ECT codepoint, then the router would have to set the CE codepoint for packets from both ECN-capable and non-ECN-capable flows. In this case, there would be no incentive for end-nodes to deploy ECN, and no viable path of incremental deployment from a non- ECN world to an ECN-capable world. Consider the first stages of such an incremental deployment, where a subset of the flows are ECN- capable. At the onset of congestion, when the packet dropping/marking rate would be low, routers would only set CE codepoints, rather than dropping packets. However, only those flows that are ECN-capable would understand and respond to CE packets. The result is that the ECN-capable flows would back off, and the non- ECN-capable flows would be unaware of the ECN signals and would continue to open their congestion windows. In this case, there are two possible outcomes: (1) the ECN-capable flows back off, the non-ECN-capable flows get all of the bandwidth, and congestion remains mild, or (2) the ECN-capable flows back off, the non-ECN-capable flows don't, and congestion increases until the router transitions from setting the CE codepoint to dropping packets. While this second outcome evens out the fairness, the ECN-capable flows would still receive little benefit from being ECN-capable, because the increased congestion would drive the router to packet- dropping behavior. A flow that advertised itself as ECN-Capable but does not respond to CE codepoints is functionally equivalent to a flow that turns off congestion control, as discussed earlier in this document. Thus, in a world when a subset of the flows are ECN-capable, but where ECN-capable flows have no mechanism for indicating that fact to the routers, there would be less effective and less fair congestion control in the Internet, resulting in a strong incentive for end nodes not to deploy ECN.20.2. The Motivation for two ECT Codepoints.
The primary motivation for the two ECT codepoints is to provide a one-bit ECN nonce. The ECN nonce allows the development of mechanisms for the sender to probabilistically verify that network elements are not erasing the CE codepoint, and that data receivers are properly reporting to the sender the receipt of packets with the CE codepoint set.
Another possibility for senders to detect misbehaving network elements or receivers would be for the data sender to occasionally send a data packet with the CE codepoint set, to see if the receiver reports receiving the CE codepoint. Of course, if these packets encountered congestion in the network, the router might make no change in the packets, because the CE codepoint would already be set. Thus, for packets sent with the CE codepoint set, the TCP end-nodes could not determine if some router intended to set the CE codepoint in these packets. For this reason, sending packets with the CE codepoint would have to be done sparingly, and would be a less effective check against misbehaving network elements and receivers than would be the ECN nonce. The assignment of the fourth ECN codepoint to ECT(1) precludes the use of this codepoint for some other purposes. For clarity, we briefly list other possible purposes here. One possibility might have been for the data sender to use the fourth ECN codepoint to indicate an alternate semantics for ECN. However, this seems to us more appropriate to be signaled using a differentiated services codepoint in the DS field. A second possible use for the fourth ECN codepoint would have been to give the router two separate codepoints for the indication of congestion, CE(0) and CE(1), for mild and severe congestion respectively. While this could be useful in some cases, this certainly does not seem a compelling requirement at this point. If there was judged to be a compelling need for this, the complications of incremental deployment would most likely necessitate more that just one codepoint for this function. A third use that has been informally proposed for the ECN codepoint is for use in some forms of multicast congestion control, based on randomized procedures for duplicating marked packets at routers. Some proposed multicast packet duplication procedures are based on a new ECN codepoint that (1) conveys the fact that congestion occurred upstream of the duplication point that marked the packet with this codepoint and (2) can detect congestion downstream of that duplication point. ECT(1) can serve this purpose because it is both distinct from ECT(0) and is replaced by CE when ECN marking occurs in response to congestion or incipient congestion. Explanation of how this enhanced version of ECN would be used by multicast congestion control is beyond the scope of this document, as are ECN-aware multicast packet duplication procedures and the processing of the ECN field at multicast receivers in all cases (i.e., irrespective of the multicast packet duplication procedure(s) used).
The specification of IP tunnel modifications for ECN in this document assumes that the only change made to the outer IP header's ECN field between tunnel endpoints is to set the CE codepoint to indicate congestion. This is not consistent with some of the proposed uses of ECT(1) by the multicast duplication procedures in the previous paragraph, and such procedures SHOULD NOT be deployed unless this inconsistency between multicast duplication procedures and IP tunnels with full ECN functionality is resolved. Limited ECN functionality may be used instead, although in practice many tunnel protocols (including IPsec) will not work correctly if multicast traffic duplication occurs within the tunnel21. Why use Two Bits in the IP Header?
Given the need for an ECT indication in the IP header, there still remains the question of whether the ECT (ECN-Capable Transport) and CE (Congestion Experienced) codepoints should have been overloaded on a single bit. This overloaded-one-bit alternative, explored in [Floyd94], would have involved a single bit with two values. One value, "ECT and not CE", would represent an ECN-Capable Transport, and the other value, "CE or not ECT", would represent either Congestion Experienced or a non-ECN-Capable transport. One difference between the one-bit and two-bit implementations concerns packets that traverse multiple congested routers. Consider a CE packet that arrives at a second congested router, and is selected by the active queue management at that router for either marking or dropping. In the one-bit implementation, the second congested router has no choice but to drop the CE packet, because it cannot distinguish between a CE packet and a non-ECT packet. In the two-bit implementation, the second congested router has the choice of either dropping the CE packet, or of leaving it alone with the CE codepoint set. Another difference between the one-bit and two-bit implementations comes from the fact that with the one-bit implementation, receivers in a single flow cannot distinguish between CE and non-ECT packets. Thus, in the one-bit implementation an ECN-capable data sender would have to unambiguously indicate to the receiver or receivers whether each packet had been sent as ECN-Capable or as non-ECN-Capable. One possibility would be for the sender to indicate in the transport header whether the packet was sent as ECN-Capable. A second possibility that would involve a functional limitation for the one- bit implementation would be for the sender to unambiguously indicate that it was going to send *all* of its packets as ECN-Capable or as non-ECN-Capable. For a multicast transport protocol, this unambiguous indication would have to be apparent to receivers joining an on-going multicast session.
Another concern that was described earlier (and recommended in this document) is that transports (particularly TCP) should not mark pure ACK packets or retransmitted packets as being ECN-Capable. A pure ACK packet from a non-ECN-capable transport could be dropped, without necessarily having an impact on the transport from a congestion control perspective (because subsequent ACKs are cumulative). An ECN-capable transport reacting to the CE codepoint in a pure ACK packet by reducing the window would be at a disadvantage in comparison to a non-ECN-capable transport. For this reason (and for reasons described earlier in relation to retransmitted packets), it is desirable to have the ECT codepoint set on a per-packet basis. Another advantage of the two-bit approach is that it is somewhat more robust. The most critical issue, discussed in Section 8, is that the default indication should be that of a non-ECN-Capable transport. In a two-bit implementation, this requirement for the default value simply means that the not-ECT codepoint should be the default. In the one-bit implementation, this means that the single overloaded bit should by default be in the "CE or not ECT" position. This is less clear and straightforward, and possibly more open to incorrect implementations either in the end nodes or in the routers. In summary, while the one-bit implementation could be a possible implementation, it has the following significant limitations relative to the two-bit implementation. First, the one-bit implementation has more limited functionality for the treatment of CE packets at a second congested router. Second, the one-bit implementation requires either that extra information be carried in the transport header of packets from ECN-Capable flows (to convey the functionality of the second bit elsewhere, namely in the transport header), or that senders in ECN-Capable flows accept the limitation that receivers must be able to determine a priori which packets are ECN-Capable and which are not ECN-Capable. Third, the one-bit implementation is possibly more open to errors from faulty implementations that choose the wrong default value for the ECN bit. We believe that the use of the extra bit in the IP header for the ECT-bit is extremely valuable to overcome these limitations.22. Historical Definitions for the IPv4 TOS Octet
RFC 791 [RFC791] defined the ToS (Type of Service) octet in the IP header. In RFC 791, bits 6 and 7 of the ToS octet are listed as "Reserved for Future Use", and are shown set to zero. The first two fields of the ToS octet were defined as the Precedence and Type of Service (TOS) fields.
0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| PRECEDENCE | TOS | 0 | 0 | RFC 791
+-----+-----+-----+-----+-----+-----+-----+-----+
RFC 1122 included bits 6 and 7 in the TOS field, though it did not
discuss any specific use for those two bits:
0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| PRECEDENCE | TOS | RFC 1122
+-----+-----+-----+-----+-----+-----+-----+-----+
The IPv4 TOS octet was redefined in RFC 1349 [RFC1349] as follows:
0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| PRECEDENCE | TOS | MBZ | RFC 1349
+-----+-----+-----+-----+-----+-----+-----+-----+
Bit 6 in the TOS field was defined in RFC 1349 for "Minimize Monetary
Cost". In addition to the Precedence and Type of Service (TOS)
fields, the last field, MBZ (for "must be zero") was defined as
currently unused. RFC 1349 stated that "The originator of a datagram
sets [the MBZ] field to zero (unless participating in an Internet
protocol experiment which makes use of that bit)."
RFC 1455 [RFC 1455] defined an experimental standard that used all
four bits in the TOS field to request a guaranteed level of link
security.
RFC 1349 and RFC 1455 have been obsoleted by "Definition of the
Differentiated Services Field (DS Field) in the IPv4 and IPv6
Headers" [RFC2474] in which bits 6 and 7 of the DS field are listed
as Currently Unused (CU). RFC 2780 [RFC2780] specified ECN as an
experimental use of the two-bit CU field. RFC 2780 updated the
definition of the DS Field to only encompass the first six bits of
this octet rather than all eight bits; these first six bits are
defined as the Differentiated Services CodePoint (DSCP):
0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| DSCP | CU | RFCs 2474,
+-----+-----+-----+-----+-----+-----+-----+-----+ 2780
Because of this unstable history, the definition of the ECN field in
this document cannot be guaranteed to be backwards compatible with
all past uses of these two bits.
Prior to RFC 2474, routers were not permitted to modify bits in either the DSCP or ECN field of packets forwarded through them, and hence routers that comply only with RFCs prior to 2474 should have no effect on ECN. For end nodes, bit 7 (the second ECN bit) must be transmitted as zero for any implementation compliant only with RFCs prior to 2474. Such nodes may transmit bit 6 (the first ECN bit) as one for the "Minimize Monetary Cost" provision of RFC 1349 or the experiment authorized by RFC 1455; neither this aspect of RFC 1349 nor the experiment in RFC 1455 were widely implemented or used. The damage that could be done by a broken, non-conformant router would include "erasing" the CE codepoint for an ECN-capable packet that arrived at the router with the CE codepoint set, or setting the CE codepoint even in the absence of congestion. This has been discussed in the section on "Non-compliance in the Network". The damage that could be done in an ECN-capable environment by a non-ECN-capable end-node transmitting packets with the ECT codepoint set has been discussed in the section on "Non-compliance by the End Nodes".23. IANA Considerations
This section contains the namespaces that have either been created in this specification, or the values assigned in existing namespaces managed by IANA.23.1. IPv4 TOS Byte and IPv6 Traffic Class Octet
The codepoints for the ECN Field of the IP header are specified by the Standards Action of this RFC, as is required by RFC 2780. When this document is published as an RFC, IANA should create a new registry, "IPv4 TOS Byte and IPv6 Traffic Class Octet", with the namespace as follows: IPv4 TOS Byte and IPv6 Traffic Class Octet Description: The registrations are identical for IPv4 and IPv6. Bits 0-5: see Differentiated Services Field Codepoints Registry (http://www.iana.org/assignments/dscp-registry)
Bits 6-7, ECN Field:
Binary Keyword References
------ ------- ----------
00 Not-ECT (Not ECN-Capable Transport) [RFC 3168]
01 ECT(1) (ECN-Capable Transport(1)) [RFC 3168]
10 ECT(0) (ECN-Capable Transport(0)) [RFC 3168]
11 CE (Congestion Experienced) [RFC 3168]
23.2. TCP Header Flags
The codepoints for the CWR and ECE flags in the TCP header are
specified by the Standards Action of this RFC, as is required by RFC
2780.
When this document is published as an RFC, IANA should create a new
registry, "TCP Header Flags", with the namespace as follows:
TCP Header Flags
The Transmission Control Protocol (TCP) included a 6-bit Reserved
field defined in RFC 793, reserved for future use, in bytes 13 and 14
of the TCP header, as illustrated below. The other six Control bits
are defined separately by RFC 793.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
| | | U | A | P | R | S | F |
| Header Length | Reserved | R | C | S | S | Y | I |
| | | G | K | H | T | N | N |
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
RFC 3168 defines two of the six bits from the Reserved field to be
used for ECN, as follows:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
| | | C | E | U | A | P | R | S | F |
| Header Length | Reserved | W | C | R | C | S | S | Y | I |
| | | R | E | G | K | H | T | N | N |
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
TCP Header Flags
Bit Name Reference
--- ---- ---------
8 CWR (Congestion Window Reduced) [RFC 3168]
9 ECE (ECN-Echo) [RFC 3168]
23.3. IPSEC Security Association Attributes
IANA allocated the IPSEC Security Association Attribute value 10 for
the ECN Tunnel use described in Section 9.2.1.2 above at the request
of David Black in November 1999. The IANA has changed the Reference
for this allocation from David Black's request to this RFC.
24. Authors' Addresses
K. K. Ramakrishnan
TeraOptic Networks, Inc.
Phone: +1 (408) 666-8650
EMail: kk@teraoptic.com
Sally Floyd
ACIRI
Phone: +1 (510) 666-2989
EMail: floyd@aciri.org
URL: http://www.aciri.org/floyd/
David L. Black
EMC Corporation
42 South St.
Hopkinton, MA 01748
Phone: +1 (508) 435-1000 x75140
EMail: black_david@emc.com
25. Full Copyright Statement
Copyright (C) The Internet Society (2001). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Acknowledgement Funding for the RFC Editor function is currently provided by the Internet Society.