The five IPv4aaS technologies can be classified based on two aspects:
-
Technology used for service provider network traversal. It can be single/double translation or encapsulation.
-
Presence or absence of per-flow state in the operator network.
|
464XLAT |
DS-Lite |
lw4o6 |
MAP-E |
MAP-T |
Translation (T) or Encapsulation (E) |
T |
E |
E |
E |
T |
Presence (+) of Per-Flow State in Operator Network |
+ |
+ |
|
|
|
Table 3: Basic Comparison among the Analyzed Technologies
464XLAT and DS-Lite use stateful NAPT at the PLAT and AFTR devices, respectively. This may cause scalability issues for the number of clients or volume of traffic, but it does not impose a limitation on the number of ports per user, as they can be allocated dynamically on-demand and the allocation policy can be centrally managed and adjusted.
A+P-based mechanisms (lw4o6, MAP-E, and MAP-T) avoid using NAPT in the service provider network. However, this means that the number of ports provided to each user (and hence the effective IPv4 address-sharing ratio) must be pre-provisioned to the client.
Changing the allocated port ranges with A+P-based technologies requires more planning and is likely to involve reprovisioning both hosts and operator-side equipment. It should be noted that due to the per-customer binding table entry used by lw4o6, a single customer can be reprovisioned (e.g., if they request a full IPv4 address) without needing to change parameters for a number of customers as in a MAP domain.
It is also worth noting that there is a direct relationship between the efficiency of public port allocations for customers and the corresponding logging overhead that may be necessary to meet data-retention requirements. This is considered in
Section 4.7.
Determining the optimal number of ports for a fixed port set is not an easy task and may also be impacted by local regulatory law (and in the Belgian case, it is not a law but more a memorandum of understanding or best current practice), which may define a maximum number of users per IP address and consequently a minimum number of ports per user.
On the one hand, the "lack of ports" situation may cause serious problems in the operation of certain applications. For example, Miyakawa has demonstrated the consequences of the session number limitation due to port number shortage in the example of Google Maps [
MIY2010]. When the limit was 15, several blocks of the map were missing, and the map was unusable. This study also provided several examples for the session numbers of different applications (the highest one was Apple's iTunes at 230-270 ports).
The port number consumption of different applications is highly varying. In the case of web browsing, it depends on several factors, including the choice of the web page, the web browser, and sometimes the operating system [
REP2014]. For example, under certain conditions, 120-160 ports were used (URL: sohu.com, browser: Firefox under Ubuntu Linux), and in some other cases, only 3-12 ports were used (URL: twitter.com, browser: Iceweasel under Debian Linux).
There may be several users behind a CE router, especially in the broadband case (e.g., Internet is used by different members of a family simultaneously), so sufficient ports must be allocated to avoid impacting user experience.
In general, assigning too few source port numbers to an end user may result in unexpected and hard-to-debug consequences; therefore, if the number of ports per end user is fixed, then we recommend assigning a conservatively large number of ports. For example, the developers of Jool used 2048 ports per user in their example for MAP-T [
JOOL-MAPT].
However, assigning too many ports per CE router will result in waste of public IPv4 addresses, which are scarce and expensive resources. Clearly, this is a big advantage in the case of 464XLAT where they are dynamically managed so that the number of IPv4 addresses for the sharing pool is smaller while the availability of ports per user doesn't need to be pre-defined and is not a limitation.
There is a direct trade-off between the optimization of client port allocations and the associated logging overhead.
Section 4.7 discusses this in more depth.
We note that common NAT44 implementations utilizing Netfilter at the CE router multiplex active sessions using a 3-tuple (source address, destination address, and destination port). This means that external source ports can be reused for unique internal source and destination addresses and port sessions. It is also noted that Netfilter cannot currently make use of multiple source port ranges (i.e., several blocks of ports distributed across the total port space as is common in MAP deployments). This may influence the design when using stateless technologies.
Stateful technologies, 464XLAT, DS-Lite, and NAT444 can therefore be much more efficient in terms of port allocation and thus public IP address saving. The price is the stateful operation in the service provider network, which allegedly does not scale up well. It should be noted that, in many cases, all those factors may depend on how it is actually implemented.
Measurements have been started to examine the scalability of a few stateful solutions in two areas:
-
How their performance scales up with the number of CPU cores
-
To what extent their performance degrades with the number of concurrent connections
The details of the measurements and their results are available from [
IPv4aaS-SCALE-TECH].
We note that some CGN-type solutions can allocate ports dynamically "on the fly". Depending on configuration, this can result in the same customer being allocated ports from different source addresses. This can cause operational issues for protocols and applications that expect multiple flows to be sourced from the same address (e.g., ECMP hashing, STUN, gaming, and content delivery networks). However, it should be noted that this is the same problem when a network has a NAT44 with multiple public IPv4 addresses, or even when applications in a dual-stack case, behave wrongly if Happy Eyeballs is flapping the flow address between IPv4 and IPv6.
The consequences of IPv4 address sharing [
RFC 6269] may impact all five technologies. However, when ports are allocated statically, more customers may get ports from the same public IPv4 address, which may result in negative consequences with higher probability. For example, many applications and service providers (Sony PlayStation Network, OpenDNS, etc.) can permanently block IPv4 ranges if they detect that they are used for address sharing.
Both cases are, again, implementation-dependent.
We note that although it is not of typical use, one can do deterministic, stateful NAT and reserve a fixed set of ports for each customer as well.
Mechanisms that rely on operator-side per-flow state do not, by themselves, offer a way for customers to present services on publicly accessible transport-layer ports.
The Port Control Protocol (PCP) [
RFC 6887] provides a mechanism for a client to request an external public port from a CGN device. For server operation, it is required with 464XLAT/NAT64, and it is supported in some DS-Lite AFTR implementations.
A+P-based mechanisms distribute a public IPv4 address and restricted range of transport-layer ports to the client. In this case, it is possible for the user to configure their device to offer a publicly accessible server on one of their allocated ports. It should be noted that operators commonly do not assign the well-known ports to users (unless they are allocating a full IPv4 address), so the user will need to run the service on an allocated port or configure port translation.
Lw4o6, MAP-E, and MAP-T may be configured to allocated clients with a full IPv4 address, allowing exclusive use of all ports and non-port-based transport-layer protocols. Thus, they may also be used to support server/services operation on their default ports. However, when public IPv4 addresses are assigned to the CE router without address sharing, there is obviously no advantage in terms of IPv4 public addresses saving.
It is also possible to configure specific ports mapping in 464XLAT/NAT64 using EAMT [
RFC 7757], which means that only those ports are "lost" from the pool of addresses, so there is a higher maximization of the total usage of IPv4 port resources.
In general, router vendors support AFTR, MAP-E BR, MAP-T BR, and NAT64. Vendors of load balancers and firewalls usually support NAT64 as well while not all of them have support for the other protocols.
A 464XLAT client (CLAT) is implemented in Windows 10, Linux (including Android), Windows Mobile, Chrome OS, and iOS, but it is not available in macOS 12.3.1.
The remaining four solutions are commonly deployed as functions in the CE device only; however, the vendors' support is poor in general (except for DS-Lite).
OpenWRT is a Linux-based open-source OS designed for CE devices. It offers a number of different 'opkg' packages as part of the distribution:
-
'464xlat' enables support for 464XLAT CLAT functionality.
-
'ds-lite' enables support for DSLite B4 functionality.
-
'map' enables support for MAP-E and lw4o6 CE functionality.
-
'map-t' enables support for MAP-T CE functionality.
At the time of publication, some free open-source implementations exist for the operator-side functionality:
-
Jool [JOOL] (CLAT, NAT64, EAMT, MAP-T CE, MAP-T BR)
-
VPP/fd.io [VPP] (MAP-BR, lwAFTR, CGN, CLAT, NAT64)
-
Snabb [SNABB] (lwAFTR)
-
AFTR [AFTR] (DSLite AFTR)
Several cellular networks use 464XLAT, whereas there are no deployments of the four other technologies in cellular networks, as they are neither standardized nor implemented in UE devices.
In broadband networks, there are some deployments of 464XLAT, MAP-E, and MAP-T. Lw4o6 and DS-Lite have more deployments, with DS-Lite being the most common, but deployments of lw4o6 have been rapidly increasing in the last few years.
Please refer to Tables 2 and 3 of [
LEN2019] for a limited set of deployment information.
As a hint to the relative complexity of the mechanisms, the code sizes reported from the OpenWRT implementations of each technology are 17 kB, 35 kB, 15 kB, 35 kB, and 48 kB for 464XLAT, lw4o6, DS-Lite, MAP-E, and MAP-T, respectively (see <
https://openwrt.org/packages/start>).
We note that the support for all five technologies requires a much smaller code size than the total sum of the above quantities, because they contain a lot of common functions (e.g., data plane is shared among several of them).
Theoretically, all five IPv4aaS technologies could be used together with DNS64 + stateful NAT64, as is done in 464XLAT. In this case, the CE router would treat the traffic between an IPv6-only client and IPv4-only server as normal IPv6 traffic, and the stateful NAT64 gateway would do a single translation, thus offloading this kind of traffic from the IPv4aaS technology. The cost of this solution would be the need to also deploy DNS64 + stateful NAT64.
However, this has not been implemented in clients or actual deployments, so only 464XLAT always uses this optimization, and the other four solutions do not use it at all.
Figures from existing deployments (through the end of 2018) show the typical traffic volumes in an IPv6-only cellular network when 464XLAT technology is used together with DNS64:
-
75% of traffic is IPv6 end-to-end (no translation).
-
24% of traffic uses DNS64 + NAT64 (one translation).
-
Less than 1% of traffic uses the CLAT in addition to NAT64 (two translations), due to an IPv4 socket and/or IPv4 literal.
Without using DNS64, 25% of the traffic would undergo double translation.
Figures from several existing deployments (through the end of 2020), mainly with residential customers, show the ranges of typical traffic volumes in an IPv6-only network, when 464XLAT is used with DNS64:
-
65%-85% of traffic is IPv6 end-to-end (no translation).
-
14%-34% of traffic uses DNS64 + NAT64 (one translation).
-
Less than 1-2% of traffic uses the CLAT in addition to NAT64 (two translations), due to an IPv4 socket and/or IPv4 literal.
Without using DNS64, 16%-35% of the traffic would undergo double translation.
This data is consistent with non-public information of actual deployments, which can be easily explained. When a wireline ISP has mainly residential customers, content providers and CDNs that are already IPv6 enabled (Google/YouTube, Netflix, Facebook, Akamai, etc.) typically account for 65-85% of the traffic in the network. Thus, when the subscribers are IPv6 enabled, about the same percentage of traffic will become IPv6.
If multiple network-side devices are needed as PLAT/AFTR/BR for capacity, then there is a need for a load-sharing mechanism. ECMP (Equal-Cost Multipath) load sharing can be used for all technologies; however, stateful technologies will be impacted by changes in network topology or device failure.
Technologies utilizing DNS64 can also distribute load across PLAT/AFTR devices, evenly or unevenly, by using different prefixes. Different network-specific prefixes can be distributed for subscribers in appropriately sized segments (like split-horizon DNS, also called "DNS views").
Stateless technologies, due to the lack of per-flow state, can make use of anycast routing for load sharing and resiliency across network devices, both ingress and egress; flows can take asymmetric paths through the network, i.e., in through one lwAFTR/BR and out via another.
Mechanisms with centralized NAPT44 state have a number of challenges specifically related to scaling and resilience. As the total amount of client traffic exceeds the capacity of a single CGN instance, additional nodes are required to handle the load. Each CGN maintains a stateful table of active client sessions, and this table may need to be synchronized between CGN instances. This is necessary for two reasons:
-
To prevent all active customer sessions from being dropped in the event of a CGN node failure.
-
To ensure a matching state table entry for an active session in the event of asymmetric routing through different egress and ingress CGN nodes.
In the case of 464XLAT and DS-Lite, the user of any given public IPv4 address and port combination will vary over time; therefore, logging is necessary to meet data-retention laws. Each entry in the PLAT/AFTR generates a logging entry. As discussed in
Section 4.2, a client may open hundreds of sessions during common tasks such as web browsing, each of which needs to be logged so the overall logging burden on the network operator is significant. In some countries, this level of logging is required to comply with data-retention legislation.
One common optimization available to reduce the logging overhead is the allocation of a block of ports to a client for the duration of their session. This means that a logging entry only needs to be made when the client's port block is released, which dramatically reduces the logging overhead. This comes as the cost of less efficient public address sharing as clients need to be allocated a port block of a fixed size regardless of the actual number of ports that they are using.
Stateless technologies that pre-allocate the IPv4 addresses and ports only require that copies of the active MAP rules (for MAP-E and MAP-T) or binding table (for lw4o6) are retained along with timestamp information of when they have been active. Support tools (e.g., those used to serve data-retention requests) may need to be updated to be aware of the mechanism in use (e.g., implementing the MAP algorithm so that IPv4 information can be linked to the IPv6 prefix delegated to a client). Stateless technologies do not have a centralized stateful element that customer traffic needs to pass through, so if data-retention laws mandate per-session logging, there is no simple way of meeting this requirement with a stateless technology alone. Thus, a centralized NAPT44 model may be the only way to meet this requirement.
Deterministic CGN [
RFC 7422] was proposed as a solution to reduce the resource consumption of logging.
Please also refer to
Section 4 of
RFC 6888 for more information about requirements for logging CGN gateways.
When IPv4-only devices or applications are behind a CE connected with IPv6-only and IPv4aaS, the IPv4-only traffic flows will necessarily be encapsulated/decapsulated (in the case of DS-Lite, lw4o6, and MAP-E) and will reach the IPv4 address of the destination, even if that service supports dual-stack. This means that the traffic flow will cross through the AFTR, lwAFTR, or BR, depending on the specific transition mechanism being used.
Even if those services are directly connected to the operator network (e.g., CDNs and caches) or located internally (such as VoIP, etc.), it is not possible to avoid that overhead.
However, in the case of those mechanisms that use a NAT46 function, in the CE (464XLAT and MAP-T), it is possible to take advantage of optimization functionalities, such as the ones described in [
OP-464XLAT/MAP-T].
Because the NAT46 has already translated the IPv4-only flow to IPv6 and the services are dual-stack, using these optimizations allows the services to be reached without the need to translate the flow back to IPv4.