Internet-Draft | IPv6 HBH and DO Forwarding In Routers | March 2024 |
Ouellette | Expires 5 September 2024 | [Page] |
It has become accepted that packets containing IPv6 Extension Headers, especially Hop-by-hop Options, are frequently dropped on the Internet. However, the question still remains as to why they get dropped and what type of devices may be dropping them. This document describes research conducted to isolate routers in a simple topology, with minimal configuration, and shows that router implementations alone are likely not the cause of packets containing IPv6 Extension Headers being dropped on the Internet.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 5 September 2024.¶
Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
It has become well known that packets containing IPv6 [RFC8200] Extension Headers (EHs), especially Hop-by-Hop Options, are often dropped when traversing the Internet. However, the question still remains as to what are the primary contributors to these packet drops and under what circumstances are the packets dropped. Some of the potential culprits include ACLs, firewalls, routers, load balancers, and more. It also is not known whether these packets are dropped intentionally (e.g., ACLs) or unintentionally (e.g., firmware bugs).¶
This document describes the research conducted and results of isolating routers to better understand what role they may play, if any, in the dropping of EHs. Specifically, this research is aiming to understand whether or not router implementations may be the cause of dropped packets containing EHs.¶
The layer 3 topology used for these experiments can be seen below in Figure 1. Layer 3 is specifically noted because layer 2 switches exist in the topology as well as some components being virtualized, however, this is not expected to have an effect on the traversal of packets containing EHs.¶
+--------+ +--------+ +--------+ | Client |-----| Router |-----| Server | +--------+ +--------+ +--------+
Both client and server are virtual machines running Ubuntu 22.04 and are configured to inject EHs into all egress packets. Additionally, the server has an out-of-the-box installation of an apache2 web server. Details of the routers used can be found below in Section 3.¶
Six routers in total were tested and are comprised of three major router manufacturers. Of the six routers, five are physical and the sixth is virtual. Due to confidentiality requirements, the specific manufacturers and models of the routers tested cannot be disclosed, therefore, the routers will be referred to with a numerical value from here on.¶
For these experiments, all RUTs have very minimal configurations. No routing protocols are enabled and no ACLs are configured. The routers are configured to simply forward IPv6 traffic between the two interfaces.¶
For all experiments, both the client and server are configured to inject varying EHs into all
egress packets. A single HTTP request is then made from the client to the server
and it was observed that the client received an HTTP reply containing the expected EHs.
This indicates that the RUT properly forwarded all packets associated with both the HTTP
request and reply that contained EHs without dropping the packets
or stripping the EHs from them [I-D.herbert-eh-inflight-removal].
To execute the HTTP request, the curl
utility was used.¶
[RFC8250] defines the PDM destination option (DO), an option used for collecting performance metrics such as round-trip delay and server delay. For this experiment, a PDM implementation leveraging Extended Berkley Packet Filter (eBPF) was used [I-D.elkins-v6ops-bpf-pdm-ebpf].¶
The PDM DO was chosen to test with as it is a proposed standard and access to an existing implementation was provided for the use of this research.¶
The second type of experiment run was testing how the RUTs processed both HBH and DO of varying header sizes. For both HBH and DO, sizes of 8, 16, 32, 64, 128, 256, and 512 bytes were tested. In addition to this, tests with an EH chain of both HBH and DO for all enumerations of the 8, 16, 32, 64, 126, and 256 header sizes were also run. Overall, there were 48 unique scenarios tested.¶
To achieve this, the IPv6 Extension Headers Injection with eBPF project was used which allows for injecting many different types of extension headers with control over the size of each header.¶
The following section reports on the findings of these experiments. Tables in this section will denote whether the router properly forwarded packets containing the EH(s) or not. Scenarios in which the router forwards the packets will be denoted with a check mark (✓) and scenarios where the router does not forward the packets will be denoted with an X (X).¶
As can be seen in Table 1, all six of the routers tested properly forwarded packets containing the PDM DO.¶
Router 1 | Router 2 | Router 3 | Router 4 | Router 5 | Router 6 |
---|---|---|---|---|---|
✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Table 2 shows the results of a single variable sized HBH or DO EH. Seen below, all but one router forwards the EHs in every scenario.¶
Scenario | Router 1 | Router 2 | Router 3 | Router 4 | Router 5 | Router 6 |
---|---|---|---|---|---|---|
HBH 8 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
HBH 16 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
HBH 32 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
HBH 64 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
HBH 128 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
HBH 256 | ✓ | ✓ | ✓ | ✓ | X | ✓ |
HBH 512 | ? | ? | ✓ | ? | ? | ? |
DO 8 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
DO 16 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
DO 32 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
DO 64 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
DO 128 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
DO 256 | ✓ | ✓ | ✓ | ✓ | X | ✓ |
DO 512 | ✓ | ✓ | ✓ | ✓ | X | ✓ |
There are two observations to note regarding these results. The first is that five of the six routers have a question mark (?) as their result for the 512-byte HBH scenario. This is because this scenario was unable to be tested for these five routers. The reasoning for this can be found in Section 5.3. The other observation to make is that out of the six routers, Router 5 was the only router unable to forward packets for each scenario. Based on these results, it suggests that Router 5 has a limitation on the maximum size of an EH in a packet. If a packet contains an EH larger than this ceiling, it is dropped. In fact, further testing has shown that a 176-byte HBH or DO is the largest Router 5 is able to properly forward. When increasing to 184-bytes, the initial TCP SYN packet is dropped and therefore the TCP connection is not established and the HTTP request cannot be made.¶
The following section includes a matrix for each RUT indicating whether or not it forwarded the HTTP traffic when the packets include both a HBH and DO of varying lengths.¶
DO | HBH | 8 | 16 | 32 | 64 | 128 | 256 |
---|---|---|---|---|---|---|---|
8 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
16 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
32 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
64 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
128 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
256 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
DO | HBH | 8 | 16 | 32 | 64 | 128 | 256 |
---|---|---|---|---|---|---|---|
8 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
16 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
32 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
64 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
128 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
256 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
DO | HBH | 8 | 16 | 32 | 64 | 128 | 256 |
---|---|---|---|---|---|---|---|
8 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
16 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
32 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
64 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
128 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
256 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
DO | HBH | 8 | 16 | 32 | 64 | 128 | 256 |
---|---|---|---|---|---|---|---|
8 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
16 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
32 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
64 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
128 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
256 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
DO | HBH | 8 | 16 | 32 | 64 | 128 | 256 |
---|---|---|---|---|---|---|---|
8 | ✓ | ✓ | ✓ | ✓ | ✓ | X | |
16 | ✓ | ✓ | ✓ | ✓ | ✓ | X | |
32 | ✓ | ✓ | ✓ | ✓ | ✓ | X | |
64 | ✓ | ✓ | ✓ | ✓ | X | X | |
128 | ✓ | ✓ | ✓ | X | X | X | |
256 | X | X | X | X | X | X |
DO | HBH | 8 | 16 | 32 | 64 | 128 | 256 |
---|---|---|---|---|---|---|---|
8 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
16 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
32 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
64 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
128 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
256 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Of the six routers, Router 5 (Table 7) was the only router that did not properly forward packets for all scenarios. As was seen in Section 5.2.1, Router 5 did not forward packets for the 256-byte and 512-byte scenarios which suggests that Router 5 only processes packets containing EHs up to a certain length. However, in conjunction with these results, it seems as if Router 5 may have limitations based on the total EH chain length rather than individual EHs. It does however properly process smaller EH configurations.¶
In these results, there are two findings to note. The first is that as seen in Section 5.2.1 and Section 5.2.2, a router was discovered (i.e., Router 5) that dropped packets in certain scenarios. However, the exact cause of this is still unknown as additional testing not described in this document has shown that whether or not packets are dropped is more nuanced than simply just being based on the length of the EH chain.¶
The second notable finding is that as seen in Table 2, the 512-byte HBH scenario was unable to be performed for five out of the six routers. The exact cause is still being investigated, however, packets specifically with a 512-byte HBH option were dropped by the layer 2 switching infrastructure. Notably, both 504-byte and 520-byte HBH options traversed the infrastructure without a problem, and only 512-byte HBH options were affected. Although not recorded in the tables above, both 504-byte and 520-byte HBH scenarios were tested on Routers 1, 2, 3, 4, and 6 and they all forwarded the traffic as expected. Because of this, it is believed this is due to a firmware bug in one of the switches and was an unexpected finding. Router 3 was the only router not affected by this due to it having a different layer 2 path and not traversing the problematic switch.¶
Even though this research was conducted on a relatively small sample size of manufacturers and routers, the results indicate that router implementations in general are likely not the culprits for dropping packets containing HBH or DO EHs. One router did face limitations when processing larger EHs, however, EH chains of that size may not be practical when used in a real-world network and therefore may not be a concern. The most surprising finding was that a layer 2 switch was the cause of dropped packets containing 512-byte HBH options.¶
Currently it is not known what the limitation in Router 5 is caused by and it is also not known why a switch in the layer 2 infrastructure is dropping packets specifically with 512-byte HBH options. Next steps will include trying to determine why these behaviors are occurring and making the manufacturers aware of these limitations with the hopes that they address them if possible.¶
A logical next step would be to test more routers, especially across a wider breadth of manufacturers. Additionally, more testing could be performed with varying connectors and link speeds.¶
This research investigated how routers handle varying sizes of single HBH and DO EHs as well as chains of the two. However, more tests could be run with other standalone EHs as well as in various chain configurations.¶
As was mentioned in Section 4, these experiments were run with a single HTTP request and therefore were not run under load which may be the case for routers on the open internet. It is possible that a router's EH processing varies based on whether it is under load or not, especially in the context of processing packets on the fast vs slow path. Because of this, running these same experiments at higher throughputs may be interesting to explore.¶
This document has no security considerations.¶
This document has no privacy considerations.¶
This document has no IANA actions.¶
The author would like to acknowledge and thank the following individuals. Nalini Elkins and Michael Ackermann for their guidance and support throughout this research. Balajinaidu V, Amogh Umesh, and Chinmaya Sharma for developing and providing access to their PDM DO implementation. Justin Iurman for developing and providing access to their IPv6 Extension Headers Injection with eBPF project. Michayla Newcombe and Ben Patton for their review and feedback of this document.¶