From 1b23f5e9973abc2137ca615d770bf23d8e45b93c Mon Sep 17 00:00:00 2001 From: Otto Sabart Date: Sun, 6 Jan 2019 00:28:59 +0100 Subject: doc: networking: prepare offload documents for conversion into RST Add small number of markups which are sufficient for conversion into reStructuredText. Unfortunately there was necessary to restructure all sections in checksum-offloads.txt file and create paragraphs separated by newline. There also must not be a space at the beginning of paragpraph. There are no semantic changes. Signed-off-by: Otto Sabart Acked-by: David S. Miller Signed-off-by: Jonathan Corbet --- Documentation/networking/checksum-offloads.txt | 179 ++++++++++++--------- Documentation/networking/segmentation-offloads.txt | 46 ++++-- 2 files changed, 130 insertions(+), 95 deletions(-) (limited to 'Documentation/networking') diff --git a/Documentation/networking/checksum-offloads.txt b/Documentation/networking/checksum-offloads.txt index 27bc09cfcf6d..1a1cd94a3f6d 100644 --- a/Documentation/networking/checksum-offloads.txt +++ b/Documentation/networking/checksum-offloads.txt @@ -1,122 +1,143 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============================================== Checksum Offloads in the Linux Networking Stack +=============================================== Introduction ============ -This document describes a set of techniques in the Linux networking stack - to take advantage of checksum offload capabilities of various NICs. +This document describes a set of techniques in the Linux networking stack to +take advantage of checksum offload capabilities of various NICs. The following technologies are described: - * TX Checksum Offload - * LCO: Local Checksum Offload - * RCO: Remote Checksum Offload + +* TX Checksum Offload +* LCO: Local Checksum Offload +* RCO: Remote Checksum Offload Things that should be documented here but aren't yet: - * RX Checksum Offload - * CHECKSUM_UNNECESSARY conversion + +* RX Checksum Offload +* CHECKSUM_UNNECESSARY conversion TX Checksum Offload =================== -The interface for offloading a transmit checksum to a device is explained - in detail in comments near the top of include/linux/skbuff.h. +The interface for offloading a transmit checksum to a device is explained in +detail in comments near the top of include/linux/skbuff.h. + In brief, it allows to request the device fill in a single ones-complement - checksum defined by the sk_buff fields skb->csum_start and - skb->csum_offset. The device should compute the 16-bit ones-complement - checksum (i.e. the 'IP-style' checksum) from csum_start to the end of the - packet, and fill in the result at (csum_start + csum_offset). -Because csum_offset cannot be negative, this ensures that the previous - value of the checksum field is included in the checksum computation, thus - it can be used to supply any needed corrections to the checksum (such as - the sum of the pseudo-header for UDP or TCP). +checksum defined by the sk_buff fields skb->csum_start and skb->csum_offset. +The device should compute the 16-bit ones-complement checksum (i.e. the +'IP-style' checksum) from csum_start to the end of the packet, and fill in the +result at (csum_start + csum_offset). + +Because csum_offset cannot be negative, this ensures that the previous value of +the checksum field is included in the checksum computation, thus it can be used +to supply any needed corrections to the checksum (such as the sum of the +pseudo-header for UDP or TCP). + This interface only allows a single checksum to be offloaded. Where - encapsulation is used, the packet may have multiple checksum fields in - different header layers, and the rest will have to be handled by another - mechanism such as LCO or RCO. +encapsulation is used, the packet may have multiple checksum fields in +different header layers, and the rest will have to be handled by another +mechanism such as LCO or RCO. + CRC32c can also be offloaded using this interface, by means of filling - skb->csum_start and skb->csum_offset as described above, and setting - skb->csum_not_inet: see skbuff.h comment (section 'D') for more details. +skb->csum_start and skb->csum_offset as described above, and setting +skb->csum_not_inet: see skbuff.h comment (section 'D') for more details. + No offloading of the IP header checksum is performed; it is always done in - software. This is OK because when we build the IP header, we obviously - have it in cache, so summing it isn't expensive. It's also rather short. +software. This is OK because when we build the IP header, we obviously have it +in cache, so summing it isn't expensive. It's also rather short. + The requirements for GSO are more complicated, because when segmenting an - encapsulated packet both the inner and outer checksums may need to be - edited or recomputed for each resulting segment. See the skbuff.h comment - (section 'E') for more details. +encapsulated packet both the inner and outer checksums may need to be edited or +recomputed for each resulting segment. See the skbuff.h comment (section 'E') +for more details. A driver declares its offload capabilities in netdev->hw_features; see - Documentation/networking/netdev-features.txt for more. Note that a device - which only advertises NETIF_F_IP[V6]_CSUM must still obey the csum_start - and csum_offset given in the SKB; if it tries to deduce these itself in - hardware (as some NICs do) the driver should check that the values in the - SKB match those which the hardware will deduce, and if not, fall back to - checksumming in software instead (with skb_csum_hwoffload_help() or one of - the skb_checksum_help() / skb_crc32c_csum_help functions, as mentioned in - include/linux/skbuff.h). - -The stack should, for the most part, assume that checksum offload is - supported by the underlying device. The only place that should check is - validate_xmit_skb(), and the functions it calls directly or indirectly. - That function compares the offload features requested by the SKB (which - may include other offloads besides TX Checksum Offload) and, if they are - not supported or enabled on the device (determined by netdev->features), - performs the corresponding offload in software. In the case of TX - Checksum Offload, that means calling skb_csum_hwoffload_help(skb, features). +Documentation/networking/netdev-features.txt for more. Note that a device +which only advertises NETIF_F_IP[V6]_CSUM must still obey the csum_start and +csum_offset given in the SKB; if it tries to deduce these itself in hardware +(as some NICs do) the driver should check that the values in the SKB match +those which the hardware will deduce, and if not, fall back to checksumming in +software instead (with skb_csum_hwoffload_help() or one of the +skb_checksum_help() / skb_crc32c_csum_help functions, as mentioned in +include/linux/skbuff.h). + +The stack should, for the most part, assume that checksum offload is supported +by the underlying device. The only place that should check is +validate_xmit_skb(), and the functions it calls directly or indirectly. That +function compares the offload features requested by the SKB (which may include +other offloads besides TX Checksum Offload) and, if they are not supported or +enabled on the device (determined by netdev->features), performs the +corresponding offload in software. In the case of TX Checksum Offload, that +means calling skb_csum_hwoffload_help(skb, features). LCO: Local Checksum Offload =========================== LCO is a technique for efficiently computing the outer checksum of an - encapsulated datagram when the inner checksum is due to be offloaded. -The ones-complement sum of a correctly checksummed TCP or UDP packet is - equal to the complement of the sum of the pseudo header, because everything - else gets 'cancelled out' by the checksum field. This is because the sum was - complemented before being written to the checksum field. +encapsulated datagram when the inner checksum is due to be offloaded. + +The ones-complement sum of a correctly checksummed TCP or UDP packet is equal +to the complement of the sum of the pseudo header, because everything else gets +'cancelled out' by the checksum field. This is because the sum was +complemented before being written to the checksum field. + More generally, this holds in any case where the 'IP-style' ones complement - checksum is used, and thus any checksum that TX Checksum Offload supports. +checksum is used, and thus any checksum that TX Checksum Offload supports. + That is, if we have set up TX Checksum Offload with a start/offset pair, we - know that after the device has filled in that checksum, the ones - complement sum from csum_start to the end of the packet will be equal to - the complement of whatever value we put in the checksum field beforehand. - This allows us to compute the outer checksum without looking at the payload: - we simply stop summing when we get to csum_start, then add the complement of - the 16-bit word at (csum_start + csum_offset). +know that after the device has filled in that checksum, the ones complement sum +from csum_start to the end of the packet will be equal to the complement of +whatever value we put in the checksum field beforehand. This allows us to +compute the outer checksum without looking at the payload: we simply stop +summing when we get to csum_start, then add the complement of the 16-bit word +at (csum_start + csum_offset). + Then, when the true inner checksum is filled in (either by hardware or by - skb_checksum_help()), the outer checksum will become correct by virtue of - the arithmetic. +skb_checksum_help()), the outer checksum will become correct by virtue of the +arithmetic. LCO is performed by the stack when constructing an outer UDP header for an - encapsulation such as VXLAN or GENEVE, in udp_set_csum(). Similarly for - the IPv6 equivalents, in udp6_set_csum(). +encapsulation such as VXLAN or GENEVE, in udp_set_csum(). Similarly for the +IPv6 equivalents, in udp6_set_csum(). + It is also performed when constructing an IPv4 GRE header, in - net/ipv4/ip_gre.c:build_header(). It is *not* currently performed when - constructing an IPv6 GRE header; the GRE checksum is computed over the - whole packet in net/ipv6/ip6_gre.c:ip6gre_xmit2(), but it should be - possible to use LCO here as IPv6 GRE still uses an IP-style checksum. +net/ipv4/ip_gre.c:build_header(). It is *not* currently performed when +constructing an IPv6 GRE header; the GRE checksum is computed over the whole +packet in net/ipv6/ip6_gre.c:ip6gre_xmit2(), but it should be possible to use +LCO here as IPv6 GRE still uses an IP-style checksum. + All of the LCO implementations use a helper function lco_csum(), in - include/linux/skbuff.h. +include/linux/skbuff.h. LCO can safely be used for nested encapsulations; in this case, the outer - encapsulation layer will sum over both its own header and the 'middle' - header. This does mean that the 'middle' header will get summed multiple - times, but there doesn't seem to be a way to avoid that without incurring - bigger costs (e.g. in SKB bloat). +encapsulation layer will sum over both its own header and the 'middle' header. +This does mean that the 'middle' header will get summed multiple times, but +there doesn't seem to be a way to avoid that without incurring bigger costs +(e.g. in SKB bloat). RCO: Remote Checksum Offload ============================ -RCO is a technique for eliding the inner checksum of an encapsulated - datagram, allowing the outer checksum to be offloaded. It does, however, - involve a change to the encapsulation protocols, which the receiver must - also support. For this reason, it is disabled by default. +RCO is a technique for eliding the inner checksum of an encapsulated datagram, +allowing the outer checksum to be offloaded. It does, however, involve a +change to the encapsulation protocols, which the receiver must also support. +For this reason, it is disabled by default. + RCO is detailed in the following Internet-Drafts: -https://tools.ietf.org/html/draft-herbert-remotecsumoffload-00 -https://tools.ietf.org/html/draft-herbert-vxlan-rco-00 -In Linux, RCO is implemented individually in each encapsulation protocol, - and most tunnel types have flags controlling its use. For instance, VXLAN - has the flag VXLAN_F_REMCSUM_TX (per struct vxlan_rdst) to indicate that - RCO should be used when transmitting to a given remote destination. + +* https://tools.ietf.org/html/draft-herbert-remotecsumoffload-00 +* https://tools.ietf.org/html/draft-herbert-vxlan-rco-00 + +In Linux, RCO is implemented individually in each encapsulation protocol, and +most tunnel types have flags controlling its use. For instance, VXLAN has the +flag VXLAN_F_REMCSUM_TX (per struct vxlan_rdst) to indicate that RCO should be +used when transmitting to a given remote destination. diff --git a/Documentation/networking/segmentation-offloads.txt b/Documentation/networking/segmentation-offloads.txt index aca542ec125c..1794bfe98196 100644 --- a/Documentation/networking/segmentation-offloads.txt +++ b/Documentation/networking/segmentation-offloads.txt @@ -1,4 +1,9 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=================================================== Segmentation Offloads in the Linux Networking Stack +=================================================== + Introduction ============ @@ -15,6 +20,7 @@ The following technologies are described: * Partial Generic Segmentation Offload - GSO_PARTIAL * SCTP accelleration with GSO - GSO_BY_FRAGS + TCP Segmentation Offload ======================== @@ -42,6 +48,7 @@ NETIF_F_TSO_MANGLEID set then the IP ID can be ignored when performing TSO and we will either increment the IP ID for all frames, or leave it at a static value based on driver preference. + UDP Fragmentation Offload ========================= @@ -54,6 +61,7 @@ UFO is deprecated: modern kernels will no longer generate UFO skbs, but can still receive them from tuntap and similar devices. Offload of UDP-based tunnel protocols is still supported. + IPIP, SIT, GRE, UDP Tunnel, and Remote Checksum Offloads ======================================================== @@ -71,17 +79,19 @@ refer to the tunnel headers as the outer headers, while the encapsulated data is normally referred to as the inner headers. Below is the list of calls to access the given headers: -IPIP/SIT Tunnel: - Outer Inner -MAC skb_mac_header -Network skb_network_header skb_inner_network_header -Transport skb_transport_header +IPIP/SIT Tunnel:: + + Outer Inner + MAC skb_mac_header + Network skb_network_header skb_inner_network_header + Transport skb_transport_header -UDP/GRE Tunnel: - Outer Inner -MAC skb_mac_header skb_inner_mac_header -Network skb_network_header skb_inner_network_header -Transport skb_transport_header skb_inner_transport_header +UDP/GRE Tunnel:: + + Outer Inner + MAC skb_mac_header skb_inner_mac_header + Network skb_network_header skb_inner_network_header + Transport skb_transport_header skb_inner_transport_header In addition to the above tunnel types there are also SKB_GSO_GRE_CSUM and SKB_GSO_UDP_TUNNEL_CSUM. These two additional tunnel types reflect the @@ -93,6 +103,7 @@ header has requested a remote checksum offload. In this case the inner headers will be left with a partial checksum and only the outer header checksum will be computed. + Generic Segmentation Offload ============================ @@ -106,6 +117,7 @@ Before enabling any hardware segmentation offload a corresponding software offload is required in GSO. Otherwise it becomes possible for a frame to be re-routed between devices and end up being unable to be transmitted. + Generic Receive Offload ======================= @@ -117,6 +129,7 @@ this is IPv4 ID in the case that the DF bit is set for a given IP header. If the value of the IPv4 ID is not sequentially incrementing it will be altered so that it is when a frame assembled via GRO is segmented via GSO. + Partial Generic Segmentation Offload ==================================== @@ -134,6 +147,7 @@ is the outer IPv4 ID field. It is up to the device drivers to guarantee that the IPv4 ID field is incremented in the case that a given header does not have the DF bit set. + SCTP accelleration with GSO =========================== @@ -157,14 +171,14 @@ appropriately. There are some helpers to make this easier: - - skb_is_gso(skb) && skb_is_gso_sctp(skb) is the best way to see if - an skb is an SCTP GSO skb. +- skb_is_gso(skb) && skb_is_gso_sctp(skb) is the best way to see if + an skb is an SCTP GSO skb. - - For size checks, the skb_gso_validate_*_len family of helpers correctly - considers GSO_BY_FRAGS. +- For size checks, the skb_gso_validate_*_len family of helpers correctly + considers GSO_BY_FRAGS. - - For manipulating packets, skb_increase_gso_size and skb_decrease_gso_size - will check for GSO_BY_FRAGS and WARN if asked to manipulate these skbs. +- For manipulating packets, skb_increase_gso_size and skb_decrease_gso_size + will check for GSO_BY_FRAGS and WARN if asked to manipulate these skbs. This also affects drivers with the NETIF_F_FRAGLIST & NETIF_F_GSO_SCTP bits set. Note also that NETIF_F_GSO_SCTP is included in NETIF_F_GSO_SOFTWARE. -- cgit v1.2.3-58-ga151 From d0dcde6426ce071ad447fb9d91c85ab649026114 Mon Sep 17 00:00:00 2001 From: Otto Sabart Date: Sun, 6 Jan 2019 00:29:15 +0100 Subject: doc: networking: convert offload files into RST and update references This patch renames offload files. This is necessary for Sphinx. Also update reference to checksum-offloads.rst file. Whole kernel code was grepped for references using: $ grep -r "\(segmentation\|checksum\)-offloads.txt" . There should be no other references to {segmentation,checksum}-offloads.txt files. Signed-off-by: Otto Sabart Acked-by: David S. Miller Signed-off-by: Jonathan Corbet --- Documentation/networking/checksum-offloads.rst | 143 ++++++++++++++++ Documentation/networking/checksum-offloads.txt | 143 ---------------- Documentation/networking/segmentation-offloads.rst | 184 +++++++++++++++++++++ Documentation/networking/segmentation-offloads.txt | 184 --------------------- include/linux/skbuff.h | 2 +- 5 files changed, 328 insertions(+), 328 deletions(-) create mode 100644 Documentation/networking/checksum-offloads.rst delete mode 100644 Documentation/networking/checksum-offloads.txt create mode 100644 Documentation/networking/segmentation-offloads.rst delete mode 100644 Documentation/networking/segmentation-offloads.txt (limited to 'Documentation/networking') diff --git a/Documentation/networking/checksum-offloads.rst b/Documentation/networking/checksum-offloads.rst new file mode 100644 index 000000000000..1a1cd94a3f6d --- /dev/null +++ b/Documentation/networking/checksum-offloads.rst @@ -0,0 +1,143 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============================================== +Checksum Offloads in the Linux Networking Stack +=============================================== + + +Introduction +============ + +This document describes a set of techniques in the Linux networking stack to +take advantage of checksum offload capabilities of various NICs. + +The following technologies are described: + +* TX Checksum Offload +* LCO: Local Checksum Offload +* RCO: Remote Checksum Offload + +Things that should be documented here but aren't yet: + +* RX Checksum Offload +* CHECKSUM_UNNECESSARY conversion + + +TX Checksum Offload +=================== + +The interface for offloading a transmit checksum to a device is explained in +detail in comments near the top of include/linux/skbuff.h. + +In brief, it allows to request the device fill in a single ones-complement +checksum defined by the sk_buff fields skb->csum_start and skb->csum_offset. +The device should compute the 16-bit ones-complement checksum (i.e. the +'IP-style' checksum) from csum_start to the end of the packet, and fill in the +result at (csum_start + csum_offset). + +Because csum_offset cannot be negative, this ensures that the previous value of +the checksum field is included in the checksum computation, thus it can be used +to supply any needed corrections to the checksum (such as the sum of the +pseudo-header for UDP or TCP). + +This interface only allows a single checksum to be offloaded. Where +encapsulation is used, the packet may have multiple checksum fields in +different header layers, and the rest will have to be handled by another +mechanism such as LCO or RCO. + +CRC32c can also be offloaded using this interface, by means of filling +skb->csum_start and skb->csum_offset as described above, and setting +skb->csum_not_inet: see skbuff.h comment (section 'D') for more details. + +No offloading of the IP header checksum is performed; it is always done in +software. This is OK because when we build the IP header, we obviously have it +in cache, so summing it isn't expensive. It's also rather short. + +The requirements for GSO are more complicated, because when segmenting an +encapsulated packet both the inner and outer checksums may need to be edited or +recomputed for each resulting segment. See the skbuff.h comment (section 'E') +for more details. + +A driver declares its offload capabilities in netdev->hw_features; see +Documentation/networking/netdev-features.txt for more. Note that a device +which only advertises NETIF_F_IP[V6]_CSUM must still obey the csum_start and +csum_offset given in the SKB; if it tries to deduce these itself in hardware +(as some NICs do) the driver should check that the values in the SKB match +those which the hardware will deduce, and if not, fall back to checksumming in +software instead (with skb_csum_hwoffload_help() or one of the +skb_checksum_help() / skb_crc32c_csum_help functions, as mentioned in +include/linux/skbuff.h). + +The stack should, for the most part, assume that checksum offload is supported +by the underlying device. The only place that should check is +validate_xmit_skb(), and the functions it calls directly or indirectly. That +function compares the offload features requested by the SKB (which may include +other offloads besides TX Checksum Offload) and, if they are not supported or +enabled on the device (determined by netdev->features), performs the +corresponding offload in software. In the case of TX Checksum Offload, that +means calling skb_csum_hwoffload_help(skb, features). + + +LCO: Local Checksum Offload +=========================== + +LCO is a technique for efficiently computing the outer checksum of an +encapsulated datagram when the inner checksum is due to be offloaded. + +The ones-complement sum of a correctly checksummed TCP or UDP packet is equal +to the complement of the sum of the pseudo header, because everything else gets +'cancelled out' by the checksum field. This is because the sum was +complemented before being written to the checksum field. + +More generally, this holds in any case where the 'IP-style' ones complement +checksum is used, and thus any checksum that TX Checksum Offload supports. + +That is, if we have set up TX Checksum Offload with a start/offset pair, we +know that after the device has filled in that checksum, the ones complement sum +from csum_start to the end of the packet will be equal to the complement of +whatever value we put in the checksum field beforehand. This allows us to +compute the outer checksum without looking at the payload: we simply stop +summing when we get to csum_start, then add the complement of the 16-bit word +at (csum_start + csum_offset). + +Then, when the true inner checksum is filled in (either by hardware or by +skb_checksum_help()), the outer checksum will become correct by virtue of the +arithmetic. + +LCO is performed by the stack when constructing an outer UDP header for an +encapsulation such as VXLAN or GENEVE, in udp_set_csum(). Similarly for the +IPv6 equivalents, in udp6_set_csum(). + +It is also performed when constructing an IPv4 GRE header, in +net/ipv4/ip_gre.c:build_header(). It is *not* currently performed when +constructing an IPv6 GRE header; the GRE checksum is computed over the whole +packet in net/ipv6/ip6_gre.c:ip6gre_xmit2(), but it should be possible to use +LCO here as IPv6 GRE still uses an IP-style checksum. + +All of the LCO implementations use a helper function lco_csum(), in +include/linux/skbuff.h. + +LCO can safely be used for nested encapsulations; in this case, the outer +encapsulation layer will sum over both its own header and the 'middle' header. +This does mean that the 'middle' header will get summed multiple times, but +there doesn't seem to be a way to avoid that without incurring bigger costs +(e.g. in SKB bloat). + + +RCO: Remote Checksum Offload +============================ + +RCO is a technique for eliding the inner checksum of an encapsulated datagram, +allowing the outer checksum to be offloaded. It does, however, involve a +change to the encapsulation protocols, which the receiver must also support. +For this reason, it is disabled by default. + +RCO is detailed in the following Internet-Drafts: + +* https://tools.ietf.org/html/draft-herbert-remotecsumoffload-00 +* https://tools.ietf.org/html/draft-herbert-vxlan-rco-00 + +In Linux, RCO is implemented individually in each encapsulation protocol, and +most tunnel types have flags controlling its use. For instance, VXLAN has the +flag VXLAN_F_REMCSUM_TX (per struct vxlan_rdst) to indicate that RCO should be +used when transmitting to a given remote destination. diff --git a/Documentation/networking/checksum-offloads.txt b/Documentation/networking/checksum-offloads.txt deleted file mode 100644 index 1a1cd94a3f6d..000000000000 --- a/Documentation/networking/checksum-offloads.txt +++ /dev/null @@ -1,143 +0,0 @@ -.. SPDX-License-Identifier: GPL-2.0 - -=============================================== -Checksum Offloads in the Linux Networking Stack -=============================================== - - -Introduction -============ - -This document describes a set of techniques in the Linux networking stack to -take advantage of checksum offload capabilities of various NICs. - -The following technologies are described: - -* TX Checksum Offload -* LCO: Local Checksum Offload -* RCO: Remote Checksum Offload - -Things that should be documented here but aren't yet: - -* RX Checksum Offload -* CHECKSUM_UNNECESSARY conversion - - -TX Checksum Offload -=================== - -The interface for offloading a transmit checksum to a device is explained in -detail in comments near the top of include/linux/skbuff.h. - -In brief, it allows to request the device fill in a single ones-complement -checksum defined by the sk_buff fields skb->csum_start and skb->csum_offset. -The device should compute the 16-bit ones-complement checksum (i.e. the -'IP-style' checksum) from csum_start to the end of the packet, and fill in the -result at (csum_start + csum_offset). - -Because csum_offset cannot be negative, this ensures that the previous value of -the checksum field is included in the checksum computation, thus it can be used -to supply any needed corrections to the checksum (such as the sum of the -pseudo-header for UDP or TCP). - -This interface only allows a single checksum to be offloaded. Where -encapsulation is used, the packet may have multiple checksum fields in -different header layers, and the rest will have to be handled by another -mechanism such as LCO or RCO. - -CRC32c can also be offloaded using this interface, by means of filling -skb->csum_start and skb->csum_offset as described above, and setting -skb->csum_not_inet: see skbuff.h comment (section 'D') for more details. - -No offloading of the IP header checksum is performed; it is always done in -software. This is OK because when we build the IP header, we obviously have it -in cache, so summing it isn't expensive. It's also rather short. - -The requirements for GSO are more complicated, because when segmenting an -encapsulated packet both the inner and outer checksums may need to be edited or -recomputed for each resulting segment. See the skbuff.h comment (section 'E') -for more details. - -A driver declares its offload capabilities in netdev->hw_features; see -Documentation/networking/netdev-features.txt for more. Note that a device -which only advertises NETIF_F_IP[V6]_CSUM must still obey the csum_start and -csum_offset given in the SKB; if it tries to deduce these itself in hardware -(as some NICs do) the driver should check that the values in the SKB match -those which the hardware will deduce, and if not, fall back to checksumming in -software instead (with skb_csum_hwoffload_help() or one of the -skb_checksum_help() / skb_crc32c_csum_help functions, as mentioned in -include/linux/skbuff.h). - -The stack should, for the most part, assume that checksum offload is supported -by the underlying device. The only place that should check is -validate_xmit_skb(), and the functions it calls directly or indirectly. That -function compares the offload features requested by the SKB (which may include -other offloads besides TX Checksum Offload) and, if they are not supported or -enabled on the device (determined by netdev->features), performs the -corresponding offload in software. In the case of TX Checksum Offload, that -means calling skb_csum_hwoffload_help(skb, features). - - -LCO: Local Checksum Offload -=========================== - -LCO is a technique for efficiently computing the outer checksum of an -encapsulated datagram when the inner checksum is due to be offloaded. - -The ones-complement sum of a correctly checksummed TCP or UDP packet is equal -to the complement of the sum of the pseudo header, because everything else gets -'cancelled out' by the checksum field. This is because the sum was -complemented before being written to the checksum field. - -More generally, this holds in any case where the 'IP-style' ones complement -checksum is used, and thus any checksum that TX Checksum Offload supports. - -That is, if we have set up TX Checksum Offload with a start/offset pair, we -know that after the device has filled in that checksum, the ones complement sum -from csum_start to the end of the packet will be equal to the complement of -whatever value we put in the checksum field beforehand. This allows us to -compute the outer checksum without looking at the payload: we simply stop -summing when we get to csum_start, then add the complement of the 16-bit word -at (csum_start + csum_offset). - -Then, when the true inner checksum is filled in (either by hardware or by -skb_checksum_help()), the outer checksum will become correct by virtue of the -arithmetic. - -LCO is performed by the stack when constructing an outer UDP header for an -encapsulation such as VXLAN or GENEVE, in udp_set_csum(). Similarly for the -IPv6 equivalents, in udp6_set_csum(). - -It is also performed when constructing an IPv4 GRE header, in -net/ipv4/ip_gre.c:build_header(). It is *not* currently performed when -constructing an IPv6 GRE header; the GRE checksum is computed over the whole -packet in net/ipv6/ip6_gre.c:ip6gre_xmit2(), but it should be possible to use -LCO here as IPv6 GRE still uses an IP-style checksum. - -All of the LCO implementations use a helper function lco_csum(), in -include/linux/skbuff.h. - -LCO can safely be used for nested encapsulations; in this case, the outer -encapsulation layer will sum over both its own header and the 'middle' header. -This does mean that the 'middle' header will get summed multiple times, but -there doesn't seem to be a way to avoid that without incurring bigger costs -(e.g. in SKB bloat). - - -RCO: Remote Checksum Offload -============================ - -RCO is a technique for eliding the inner checksum of an encapsulated datagram, -allowing the outer checksum to be offloaded. It does, however, involve a -change to the encapsulation protocols, which the receiver must also support. -For this reason, it is disabled by default. - -RCO is detailed in the following Internet-Drafts: - -* https://tools.ietf.org/html/draft-herbert-remotecsumoffload-00 -* https://tools.ietf.org/html/draft-herbert-vxlan-rco-00 - -In Linux, RCO is implemented individually in each encapsulation protocol, and -most tunnel types have flags controlling its use. For instance, VXLAN has the -flag VXLAN_F_REMCSUM_TX (per struct vxlan_rdst) to indicate that RCO should be -used when transmitting to a given remote destination. diff --git a/Documentation/networking/segmentation-offloads.rst b/Documentation/networking/segmentation-offloads.rst new file mode 100644 index 000000000000..1794bfe98196 --- /dev/null +++ b/Documentation/networking/segmentation-offloads.rst @@ -0,0 +1,184 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=================================================== +Segmentation Offloads in the Linux Networking Stack +=================================================== + + +Introduction +============ + +This document describes a set of techniques in the Linux networking stack +to take advantage of segmentation offload capabilities of various NICs. + +The following technologies are described: + * TCP Segmentation Offload - TSO + * UDP Fragmentation Offload - UFO + * IPIP, SIT, GRE, and UDP Tunnel Offloads + * Generic Segmentation Offload - GSO + * Generic Receive Offload - GRO + * Partial Generic Segmentation Offload - GSO_PARTIAL + * SCTP accelleration with GSO - GSO_BY_FRAGS + + +TCP Segmentation Offload +======================== + +TCP segmentation allows a device to segment a single frame into multiple +frames with a data payload size specified in skb_shinfo()->gso_size. +When TCP segmentation requested the bit for either SKB_GSO_TCPV4 or +SKB_GSO_TCPV6 should be set in skb_shinfo()->gso_type and +skb_shinfo()->gso_size should be set to a non-zero value. + +TCP segmentation is dependent on support for the use of partial checksum +offload. For this reason TSO is normally disabled if the Tx checksum +offload for a given device is disabled. + +In order to support TCP segmentation offload it is necessary to populate +the network and transport header offsets of the skbuff so that the device +drivers will be able determine the offsets of the IP or IPv6 header and the +TCP header. In addition as CHECKSUM_PARTIAL is required csum_start should +also point to the TCP header of the packet. + +For IPv4 segmentation we support one of two types in terms of the IP ID. +The default behavior is to increment the IP ID with every segment. If the +GSO type SKB_GSO_TCP_FIXEDID is specified then we will not increment the IP +ID and all segments will use the same IP ID. If a device has +NETIF_F_TSO_MANGLEID set then the IP ID can be ignored when performing TSO +and we will either increment the IP ID for all frames, or leave it at a +static value based on driver preference. + + +UDP Fragmentation Offload +========================= + +UDP fragmentation offload allows a device to fragment an oversized UDP +datagram into multiple IPv4 fragments. Many of the requirements for UDP +fragmentation offload are the same as TSO. However the IPv4 ID for +fragments should not increment as a single IPv4 datagram is fragmented. + +UFO is deprecated: modern kernels will no longer generate UFO skbs, but can +still receive them from tuntap and similar devices. Offload of UDP-based +tunnel protocols is still supported. + + +IPIP, SIT, GRE, UDP Tunnel, and Remote Checksum Offloads +======================================================== + +In addition to the offloads described above it is possible for a frame to +contain additional headers such as an outer tunnel. In order to account +for such instances an additional set of segmentation offload types were +introduced including SKB_GSO_IPXIP4, SKB_GSO_IPXIP6, SKB_GSO_GRE, and +SKB_GSO_UDP_TUNNEL. These extra segmentation types are used to identify +cases where there are more than just 1 set of headers. For example in the +case of IPIP and SIT we should have the network and transport headers moved +from the standard list of headers to "inner" header offsets. + +Currently only two levels of headers are supported. The convention is to +refer to the tunnel headers as the outer headers, while the encapsulated +data is normally referred to as the inner headers. Below is the list of +calls to access the given headers: + +IPIP/SIT Tunnel:: + + Outer Inner + MAC skb_mac_header + Network skb_network_header skb_inner_network_header + Transport skb_transport_header + +UDP/GRE Tunnel:: + + Outer Inner + MAC skb_mac_header skb_inner_mac_header + Network skb_network_header skb_inner_network_header + Transport skb_transport_header skb_inner_transport_header + +In addition to the above tunnel types there are also SKB_GSO_GRE_CSUM and +SKB_GSO_UDP_TUNNEL_CSUM. These two additional tunnel types reflect the +fact that the outer header also requests to have a non-zero checksum +included in the outer header. + +Finally there is SKB_GSO_TUNNEL_REMCSUM which indicates that a given tunnel +header has requested a remote checksum offload. In this case the inner +headers will be left with a partial checksum and only the outer header +checksum will be computed. + + +Generic Segmentation Offload +============================ + +Generic segmentation offload is a pure software offload that is meant to +deal with cases where device drivers cannot perform the offloads described +above. What occurs in GSO is that a given skbuff will have its data broken +out over multiple skbuffs that have been resized to match the MSS provided +via skb_shinfo()->gso_size. + +Before enabling any hardware segmentation offload a corresponding software +offload is required in GSO. Otherwise it becomes possible for a frame to +be re-routed between devices and end up being unable to be transmitted. + + +Generic Receive Offload +======================= + +Generic receive offload is the complement to GSO. Ideally any frame +assembled by GRO should be segmented to create an identical sequence of +frames using GSO, and any sequence of frames segmented by GSO should be +able to be reassembled back to the original by GRO. The only exception to +this is IPv4 ID in the case that the DF bit is set for a given IP header. +If the value of the IPv4 ID is not sequentially incrementing it will be +altered so that it is when a frame assembled via GRO is segmented via GSO. + + +Partial Generic Segmentation Offload +==================================== + +Partial generic segmentation offload is a hybrid between TSO and GSO. What +it effectively does is take advantage of certain traits of TCP and tunnels +so that instead of having to rewrite the packet headers for each segment +only the inner-most transport header and possibly the outer-most network +header need to be updated. This allows devices that do not support tunnel +offloads or tunnel offloads with checksum to still make use of segmentation. + +With the partial offload what occurs is that all headers excluding the +inner transport header are updated such that they will contain the correct +values for if the header was simply duplicated. The one exception to this +is the outer IPv4 ID field. It is up to the device drivers to guarantee +that the IPv4 ID field is incremented in the case that a given header does +not have the DF bit set. + + +SCTP accelleration with GSO +=========================== + +SCTP - despite the lack of hardware support - can still take advantage of +GSO to pass one large packet through the network stack, rather than +multiple small packets. + +This requires a different approach to other offloads, as SCTP packets +cannot be just segmented to (P)MTU. Rather, the chunks must be contained in +IP segments, padding respected. So unlike regular GSO, SCTP can't just +generate a big skb, set gso_size to the fragmentation point and deliver it +to IP layer. + +Instead, the SCTP protocol layer builds an skb with the segments correctly +padded and stored as chained skbs, and skb_segment() splits based on those. +To signal this, gso_size is set to the special value GSO_BY_FRAGS. + +Therefore, any code in the core networking stack must be aware of the +possibility that gso_size will be GSO_BY_FRAGS and handle that case +appropriately. + +There are some helpers to make this easier: + +- skb_is_gso(skb) && skb_is_gso_sctp(skb) is the best way to see if + an skb is an SCTP GSO skb. + +- For size checks, the skb_gso_validate_*_len family of helpers correctly + considers GSO_BY_FRAGS. + +- For manipulating packets, skb_increase_gso_size and skb_decrease_gso_size + will check for GSO_BY_FRAGS and WARN if asked to manipulate these skbs. + +This also affects drivers with the NETIF_F_FRAGLIST & NETIF_F_GSO_SCTP bits +set. Note also that NETIF_F_GSO_SCTP is included in NETIF_F_GSO_SOFTWARE. diff --git a/Documentation/networking/segmentation-offloads.txt b/Documentation/networking/segmentation-offloads.txt deleted file mode 100644 index 1794bfe98196..000000000000 --- a/Documentation/networking/segmentation-offloads.txt +++ /dev/null @@ -1,184 +0,0 @@ -.. SPDX-License-Identifier: GPL-2.0 - -=================================================== -Segmentation Offloads in the Linux Networking Stack -=================================================== - - -Introduction -============ - -This document describes a set of techniques in the Linux networking stack -to take advantage of segmentation offload capabilities of various NICs. - -The following technologies are described: - * TCP Segmentation Offload - TSO - * UDP Fragmentation Offload - UFO - * IPIP, SIT, GRE, and UDP Tunnel Offloads - * Generic Segmentation Offload - GSO - * Generic Receive Offload - GRO - * Partial Generic Segmentation Offload - GSO_PARTIAL - * SCTP accelleration with GSO - GSO_BY_FRAGS - - -TCP Segmentation Offload -======================== - -TCP segmentation allows a device to segment a single frame into multiple -frames with a data payload size specified in skb_shinfo()->gso_size. -When TCP segmentation requested the bit for either SKB_GSO_TCPV4 or -SKB_GSO_TCPV6 should be set in skb_shinfo()->gso_type and -skb_shinfo()->gso_size should be set to a non-zero value. - -TCP segmentation is dependent on support for the use of partial checksum -offload. For this reason TSO is normally disabled if the Tx checksum -offload for a given device is disabled. - -In order to support TCP segmentation offload it is necessary to populate -the network and transport header offsets of the skbuff so that the device -drivers will be able determine the offsets of the IP or IPv6 header and the -TCP header. In addition as CHECKSUM_PARTIAL is required csum_start should -also point to the TCP header of the packet. - -For IPv4 segmentation we support one of two types in terms of the IP ID. -The default behavior is to increment the IP ID with every segment. If the -GSO type SKB_GSO_TCP_FIXEDID is specified then we will not increment the IP -ID and all segments will use the same IP ID. If a device has -NETIF_F_TSO_MANGLEID set then the IP ID can be ignored when performing TSO -and we will either increment the IP ID for all frames, or leave it at a -static value based on driver preference. - - -UDP Fragmentation Offload -========================= - -UDP fragmentation offload allows a device to fragment an oversized UDP -datagram into multiple IPv4 fragments. Many of the requirements for UDP -fragmentation offload are the same as TSO. However the IPv4 ID for -fragments should not increment as a single IPv4 datagram is fragmented. - -UFO is deprecated: modern kernels will no longer generate UFO skbs, but can -still receive them from tuntap and similar devices. Offload of UDP-based -tunnel protocols is still supported. - - -IPIP, SIT, GRE, UDP Tunnel, and Remote Checksum Offloads -======================================================== - -In addition to the offloads described above it is possible for a frame to -contain additional headers such as an outer tunnel. In order to account -for such instances an additional set of segmentation offload types were -introduced including SKB_GSO_IPXIP4, SKB_GSO_IPXIP6, SKB_GSO_GRE, and -SKB_GSO_UDP_TUNNEL. These extra segmentation types are used to identify -cases where there are more than just 1 set of headers. For example in the -case of IPIP and SIT we should have the network and transport headers moved -from the standard list of headers to "inner" header offsets. - -Currently only two levels of headers are supported. The convention is to -refer to the tunnel headers as the outer headers, while the encapsulated -data is normally referred to as the inner headers. Below is the list of -calls to access the given headers: - -IPIP/SIT Tunnel:: - - Outer Inner - MAC skb_mac_header - Network skb_network_header skb_inner_network_header - Transport skb_transport_header - -UDP/GRE Tunnel:: - - Outer Inner - MAC skb_mac_header skb_inner_mac_header - Network skb_network_header skb_inner_network_header - Transport skb_transport_header skb_inner_transport_header - -In addition to the above tunnel types there are also SKB_GSO_GRE_CSUM and -SKB_GSO_UDP_TUNNEL_CSUM. These two additional tunnel types reflect the -fact that the outer header also requests to have a non-zero checksum -included in the outer header. - -Finally there is SKB_GSO_TUNNEL_REMCSUM which indicates that a given tunnel -header has requested a remote checksum offload. In this case the inner -headers will be left with a partial checksum and only the outer header -checksum will be computed. - - -Generic Segmentation Offload -============================ - -Generic segmentation offload is a pure software offload that is meant to -deal with cases where device drivers cannot perform the offloads described -above. What occurs in GSO is that a given skbuff will have its data broken -out over multiple skbuffs that have been resized to match the MSS provided -via skb_shinfo()->gso_size. - -Before enabling any hardware segmentation offload a corresponding software -offload is required in GSO. Otherwise it becomes possible for a frame to -be re-routed between devices and end up being unable to be transmitted. - - -Generic Receive Offload -======================= - -Generic receive offload is the complement to GSO. Ideally any frame -assembled by GRO should be segmented to create an identical sequence of -frames using GSO, and any sequence of frames segmented by GSO should be -able to be reassembled back to the original by GRO. The only exception to -this is IPv4 ID in the case that the DF bit is set for a given IP header. -If the value of the IPv4 ID is not sequentially incrementing it will be -altered so that it is when a frame assembled via GRO is segmented via GSO. - - -Partial Generic Segmentation Offload -==================================== - -Partial generic segmentation offload is a hybrid between TSO and GSO. What -it effectively does is take advantage of certain traits of TCP and tunnels -so that instead of having to rewrite the packet headers for each segment -only the inner-most transport header and possibly the outer-most network -header need to be updated. This allows devices that do not support tunnel -offloads or tunnel offloads with checksum to still make use of segmentation. - -With the partial offload what occurs is that all headers excluding the -inner transport header are updated such that they will contain the correct -values for if the header was simply duplicated. The one exception to this -is the outer IPv4 ID field. It is up to the device drivers to guarantee -that the IPv4 ID field is incremented in the case that a given header does -not have the DF bit set. - - -SCTP accelleration with GSO -=========================== - -SCTP - despite the lack of hardware support - can still take advantage of -GSO to pass one large packet through the network stack, rather than -multiple small packets. - -This requires a different approach to other offloads, as SCTP packets -cannot be just segmented to (P)MTU. Rather, the chunks must be contained in -IP segments, padding respected. So unlike regular GSO, SCTP can't just -generate a big skb, set gso_size to the fragmentation point and deliver it -to IP layer. - -Instead, the SCTP protocol layer builds an skb with the segments correctly -padded and stored as chained skbs, and skb_segment() splits based on those. -To signal this, gso_size is set to the special value GSO_BY_FRAGS. - -Therefore, any code in the core networking stack must be aware of the -possibility that gso_size will be GSO_BY_FRAGS and handle that case -appropriately. - -There are some helpers to make this easier: - -- skb_is_gso(skb) && skb_is_gso_sctp(skb) is the best way to see if - an skb is an SCTP GSO skb. - -- For size checks, the skb_gso_validate_*_len family of helpers correctly - considers GSO_BY_FRAGS. - -- For manipulating packets, skb_increase_gso_size and skb_decrease_gso_size - will check for GSO_BY_FRAGS and WARN if asked to manipulate these skbs. - -This also affects drivers with the NETIF_F_FRAGLIST & NETIF_F_GSO_SCTP bits -set. Note also that NETIF_F_GSO_SCTP is included in NETIF_F_GSO_SOFTWARE. diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 93f56fddd92a..4e671b46e767 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -4296,7 +4296,7 @@ static inline bool skb_head_is_locked(const struct sk_buff *skb) /* Local Checksum Offload. * Compute outer checksum based on the assumption that the * inner checksum will be offloaded later. - * See Documentation/networking/checksum-offloads.txt for + * See Documentation/networking/checksum-offloads.rst for * explanation of how this works. * Fill in outer checksum adjustment (e.g. with sum of outer * pseudo-header) before calling. -- cgit v1.2.3-58-ga151 From b83eb68cb939e4486c7792f26cacab9ce7a83dc4 Mon Sep 17 00:00:00 2001 From: Otto Sabart Date: Sun, 6 Jan 2019 00:29:28 +0100 Subject: doc: networking: shorten the main title in offloads documents The titles do not look very nice in the table of contents generated by Sphinx. I also think it is obvious that the documents are describing offloads in the Linux Networking Stack. Signed-off-by: Otto Sabart Acked-by: David S. Miller Signed-off-by: Jonathan Corbet --- Documentation/networking/checksum-offloads.rst | 6 +++--- Documentation/networking/segmentation-offloads.rst | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) (limited to 'Documentation/networking') diff --git a/Documentation/networking/checksum-offloads.rst b/Documentation/networking/checksum-offloads.rst index 1a1cd94a3f6d..905c8a84b103 100644 --- a/Documentation/networking/checksum-offloads.rst +++ b/Documentation/networking/checksum-offloads.rst @@ -1,8 +1,8 @@ .. SPDX-License-Identifier: GPL-2.0 -=============================================== -Checksum Offloads in the Linux Networking Stack -=============================================== +================= +Checksum Offloads +================= Introduction diff --git a/Documentation/networking/segmentation-offloads.rst b/Documentation/networking/segmentation-offloads.rst index 1794bfe98196..89d1ee933e9f 100644 --- a/Documentation/networking/segmentation-offloads.rst +++ b/Documentation/networking/segmentation-offloads.rst @@ -1,8 +1,8 @@ .. SPDX-License-Identifier: GPL-2.0 -=================================================== -Segmentation Offloads in the Linux Networking Stack -=================================================== +===================== +Segmentation Offloads +===================== Introduction -- cgit v1.2.3-58-ga151 From d96bedb2b2488519818f1c2654b31069b5136401 Mon Sep 17 00:00:00 2001 From: Otto Sabart Date: Sun, 6 Jan 2019 00:29:41 +0100 Subject: doc: networking: add offload documents into main index file This patch just adds references to offload documents into main table of contents in network documentation. Signed-off-by: Otto Sabart Acked-by: David S. Miller Signed-off-by: Jonathan Corbet --- Documentation/networking/index.rst | 2 ++ 1 file changed, 2 insertions(+) (limited to 'Documentation/networking') diff --git a/Documentation/networking/index.rst b/Documentation/networking/index.rst index 6a47629ef8ed..7034214cf87e 100644 --- a/Documentation/networking/index.rst +++ b/Documentation/networking/index.rst @@ -32,6 +32,8 @@ Contents: alias bridge snmp_counter + checksum-offloads + segmentation-offloads .. only:: subproject -- cgit v1.2.3-58-ga151 From e6e37f636815075c0055f33f42ebea4fb057def6 Mon Sep 17 00:00:00 2001 From: Otto Sabart Date: Fri, 18 Jan 2019 21:38:32 +0100 Subject: doc: networking: integrate scaling document into doc tree Convert scaling document into reStructuredText and add reference to scaling document into main table of contents in network documentation. There are no semantic changes. There are no references to "scaling.txt" file. Whole kernel tree was checked using: $ grep -r "scaling\.txt" Signed-off-by: Otto Sabart Signed-off-by: Jonathan Corbet --- Documentation/networking/index.rst | 1 + Documentation/networking/scaling.rst | 523 +++++++++++++++++++++++++++++++++++ Documentation/networking/scaling.txt | 484 -------------------------------- 3 files changed, 524 insertions(+), 484 deletions(-) create mode 100644 Documentation/networking/scaling.rst delete mode 100644 Documentation/networking/scaling.txt (limited to 'Documentation/networking') diff --git a/Documentation/networking/index.rst b/Documentation/networking/index.rst index 7034214cf87e..44a0f1939fcd 100644 --- a/Documentation/networking/index.rst +++ b/Documentation/networking/index.rst @@ -34,6 +34,7 @@ Contents: snmp_counter checksum-offloads segmentation-offloads + scaling .. only:: subproject diff --git a/Documentation/networking/scaling.rst b/Documentation/networking/scaling.rst new file mode 100644 index 000000000000..f78d7bf27ff5 --- /dev/null +++ b/Documentation/networking/scaling.rst @@ -0,0 +1,523 @@ +.. SPDX-License-Identifier: GPL-2.0 + +===================================== +Scaling in the Linux Networking Stack +===================================== + + +Introduction +============ + +This document describes a set of complementary techniques in the Linux +networking stack to increase parallelism and improve performance for +multi-processor systems. + +The following technologies are described: + +- RSS: Receive Side Scaling +- RPS: Receive Packet Steering +- RFS: Receive Flow Steering +- Accelerated Receive Flow Steering +- XPS: Transmit Packet Steering + + +RSS: Receive Side Scaling +========================= + +Contemporary NICs support multiple receive and transmit descriptor queues +(multi-queue). On reception, a NIC can send different packets to different +queues to distribute processing among CPUs. The NIC distributes packets by +applying a filter to each packet that assigns it to one of a small number +of logical flows. Packets for each flow are steered to a separate receive +queue, which in turn can be processed by separate CPUs. This mechanism is +generally known as “Receive-side Scaling” (RSS). The goal of RSS and +the other scaling techniques is to increase performance uniformly. +Multi-queue distribution can also be used for traffic prioritization, but +that is not the focus of these techniques. + +The filter used in RSS is typically a hash function over the network +and/or transport layer headers-- for example, a 4-tuple hash over +IP addresses and TCP ports of a packet. The most common hardware +implementation of RSS uses a 128-entry indirection table where each entry +stores a queue number. The receive queue for a packet is determined +by masking out the low order seven bits of the computed hash for the +packet (usually a Toeplitz hash), taking this number as a key into the +indirection table and reading the corresponding value. + +Some advanced NICs allow steering packets to queues based on +programmable filters. For example, webserver bound TCP port 80 packets +can be directed to their own receive queue. Such “n-tuple” filters can +be configured from ethtool (--config-ntuple). + + +RSS Configuration +----------------- + +The driver for a multi-queue capable NIC typically provides a kernel +module parameter for specifying the number of hardware queues to +configure. In the bnx2x driver, for instance, this parameter is called +num_queues. A typical RSS configuration would be to have one receive queue +for each CPU if the device supports enough queues, or otherwise at least +one for each memory domain, where a memory domain is a set of CPUs that +share a particular memory level (L1, L2, NUMA node, etc.). + +The indirection table of an RSS device, which resolves a queue by masked +hash, is usually programmed by the driver at initialization. The +default mapping is to distribute the queues evenly in the table, but the +indirection table can be retrieved and modified at runtime using ethtool +commands (--show-rxfh-indir and --set-rxfh-indir). Modifying the +indirection table could be done to give different queues different +relative weights. + + +RSS IRQ Configuration +~~~~~~~~~~~~~~~~~~~~~ + +Each receive queue has a separate IRQ associated with it. The NIC triggers +this to notify a CPU when new packets arrive on the given queue. The +signaling path for PCIe devices uses message signaled interrupts (MSI-X), +that can route each interrupt to a particular CPU. The active mapping +of queues to IRQs can be determined from /proc/interrupts. By default, +an IRQ may be handled on any CPU. Because a non-negligible part of packet +processing takes place in receive interrupt handling, it is advantageous +to spread receive interrupts between CPUs. To manually adjust the IRQ +affinity of each interrupt see Documentation/IRQ-affinity.txt. Some systems +will be running irqbalance, a daemon that dynamically optimizes IRQ +assignments and as a result may override any manual settings. + + +Suggested Configuration +~~~~~~~~~~~~~~~~~~~~~~~ + +RSS should be enabled when latency is a concern or whenever receive +interrupt processing forms a bottleneck. Spreading load between CPUs +decreases queue length. For low latency networking, the optimal setting +is to allocate as many queues as there are CPUs in the system (or the +NIC maximum, if lower). The most efficient high-rate configuration +is likely the one with the smallest number of receive queues where no +receive queue overflows due to a saturated CPU, because in default +mode with interrupt coalescing enabled, the aggregate number of +interrupts (and thus work) grows with each additional queue. + +Per-cpu load can be observed using the mpstat utility, but note that on +processors with hyperthreading (HT), each hyperthread is represented as +a separate CPU. For interrupt handling, HT has shown no benefit in +initial tests, so limit the number of queues to the number of CPU cores +in the system. + + +RPS: Receive Packet Steering +============================ + +Receive Packet Steering (RPS) is logically a software implementation of +RSS. Being in software, it is necessarily called later in the datapath. +Whereas RSS selects the queue and hence CPU that will run the hardware +interrupt handler, RPS selects the CPU to perform protocol processing +above the interrupt handler. This is accomplished by placing the packet +on the desired CPU’s backlog queue and waking up the CPU for processing. +RPS has some advantages over RSS: + +1) it can be used with any NIC +2) software filters can easily be added to hash over new protocols +3) it does not increase hardware device interrupt rate (although it does + introduce inter-processor interrupts (IPIs)) + +RPS is called during bottom half of the receive interrupt handler, when +a driver sends a packet up the network stack with netif_rx() or +netif_receive_skb(). These call the get_rps_cpu() function, which +selects the queue that should process a packet. + +The first step in determining the target CPU for RPS is to calculate a +flow hash over the packet’s addresses or ports (2-tuple or 4-tuple hash +depending on the protocol). This serves as a consistent hash of the +associated flow of the packet. The hash is either provided by hardware +or will be computed in the stack. Capable hardware can pass the hash in +the receive descriptor for the packet; this would usually be the same +hash used for RSS (e.g. computed Toeplitz hash). The hash is saved in +skb->hash and can be used elsewhere in the stack as a hash of the +packet’s flow. + +Each receive hardware queue has an associated list of CPUs to which +RPS may enqueue packets for processing. For each received packet, +an index into the list is computed from the flow hash modulo the size +of the list. The indexed CPU is the target for processing the packet, +and the packet is queued to the tail of that CPU’s backlog queue. At +the end of the bottom half routine, IPIs are sent to any CPUs for which +packets have been queued to their backlog queue. The IPI wakes backlog +processing on the remote CPU, and any queued packets are then processed +up the networking stack. + + +RPS Configuration +----------------- + +RPS requires a kernel compiled with the CONFIG_RPS kconfig symbol (on +by default for SMP). Even when compiled in, RPS remains disabled until +explicitly configured. The list of CPUs to which RPS may forward traffic +can be configured for each receive queue using a sysfs file entry:: + + /sys/class/net//queues/rx-/rps_cpus + +This file implements a bitmap of CPUs. RPS is disabled when it is zero +(the default), in which case packets are processed on the interrupting +CPU. Documentation/IRQ-affinity.txt explains how CPUs are assigned to +the bitmap. + + +Suggested Configuration +~~~~~~~~~~~~~~~~~~~~~~~ + +For a single queue device, a typical RPS configuration would be to set +the rps_cpus to the CPUs in the same memory domain of the interrupting +CPU. If NUMA locality is not an issue, this could also be all CPUs in +the system. At high interrupt rate, it might be wise to exclude the +interrupting CPU from the map since that already performs much work. + +For a multi-queue system, if RSS is configured so that a hardware +receive queue is mapped to each CPU, then RPS is probably redundant +and unnecessary. If there are fewer hardware queues than CPUs, then +RPS might be beneficial if the rps_cpus for each queue are the ones that +share the same memory domain as the interrupting CPU for that queue. + + +RPS Flow Limit +-------------- + +RPS scales kernel receive processing across CPUs without introducing +reordering. The trade-off to sending all packets from the same flow +to the same CPU is CPU load imbalance if flows vary in packet rate. +In the extreme case a single flow dominates traffic. Especially on +common server workloads with many concurrent connections, such +behavior indicates a problem such as a misconfiguration or spoofed +source Denial of Service attack. + +Flow Limit is an optional RPS feature that prioritizes small flows +during CPU contention by dropping packets from large flows slightly +ahead of those from small flows. It is active only when an RPS or RFS +destination CPU approaches saturation. Once a CPU's input packet +queue exceeds half the maximum queue length (as set by sysctl +net.core.netdev_max_backlog), the kernel starts a per-flow packet +count over the last 256 packets. If a flow exceeds a set ratio (by +default, half) of these packets when a new packet arrives, then the +new packet is dropped. Packets from other flows are still only +dropped once the input packet queue reaches netdev_max_backlog. +No packets are dropped when the input packet queue length is below +the threshold, so flow limit does not sever connections outright: +even large flows maintain connectivity. + + +Interface +~~~~~~~~~ + +Flow limit is compiled in by default (CONFIG_NET_FLOW_LIMIT), but not +turned on. It is implemented for each CPU independently (to avoid lock +and cache contention) and toggled per CPU by setting the relevant bit +in sysctl net.core.flow_limit_cpu_bitmap. It exposes the same CPU +bitmap interface as rps_cpus (see above) when called from procfs:: + + /proc/sys/net/core/flow_limit_cpu_bitmap + +Per-flow rate is calculated by hashing each packet into a hashtable +bucket and incrementing a per-bucket counter. The hash function is +the same that selects a CPU in RPS, but as the number of buckets can +be much larger than the number of CPUs, flow limit has finer-grained +identification of large flows and fewer false positives. The default +table has 4096 buckets. This value can be modified through sysctl:: + + net.core.flow_limit_table_len + +The value is only consulted when a new table is allocated. Modifying +it does not update active tables. + + +Suggested Configuration +~~~~~~~~~~~~~~~~~~~~~~~ + +Flow limit is useful on systems with many concurrent connections, +where a single connection taking up 50% of a CPU indicates a problem. +In such environments, enable the feature on all CPUs that handle +network rx interrupts (as set in /proc/irq/N/smp_affinity). + +The feature depends on the input packet queue length to exceed +the flow limit threshold (50%) + the flow history length (256). +Setting net.core.netdev_max_backlog to either 1000 or 10000 +performed well in experiments. + + +RFS: Receive Flow Steering +========================== + +While RPS steers packets solely based on hash, and thus generally +provides good load distribution, it does not take into account +application locality. This is accomplished by Receive Flow Steering +(RFS). The goal of RFS is to increase datacache hitrate by steering +kernel processing of packets to the CPU where the application thread +consuming the packet is running. RFS relies on the same RPS mechanisms +to enqueue packets onto the backlog of another CPU and to wake up that +CPU. + +In RFS, packets are not forwarded directly by the value of their hash, +but the hash is used as index into a flow lookup table. This table maps +flows to the CPUs where those flows are being processed. The flow hash +(see RPS section above) is used to calculate the index into this table. +The CPU recorded in each entry is the one which last processed the flow. +If an entry does not hold a valid CPU, then packets mapped to that entry +are steered using plain RPS. Multiple table entries may point to the +same CPU. Indeed, with many flows and few CPUs, it is very likely that +a single application thread handles flows with many different flow hashes. + +rps_sock_flow_table is a global flow table that contains the *desired* CPU +for flows: the CPU that is currently processing the flow in userspace. +Each table value is a CPU index that is updated during calls to recvmsg +and sendmsg (specifically, inet_recvmsg(), inet_sendmsg(), inet_sendpage() +and tcp_splice_read()). + +When the scheduler moves a thread to a new CPU while it has outstanding +receive packets on the old CPU, packets may arrive out of order. To +avoid this, RFS uses a second flow table to track outstanding packets +for each flow: rps_dev_flow_table is a table specific to each hardware +receive queue of each device. Each table value stores a CPU index and a +counter. The CPU index represents the *current* CPU onto which packets +for this flow are enqueued for further kernel processing. Ideally, kernel +and userspace processing occur on the same CPU, and hence the CPU index +in both tables is identical. This is likely false if the scheduler has +recently migrated a userspace thread while the kernel still has packets +enqueued for kernel processing on the old CPU. + +The counter in rps_dev_flow_table values records the length of the current +CPU's backlog when a packet in this flow was last enqueued. Each backlog +queue has a head counter that is incremented on dequeue. A tail counter +is computed as head counter + queue length. In other words, the counter +in rps_dev_flow[i] records the last element in flow i that has +been enqueued onto the currently designated CPU for flow i (of course, +entry i is actually selected by hash and multiple flows may hash to the +same entry i). + +And now the trick for avoiding out of order packets: when selecting the +CPU for packet processing (from get_rps_cpu()) the rps_sock_flow table +and the rps_dev_flow table of the queue that the packet was received on +are compared. If the desired CPU for the flow (found in the +rps_sock_flow table) matches the current CPU (found in the rps_dev_flow +table), the packet is enqueued onto that CPU’s backlog. If they differ, +the current CPU is updated to match the desired CPU if one of the +following is true: + + - The current CPU's queue head counter >= the recorded tail counter + value in rps_dev_flow[i] + - The current CPU is unset (>= nr_cpu_ids) + - The current CPU is offline + +After this check, the packet is sent to the (possibly updated) current +CPU. These rules aim to ensure that a flow only moves to a new CPU when +there are no packets outstanding on the old CPU, as the outstanding +packets could arrive later than those about to be processed on the new +CPU. + + +RFS Configuration +----------------- + +RFS is only available if the kconfig symbol CONFIG_RPS is enabled (on +by default for SMP). The functionality remains disabled until explicitly +configured. The number of entries in the global flow table is set through:: + + /proc/sys/net/core/rps_sock_flow_entries + +The number of entries in the per-queue flow table are set through:: + + /sys/class/net//queues/rx-/rps_flow_cnt + + +Suggested Configuration +~~~~~~~~~~~~~~~~~~~~~~~ + +Both of these need to be set before RFS is enabled for a receive queue. +Values for both are rounded up to the nearest power of two. The +suggested flow count depends on the expected number of active connections +at any given time, which may be significantly less than the number of open +connections. We have found that a value of 32768 for rps_sock_flow_entries +works fairly well on a moderately loaded server. + +For a single queue device, the rps_flow_cnt value for the single queue +would normally be configured to the same value as rps_sock_flow_entries. +For a multi-queue device, the rps_flow_cnt for each queue might be +configured as rps_sock_flow_entries / N, where N is the number of +queues. So for instance, if rps_sock_flow_entries is set to 32768 and there +are 16 configured receive queues, rps_flow_cnt for each queue might be +configured as 2048. + + +Accelerated RFS +=============== + +Accelerated RFS is to RFS what RSS is to RPS: a hardware-accelerated load +balancing mechanism that uses soft state to steer flows based on where +the application thread consuming the packets of each flow is running. +Accelerated RFS should perform better than RFS since packets are sent +directly to a CPU local to the thread consuming the data. The target CPU +will either be the same CPU where the application runs, or at least a CPU +which is local to the application thread’s CPU in the cache hierarchy. + +To enable accelerated RFS, the networking stack calls the +ndo_rx_flow_steer driver function to communicate the desired hardware +queue for packets matching a particular flow. The network stack +automatically calls this function every time a flow entry in +rps_dev_flow_table is updated. The driver in turn uses a device specific +method to program the NIC to steer the packets. + +The hardware queue for a flow is derived from the CPU recorded in +rps_dev_flow_table. The stack consults a CPU to hardware queue map which +is maintained by the NIC driver. This is an auto-generated reverse map of +the IRQ affinity table shown by /proc/interrupts. Drivers can use +functions in the cpu_rmap (“CPU affinity reverse map”) kernel library +to populate the map. For each CPU, the corresponding queue in the map is +set to be one whose processing CPU is closest in cache locality. + + +Accelerated RFS Configuration +----------------------------- + +Accelerated RFS is only available if the kernel is compiled with +CONFIG_RFS_ACCEL and support is provided by the NIC device and driver. +It also requires that ntuple filtering is enabled via ethtool. The map +of CPU to queues is automatically deduced from the IRQ affinities +configured for each receive queue by the driver, so no additional +configuration should be necessary. + + +Suggested Configuration +~~~~~~~~~~~~~~~~~~~~~~~ + +This technique should be enabled whenever one wants to use RFS and the +NIC supports hardware acceleration. + + +XPS: Transmit Packet Steering +============================= + +Transmit Packet Steering is a mechanism for intelligently selecting +which transmit queue to use when transmitting a packet on a multi-queue +device. This can be accomplished by recording two kinds of maps, either +a mapping of CPU to hardware queue(s) or a mapping of receive queue(s) +to hardware transmit queue(s). + +1. XPS using CPUs map + +The goal of this mapping is usually to assign queues +exclusively to a subset of CPUs, where the transmit completions for +these queues are processed on a CPU within this set. This choice +provides two benefits. First, contention on the device queue lock is +significantly reduced since fewer CPUs contend for the same queue +(contention can be eliminated completely if each CPU has its own +transmit queue). Secondly, cache miss rate on transmit completion is +reduced, in particular for data cache lines that hold the sk_buff +structures. + +2. XPS using receive queues map + +This mapping is used to pick transmit queue based on the receive +queue(s) map configuration set by the administrator. A set of receive +queues can be mapped to a set of transmit queues (many:many), although +the common use case is a 1:1 mapping. This will enable sending packets +on the same queue associations for transmit and receive. This is useful for +busy polling multi-threaded workloads where there are challenges in +associating a given CPU to a given application thread. The application +threads are not pinned to CPUs and each thread handles packets +received on a single queue. The receive queue number is cached in the +socket for the connection. In this model, sending the packets on the same +transmit queue corresponding to the associated receive queue has benefits +in keeping the CPU overhead low. Transmit completion work is locked into +the same queue-association that a given application is polling on. This +avoids the overhead of triggering an interrupt on another CPU. When the +application cleans up the packets during the busy poll, transmit completion +may be processed along with it in the same thread context and so result in +reduced latency. + +XPS is configured per transmit queue by setting a bitmap of +CPUs/receive-queues that may use that queue to transmit. The reverse +mapping, from CPUs to transmit queues or from receive-queues to transmit +queues, is computed and maintained for each network device. When +transmitting the first packet in a flow, the function get_xps_queue() is +called to select a queue. This function uses the ID of the receive queue +for the socket connection for a match in the receive queue-to-transmit queue +lookup table. Alternatively, this function can also use the ID of the +running CPU as a key into the CPU-to-queue lookup table. If the +ID matches a single queue, that is used for transmission. If multiple +queues match, one is selected by using the flow hash to compute an index +into the set. When selecting the transmit queue based on receive queue(s) +map, the transmit device is not validated against the receive device as it +requires expensive lookup operation in the datapath. + +The queue chosen for transmitting a particular flow is saved in the +corresponding socket structure for the flow (e.g. a TCP connection). +This transmit queue is used for subsequent packets sent on the flow to +prevent out of order (ooo) packets. The choice also amortizes the cost +of calling get_xps_queues() over all packets in the flow. To avoid +ooo packets, the queue for a flow can subsequently only be changed if +skb->ooo_okay is set for a packet in the flow. This flag indicates that +there are no outstanding packets in the flow, so the transmit queue can +change without the risk of generating out of order packets. The +transport layer is responsible for setting ooo_okay appropriately. TCP, +for instance, sets the flag when all data for a connection has been +acknowledged. + +XPS Configuration +----------------- + +XPS is only available if the kconfig symbol CONFIG_XPS is enabled (on by +default for SMP). The functionality remains disabled until explicitly +configured. To enable XPS, the bitmap of CPUs/receive-queues that may +use a transmit queue is configured using the sysfs file entry: + +For selection based on CPUs map:: + + /sys/class/net//queues/tx-/xps_cpus + +For selection based on receive-queues map:: + + /sys/class/net//queues/tx-/xps_rxqs + + +Suggested Configuration +~~~~~~~~~~~~~~~~~~~~~~~ + +For a network device with a single transmission queue, XPS configuration +has no effect, since there is no choice in this case. In a multi-queue +system, XPS is preferably configured so that each CPU maps onto one queue. +If there are as many queues as there are CPUs in the system, then each +queue can also map onto one CPU, resulting in exclusive pairings that +experience no contention. If there are fewer queues than CPUs, then the +best CPUs to share a given queue are probably those that share the cache +with the CPU that processes transmit completions for that queue +(transmit interrupts). + +For transmit queue selection based on receive queue(s), XPS has to be +explicitly configured mapping receive-queue(s) to transmit queue(s). If the +user configuration for receive-queue map does not apply, then the transmit +queue is selected based on the CPUs map. + + +Per TX Queue rate limitation +============================ + +These are rate-limitation mechanisms implemented by HW, where currently +a max-rate attribute is supported, by setting a Mbps value to:: + + /sys/class/net//queues/tx-/tx_maxrate + +A value of zero means disabled, and this is the default. + + +Further Information +=================== +RPS and RFS were introduced in kernel 2.6.35. XPS was incorporated into +2.6.38. Original patches were submitted by Tom Herbert +(therbert@google.com) + +Accelerated RFS was introduced in 2.6.35. Original patches were +submitted by Ben Hutchings (bwh@kernel.org) + +Authors: + +- Tom Herbert (therbert@google.com) +- Willem de Bruijn (willemb@google.com) diff --git a/Documentation/networking/scaling.txt b/Documentation/networking/scaling.txt deleted file mode 100644 index b7056a8a0540..000000000000 --- a/Documentation/networking/scaling.txt +++ /dev/null @@ -1,484 +0,0 @@ -Scaling in the Linux Networking Stack - - -Introduction -============ - -This document describes a set of complementary techniques in the Linux -networking stack to increase parallelism and improve performance for -multi-processor systems. - -The following technologies are described: - - RSS: Receive Side Scaling - RPS: Receive Packet Steering - RFS: Receive Flow Steering - Accelerated Receive Flow Steering - XPS: Transmit Packet Steering - - -RSS: Receive Side Scaling -========================= - -Contemporary NICs support multiple receive and transmit descriptor queues -(multi-queue). On reception, a NIC can send different packets to different -queues to distribute processing among CPUs. The NIC distributes packets by -applying a filter to each packet that assigns it to one of a small number -of logical flows. Packets for each flow are steered to a separate receive -queue, which in turn can be processed by separate CPUs. This mechanism is -generally known as “Receive-side Scaling” (RSS). The goal of RSS and -the other scaling techniques is to increase performance uniformly. -Multi-queue distribution can also be used for traffic prioritization, but -that is not the focus of these techniques. - -The filter used in RSS is typically a hash function over the network -and/or transport layer headers-- for example, a 4-tuple hash over -IP addresses and TCP ports of a packet. The most common hardware -implementation of RSS uses a 128-entry indirection table where each entry -stores a queue number. The receive queue for a packet is determined -by masking out the low order seven bits of the computed hash for the -packet (usually a Toeplitz hash), taking this number as a key into the -indirection table and reading the corresponding value. - -Some advanced NICs allow steering packets to queues based on -programmable filters. For example, webserver bound TCP port 80 packets -can be directed to their own receive queue. Such “n-tuple” filters can -be configured from ethtool (--config-ntuple). - -==== RSS Configuration - -The driver for a multi-queue capable NIC typically provides a kernel -module parameter for specifying the number of hardware queues to -configure. In the bnx2x driver, for instance, this parameter is called -num_queues. A typical RSS configuration would be to have one receive queue -for each CPU if the device supports enough queues, or otherwise at least -one for each memory domain, where a memory domain is a set of CPUs that -share a particular memory level (L1, L2, NUMA node, etc.). - -The indirection table of an RSS device, which resolves a queue by masked -hash, is usually programmed by the driver at initialization. The -default mapping is to distribute the queues evenly in the table, but the -indirection table can be retrieved and modified at runtime using ethtool -commands (--show-rxfh-indir and --set-rxfh-indir). Modifying the -indirection table could be done to give different queues different -relative weights. - -== RSS IRQ Configuration - -Each receive queue has a separate IRQ associated with it. The NIC triggers -this to notify a CPU when new packets arrive on the given queue. The -signaling path for PCIe devices uses message signaled interrupts (MSI-X), -that can route each interrupt to a particular CPU. The active mapping -of queues to IRQs can be determined from /proc/interrupts. By default, -an IRQ may be handled on any CPU. Because a non-negligible part of packet -processing takes place in receive interrupt handling, it is advantageous -to spread receive interrupts between CPUs. To manually adjust the IRQ -affinity of each interrupt see Documentation/IRQ-affinity.txt. Some systems -will be running irqbalance, a daemon that dynamically optimizes IRQ -assignments and as a result may override any manual settings. - -== Suggested Configuration - -RSS should be enabled when latency is a concern or whenever receive -interrupt processing forms a bottleneck. Spreading load between CPUs -decreases queue length. For low latency networking, the optimal setting -is to allocate as many queues as there are CPUs in the system (or the -NIC maximum, if lower). The most efficient high-rate configuration -is likely the one with the smallest number of receive queues where no -receive queue overflows due to a saturated CPU, because in default -mode with interrupt coalescing enabled, the aggregate number of -interrupts (and thus work) grows with each additional queue. - -Per-cpu load can be observed using the mpstat utility, but note that on -processors with hyperthreading (HT), each hyperthread is represented as -a separate CPU. For interrupt handling, HT has shown no benefit in -initial tests, so limit the number of queues to the number of CPU cores -in the system. - - -RPS: Receive Packet Steering -============================ - -Receive Packet Steering (RPS) is logically a software implementation of -RSS. Being in software, it is necessarily called later in the datapath. -Whereas RSS selects the queue and hence CPU that will run the hardware -interrupt handler, RPS selects the CPU to perform protocol processing -above the interrupt handler. This is accomplished by placing the packet -on the desired CPU’s backlog queue and waking up the CPU for processing. -RPS has some advantages over RSS: 1) it can be used with any NIC, -2) software filters can easily be added to hash over new protocols, -3) it does not increase hardware device interrupt rate (although it does -introduce inter-processor interrupts (IPIs)). - -RPS is called during bottom half of the receive interrupt handler, when -a driver sends a packet up the network stack with netif_rx() or -netif_receive_skb(). These call the get_rps_cpu() function, which -selects the queue that should process a packet. - -The first step in determining the target CPU for RPS is to calculate a -flow hash over the packet’s addresses or ports (2-tuple or 4-tuple hash -depending on the protocol). This serves as a consistent hash of the -associated flow of the packet. The hash is either provided by hardware -or will be computed in the stack. Capable hardware can pass the hash in -the receive descriptor for the packet; this would usually be the same -hash used for RSS (e.g. computed Toeplitz hash). The hash is saved in -skb->hash and can be used elsewhere in the stack as a hash of the -packet’s flow. - -Each receive hardware queue has an associated list of CPUs to which -RPS may enqueue packets for processing. For each received packet, -an index into the list is computed from the flow hash modulo the size -of the list. The indexed CPU is the target for processing the packet, -and the packet is queued to the tail of that CPU’s backlog queue. At -the end of the bottom half routine, IPIs are sent to any CPUs for which -packets have been queued to their backlog queue. The IPI wakes backlog -processing on the remote CPU, and any queued packets are then processed -up the networking stack. - -==== RPS Configuration - -RPS requires a kernel compiled with the CONFIG_RPS kconfig symbol (on -by default for SMP). Even when compiled in, RPS remains disabled until -explicitly configured. The list of CPUs to which RPS may forward traffic -can be configured for each receive queue using a sysfs file entry: - - /sys/class/net//queues/rx-/rps_cpus - -This file implements a bitmap of CPUs. RPS is disabled when it is zero -(the default), in which case packets are processed on the interrupting -CPU. Documentation/IRQ-affinity.txt explains how CPUs are assigned to -the bitmap. - -== Suggested Configuration - -For a single queue device, a typical RPS configuration would be to set -the rps_cpus to the CPUs in the same memory domain of the interrupting -CPU. If NUMA locality is not an issue, this could also be all CPUs in -the system. At high interrupt rate, it might be wise to exclude the -interrupting CPU from the map since that already performs much work. - -For a multi-queue system, if RSS is configured so that a hardware -receive queue is mapped to each CPU, then RPS is probably redundant -and unnecessary. If there are fewer hardware queues than CPUs, then -RPS might be beneficial if the rps_cpus for each queue are the ones that -share the same memory domain as the interrupting CPU for that queue. - -==== RPS Flow Limit - -RPS scales kernel receive processing across CPUs without introducing -reordering. The trade-off to sending all packets from the same flow -to the same CPU is CPU load imbalance if flows vary in packet rate. -In the extreme case a single flow dominates traffic. Especially on -common server workloads with many concurrent connections, such -behavior indicates a problem such as a misconfiguration or spoofed -source Denial of Service attack. - -Flow Limit is an optional RPS feature that prioritizes small flows -during CPU contention by dropping packets from large flows slightly -ahead of those from small flows. It is active only when an RPS or RFS -destination CPU approaches saturation. Once a CPU's input packet -queue exceeds half the maximum queue length (as set by sysctl -net.core.netdev_max_backlog), the kernel starts a per-flow packet -count over the last 256 packets. If a flow exceeds a set ratio (by -default, half) of these packets when a new packet arrives, then the -new packet is dropped. Packets from other flows are still only -dropped once the input packet queue reaches netdev_max_backlog. -No packets are dropped when the input packet queue length is below -the threshold, so flow limit does not sever connections outright: -even large flows maintain connectivity. - -== Interface - -Flow limit is compiled in by default (CONFIG_NET_FLOW_LIMIT), but not -turned on. It is implemented for each CPU independently (to avoid lock -and cache contention) and toggled per CPU by setting the relevant bit -in sysctl net.core.flow_limit_cpu_bitmap. It exposes the same CPU -bitmap interface as rps_cpus (see above) when called from procfs: - - /proc/sys/net/core/flow_limit_cpu_bitmap - -Per-flow rate is calculated by hashing each packet into a hashtable -bucket and incrementing a per-bucket counter. The hash function is -the same that selects a CPU in RPS, but as the number of buckets can -be much larger than the number of CPUs, flow limit has finer-grained -identification of large flows and fewer false positives. The default -table has 4096 buckets. This value can be modified through sysctl - - net.core.flow_limit_table_len - -The value is only consulted when a new table is allocated. Modifying -it does not update active tables. - -== Suggested Configuration - -Flow limit is useful on systems with many concurrent connections, -where a single connection taking up 50% of a CPU indicates a problem. -In such environments, enable the feature on all CPUs that handle -network rx interrupts (as set in /proc/irq/N/smp_affinity). - -The feature depends on the input packet queue length to exceed -the flow limit threshold (50%) + the flow history length (256). -Setting net.core.netdev_max_backlog to either 1000 or 10000 -performed well in experiments. - - -RFS: Receive Flow Steering -========================== - -While RPS steers packets solely based on hash, and thus generally -provides good load distribution, it does not take into account -application locality. This is accomplished by Receive Flow Steering -(RFS). The goal of RFS is to increase datacache hitrate by steering -kernel processing of packets to the CPU where the application thread -consuming the packet is running. RFS relies on the same RPS mechanisms -to enqueue packets onto the backlog of another CPU and to wake up that -CPU. - -In RFS, packets are not forwarded directly by the value of their hash, -but the hash is used as index into a flow lookup table. This table maps -flows to the CPUs where those flows are being processed. The flow hash -(see RPS section above) is used to calculate the index into this table. -The CPU recorded in each entry is the one which last processed the flow. -If an entry does not hold a valid CPU, then packets mapped to that entry -are steered using plain RPS. Multiple table entries may point to the -same CPU. Indeed, with many flows and few CPUs, it is very likely that -a single application thread handles flows with many different flow hashes. - -rps_sock_flow_table is a global flow table that contains the *desired* CPU -for flows: the CPU that is currently processing the flow in userspace. -Each table value is a CPU index that is updated during calls to recvmsg -and sendmsg (specifically, inet_recvmsg(), inet_sendmsg(), inet_sendpage() -and tcp_splice_read()). - -When the scheduler moves a thread to a new CPU while it has outstanding -receive packets on the old CPU, packets may arrive out of order. To -avoid this, RFS uses a second flow table to track outstanding packets -for each flow: rps_dev_flow_table is a table specific to each hardware -receive queue of each device. Each table value stores a CPU index and a -counter. The CPU index represents the *current* CPU onto which packets -for this flow are enqueued for further kernel processing. Ideally, kernel -and userspace processing occur on the same CPU, and hence the CPU index -in both tables is identical. This is likely false if the scheduler has -recently migrated a userspace thread while the kernel still has packets -enqueued for kernel processing on the old CPU. - -The counter in rps_dev_flow_table values records the length of the current -CPU's backlog when a packet in this flow was last enqueued. Each backlog -queue has a head counter that is incremented on dequeue. A tail counter -is computed as head counter + queue length. In other words, the counter -in rps_dev_flow[i] records the last element in flow i that has -been enqueued onto the currently designated CPU for flow i (of course, -entry i is actually selected by hash and multiple flows may hash to the -same entry i). - -And now the trick for avoiding out of order packets: when selecting the -CPU for packet processing (from get_rps_cpu()) the rps_sock_flow table -and the rps_dev_flow table of the queue that the packet was received on -are compared. If the desired CPU for the flow (found in the -rps_sock_flow table) matches the current CPU (found in the rps_dev_flow -table), the packet is enqueued onto that CPU’s backlog. If they differ, -the current CPU is updated to match the desired CPU if one of the -following is true: - -- The current CPU's queue head counter >= the recorded tail counter - value in rps_dev_flow[i] -- The current CPU is unset (>= nr_cpu_ids) -- The current CPU is offline - -After this check, the packet is sent to the (possibly updated) current -CPU. These rules aim to ensure that a flow only moves to a new CPU when -there are no packets outstanding on the old CPU, as the outstanding -packets could arrive later than those about to be processed on the new -CPU. - -==== RFS Configuration - -RFS is only available if the kconfig symbol CONFIG_RPS is enabled (on -by default for SMP). The functionality remains disabled until explicitly -configured. The number of entries in the global flow table is set through: - - /proc/sys/net/core/rps_sock_flow_entries - -The number of entries in the per-queue flow table are set through: - - /sys/class/net//queues/rx-/rps_flow_cnt - -== Suggested Configuration - -Both of these need to be set before RFS is enabled for a receive queue. -Values for both are rounded up to the nearest power of two. The -suggested flow count depends on the expected number of active connections -at any given time, which may be significantly less than the number of open -connections. We have found that a value of 32768 for rps_sock_flow_entries -works fairly well on a moderately loaded server. - -For a single queue device, the rps_flow_cnt value for the single queue -would normally be configured to the same value as rps_sock_flow_entries. -For a multi-queue device, the rps_flow_cnt for each queue might be -configured as rps_sock_flow_entries / N, where N is the number of -queues. So for instance, if rps_sock_flow_entries is set to 32768 and there -are 16 configured receive queues, rps_flow_cnt for each queue might be -configured as 2048. - - -Accelerated RFS -=============== - -Accelerated RFS is to RFS what RSS is to RPS: a hardware-accelerated load -balancing mechanism that uses soft state to steer flows based on where -the application thread consuming the packets of each flow is running. -Accelerated RFS should perform better than RFS since packets are sent -directly to a CPU local to the thread consuming the data. The target CPU -will either be the same CPU where the application runs, or at least a CPU -which is local to the application thread’s CPU in the cache hierarchy. - -To enable accelerated RFS, the networking stack calls the -ndo_rx_flow_steer driver function to communicate the desired hardware -queue for packets matching a particular flow. The network stack -automatically calls this function every time a flow entry in -rps_dev_flow_table is updated. The driver in turn uses a device specific -method to program the NIC to steer the packets. - -The hardware queue for a flow is derived from the CPU recorded in -rps_dev_flow_table. The stack consults a CPU to hardware queue map which -is maintained by the NIC driver. This is an auto-generated reverse map of -the IRQ affinity table shown by /proc/interrupts. Drivers can use -functions in the cpu_rmap (“CPU affinity reverse map”) kernel library -to populate the map. For each CPU, the corresponding queue in the map is -set to be one whose processing CPU is closest in cache locality. - -==== Accelerated RFS Configuration - -Accelerated RFS is only available if the kernel is compiled with -CONFIG_RFS_ACCEL and support is provided by the NIC device and driver. -It also requires that ntuple filtering is enabled via ethtool. The map -of CPU to queues is automatically deduced from the IRQ affinities -configured for each receive queue by the driver, so no additional -configuration should be necessary. - -== Suggested Configuration - -This technique should be enabled whenever one wants to use RFS and the -NIC supports hardware acceleration. - -XPS: Transmit Packet Steering -============================= - -Transmit Packet Steering is a mechanism for intelligently selecting -which transmit queue to use when transmitting a packet on a multi-queue -device. This can be accomplished by recording two kinds of maps, either -a mapping of CPU to hardware queue(s) or a mapping of receive queue(s) -to hardware transmit queue(s). - -1. XPS using CPUs map - -The goal of this mapping is usually to assign queues -exclusively to a subset of CPUs, where the transmit completions for -these queues are processed on a CPU within this set. This choice -provides two benefits. First, contention on the device queue lock is -significantly reduced since fewer CPUs contend for the same queue -(contention can be eliminated completely if each CPU has its own -transmit queue). Secondly, cache miss rate on transmit completion is -reduced, in particular for data cache lines that hold the sk_buff -structures. - -2. XPS using receive queues map - -This mapping is used to pick transmit queue based on the receive -queue(s) map configuration set by the administrator. A set of receive -queues can be mapped to a set of transmit queues (many:many), although -the common use case is a 1:1 mapping. This will enable sending packets -on the same queue associations for transmit and receive. This is useful for -busy polling multi-threaded workloads where there are challenges in -associating a given CPU to a given application thread. The application -threads are not pinned to CPUs and each thread handles packets -received on a single queue. The receive queue number is cached in the -socket for the connection. In this model, sending the packets on the same -transmit queue corresponding to the associated receive queue has benefits -in keeping the CPU overhead low. Transmit completion work is locked into -the same queue-association that a given application is polling on. This -avoids the overhead of triggering an interrupt on another CPU. When the -application cleans up the packets during the busy poll, transmit completion -may be processed along with it in the same thread context and so result in -reduced latency. - -XPS is configured per transmit queue by setting a bitmap of -CPUs/receive-queues that may use that queue to transmit. The reverse -mapping, from CPUs to transmit queues or from receive-queues to transmit -queues, is computed and maintained for each network device. When -transmitting the first packet in a flow, the function get_xps_queue() is -called to select a queue. This function uses the ID of the receive queue -for the socket connection for a match in the receive queue-to-transmit queue -lookup table. Alternatively, this function can also use the ID of the -running CPU as a key into the CPU-to-queue lookup table. If the -ID matches a single queue, that is used for transmission. If multiple -queues match, one is selected by using the flow hash to compute an index -into the set. When selecting the transmit queue based on receive queue(s) -map, the transmit device is not validated against the receive device as it -requires expensive lookup operation in the datapath. - -The queue chosen for transmitting a particular flow is saved in the -corresponding socket structure for the flow (e.g. a TCP connection). -This transmit queue is used for subsequent packets sent on the flow to -prevent out of order (ooo) packets. The choice also amortizes the cost -of calling get_xps_queues() over all packets in the flow. To avoid -ooo packets, the queue for a flow can subsequently only be changed if -skb->ooo_okay is set for a packet in the flow. This flag indicates that -there are no outstanding packets in the flow, so the transmit queue can -change without the risk of generating out of order packets. The -transport layer is responsible for setting ooo_okay appropriately. TCP, -for instance, sets the flag when all data for a connection has been -acknowledged. - -==== XPS Configuration - -XPS is only available if the kconfig symbol CONFIG_XPS is enabled (on by -default for SMP). The functionality remains disabled until explicitly -configured. To enable XPS, the bitmap of CPUs/receive-queues that may -use a transmit queue is configured using the sysfs file entry: - -For selection based on CPUs map: -/sys/class/net//queues/tx-/xps_cpus - -For selection based on receive-queues map: -/sys/class/net//queues/tx-/xps_rxqs - -== Suggested Configuration - -For a network device with a single transmission queue, XPS configuration -has no effect, since there is no choice in this case. In a multi-queue -system, XPS is preferably configured so that each CPU maps onto one queue. -If there are as many queues as there are CPUs in the system, then each -queue can also map onto one CPU, resulting in exclusive pairings that -experience no contention. If there are fewer queues than CPUs, then the -best CPUs to share a given queue are probably those that share the cache -with the CPU that processes transmit completions for that queue -(transmit interrupts). - -For transmit queue selection based on receive queue(s), XPS has to be -explicitly configured mapping receive-queue(s) to transmit queue(s). If the -user configuration for receive-queue map does not apply, then the transmit -queue is selected based on the CPUs map. - -Per TX Queue rate limitation: -============================= - -These are rate-limitation mechanisms implemented by HW, where currently -a max-rate attribute is supported, by setting a Mbps value to - -/sys/class/net//queues/tx-/tx_maxrate - -A value of zero means disabled, and this is the default. - -Further Information -=================== -RPS and RFS were introduced in kernel 2.6.35. XPS was incorporated into -2.6.38. Original patches were submitted by Tom Herbert -(therbert@google.com) - -Accelerated RFS was introduced in 2.6.35. Original patches were -submitted by Ben Hutchings (bwh@kernel.org) - -Authors: -Tom Herbert (therbert@google.com) -Willem de Bruijn (willemb@google.com) -- cgit v1.2.3-58-ga151