TSO GRO RSS Configuration Guide
Overview
This guide explains the TSO, GRO and RSS configuration process.
Enabling GRO and TSO on SE
Upgrading from a prior version will carry forward the GRO configuration for a SE group. However, if a SE group is newly created in NSX Advanced Load Balancer version 22.1.2 , GRO config is in auto-mode. If the SE group has SEs with greater than or equal to 8 vCPUs, GRO will be enabled.
Notes:
-
Enabling TSO/ GRO is non-disruptive, and it does not require an SE restart.
-
If a PUT API call is made to update existing
ServiceEngineGroupproperties with no explicit value set for GRO config knob ("disable_gro": true); then GRO will be reset/ enabled by default starting with NSX Advanced Load Balancer version 22.1.1. This behavior is not noticed when SE group properties is updated through CLI.
You can enable this as follows:
-
Login to the NSX Advanced Load Balancer CLI and use the configure
serviceenginegroupcommand to enable TSO and GRO features.[admin:cntrl]: > configure serviceenginegroup Default-Group Updating an existing object. Currently, the object is: | disable_gro | True | | disable_tso | True | [admin:cntrl]: serviceenginegroup> no disable_gro Overwriting the previously entered value for disable_gro [admin:cntrl]: serviceenginegroup> no disable_tso Overwriting the previously entered value for disable_tso [admin:cntrl]: serviceenginegroup> save | disable_gro | False | | disable_tso | False | -
To verify if the features have been correctly turned ON in the SE, you can check the following statistics on the Controller CLI:
a. GRO statistics are part of interface statistics. For GRO, check the statistics for the following parameters:
-
gro_mbufs_coalesced -
gro_mbufs_coalesced
b. TSO statistics are part of mbuf statistics. For TSO, check the statistics for the following parameters:
-
num_tso_bytes -
num_tso_chain
-
-
Execute the
show serviceengine <interface IP address>interface command and filter the output usinggrepcommand shown as follows:
[admin:cntrl]: > show serviceengine 10.1.1.1 interface | grep gro
| gro_mbufs_coalesced | 1157967 |
| gro_mbufs_coalesced | 1157967 |
Note: The sample output mentioned above is for 1-queue (No RSS).
Refer to the output mentioned below for RSS-enabled, a 4-queue RSS.
Note: In case of a port-channel interface, provide the relevant physical interface name as the filter in the intfname option. For reference, refer to the output mentioned below for the Ethernet 4 interface.
show serviceengine 10.1.1.1 interface filter intfname eth4 | grep gro
| gro_mbufs_coalesced | 320370 |
| gro_mbufs_coalesced | 283307 |
| gro_mbufs_coalesced | 343143 |
| gro_mbufs_coalesced | 217442 |
| gro_mbufs_coalesced | 1164262 |
Note: The statistics for a NIC is the sum of the statistics for each queue for the specific interface.
[admin:cntrl]: > show serviceengine 10.1.1.1 mbufstats | grep tso
| num_tso_bytes | 4262518516 |
| num_tso_chains | 959426 |
If the features are enabled, the statistics in the output mentioned above will reflect non-zero values for TSO parameters.
Enabling RSS on an SE
The distribute_queues knob, in the SE-group properties enable and disable the RSS. Login to the CLI, and use distribute_queues command to enable the RSS feature.
Note: Any change in the distribute_queues parameters requires an SE restart.
The distribute_queues knob in the SE-group properties enable and disable RSS. Login to the CLI and use distribute_queues command to enable the RSS feature.
Note: Any change in the distribute_queues parameters requires an SE restart.
| distribute_queues | False |
[admin:cntrl]: serviceenginegroup> distribute_queues
Overwriting the previously entered value for distribute_queues
[admin:cntrl]: serviceenginegroup> save
| distribute_queues | True |
When RSS is turned ON, all the NICs in the SE configure and use an optimum number of queue pairs as calculated by the SE. The calculation of this optimum number is described in the section on configurable dispatchers.
For instance, the output of a 4-queue RSS-supported interface will be as follows:
[admin:cntrl]: > show serviceengine 10.1.1.1 interface filter intfname bond1 | grep ifq
| ifq_stats[1] |
| ifq_stats[2] |
| ifq_stats[3] |
| ifq_stats[4] |
For a 4-queue RSS, the output will be as shown below.
The value of counters for ipackets (input packets) and opackets (output packets) per interface queue will be a non-zero value as shown below:
[admin:cntrl]: > show serviceengine 10.1.1.1 interface filter intfname bond1 | grep pack
| ipackets | 40424864 |
| opackets | 42002516 |
| ipackets | 10108559 |
| opackets | 11017612 |
| ipackets | 10191660 |
| opackets | 10503881 |
| ipackets | 9873611 |
| opackets | 10272103 |
| ipackets | 10251034 |
| opackets | 10208920 |
Statistics of each queue and one combined statistics overall for the NIC.
Configuration Samples
The example mentioned below exhibits the configuration on a bare-metal machine with 24 vCPUs, two 10G NICs, and one bonded of two 10G NICs, and distribute_queues enabled.
- Set the value of the
configure num_dispatcher_coresparameter to 8.
[admin:cntrl]: serviceenginegroup> num_dispatcher_cores 8
Overwriting the previously entered value for num_dispatcher_cores
[admin-ctrlr]: serviceenginegroup> save
[admin:cntrl]:> show serviceengine 10.1.1.1 seagent | grep -E "dispatcher|queues"
|num_dispatcher_cpu | 8
|num_queues | 8 - Set the value of the
configure num_dispatcher_coresparameter to 0 (the default value).
After restarting the SE, though the configured value for dispatchers is set to 0, the number of queues, and hence the number of dispatchers is changed to 4 as shown below.
[admin:cntrl]:> show serviceengine 10.1.1.1 seagent | grep -E "dispatcher|queues"
|num_dispatcher_cpu | 4
|num_queues | 4 Configuring Maximum Queues per vNIC
The max_queues_per_vnic parameter supports the following values:
-
Zero (Reserved) — Auto (deduces optimal number of queues per dispatcher based on the NIC and operating environment)
-
One (Reserved) — One Queue per NIC (Default)
-
Integer Value — Power of 2; maximum limit is 16.
The max_queues_per_vnic parameter deprecates the distribute_queues parameter which was used to enable RSS mode of operation; wherein, the number of queues were equal to the number of dispatchers.
The migration routine ensures that the max_queues_per_vnic parameter is set to num_dispatcher_cores if the distribute_queues is enabled, else max_queues_per_vnic will be set to 1.
You can use the following command to configure max_queues_per_vnic:
[admin:admin-controller-1]: serviceenginegroup> max_queues_per_vnic
INTEGER 0,1,2,4,8,16 Maximum number of queues per vnic Setting to '0' utilises all queues that are distributed across dispatcher cores.
[admin:admin-controller-1]: > configure serviceenginegroup Default-Group
Updating an existing object. Currently, the object is:
+-----------------------------------------+---------------------------------+
| Field | Value |
+-----------------------------------------+---------------------------------+
[output truncated]
| se_rum_sampling_nav_percent | 1 |
| se_rum_sampling_res_percent | 100 |
| se_rum_sampling_nav_interval | 1 sec |
| se_rum_sampling_res_interval | 2 sec |
| se_kni_burst_factor | 2 |
| max_queues_per_vnic | 1 |
| core_shm_app_learning | False |
| core_shm_app_cache | False |
| pcap_tx_mode | PCAP_TX_AUTO |
+-----------------------------------------+---------------------------------+
[admin:admin-controller-1]: serviceenginegroup> max_queues_per_vnic 2
Overwriting the previously entered value for max_queues_per_vnic
[admin:admin-controller-1]: serviceenginegroup> save
The show serviceegine [se] seagent displays the number of queues per dispatcher and total number of queues per interface.
show serviceengine [se] seagent
| num_dp_heartbeat_miss | 0 |
| se_registration_count | 2 |
| se_registration_fail_count | 0 |
| num_dispatcher_cpu | 1 |
| -------------------------- truncated output-------------------------------|
| num_flow_cpu | 1 |
| num_queues | 1 |
| num_queues_per_dispatcher | 1 |
Configuring Hybrid RSS
The hybrid_rss_mode is a SE-Group configurable property. Toggles SE hybrid only mode of operation in DPDK mode with RSS configured; where-in each SE data path instance operates as an independent standalone hybrid instance performing both dispatcher and proxy function. This requires reboot.
The following is the configuration command:
configure serviceenginegroup <se-group> > hybrid_rss_mode
[admin:10-102-66-36]: serviceenginegroup> max_queues_per_vnic 0
Overwriting the previously entered value for max_queues_per_vnic
[admin:10-102-66-36]: serviceenginegroup>> hybrid_rss_mode
Overwriting the previously entered value for hybrid_rss_mode
[admin:10-102-66-36]: serviceenginegroup> save
Note: The hybrid_rss_mode is protected by a must check that requires RSS to be enabled before toggling this property to True.
-
The property also shows up per SE. The following is the configuration command:
----------------------------------------+ | num_dispatcher_cpu | 4 | | num_flow_cpu | 4 | | num_queues | 4 | | num_queues_per_dispatcher | 1 | | hybrid_rss_mode | True | +---------------------------------------+
Configuring Auto RSS for Public Clouds
This features enables you to configure RSS for NSX Advanced Load Balancer Service Engines deployed on public clouds, such as, AWS, Azure, GCP.
You can set the SE group properties max_queues_per_vnic and num_dispatcher_cores to zero. The SE requires a reboot post configuration.
[admin:controller]: > configure serviceenginegroup Default-Group
[admin:controller]: serviceenginegroup> max_queues_per_vnic 0
[admin:controller]: serviceenginegroup> num_dispatcher_cores 0
[admin:controller]: serviceenginegroup> save
The default value of RSS knobs are as follows:
max_queues_per_vnic = 1
num_dispacther_core = 0
Configuring Dedicated Dispatcher for Public Clouds
The Dedicated Dispatcher (DD) is a runtime SE-Group configurable property which can be set/ unset by toggling dedicated_dispatcher_core knob.
[admin:controller]: > configure serviceenginegroup Default-Group
[admin:controller]: serviceenginegroup> dedicated_dispatcher_core
[admin:controller]: serviceenginegroup> save
[admin:controller]: > configure serviceenginegroup Default-Group
[admin:controller]: serviceenginegroup> no dedicated_dispatcher_core
[admin:controller]: serviceenginegroup> save