Recommendation for better performance of Avi Service Engines
Depending on the real-life workloads, the Avi Service Engine settings can be tweaked by using configurations at Service Engine Group level.
The following are the guidelines to follow while planning capacity for Avi Service Engines:
The following are the general guidelines:
CPU and memory reservations are recommended for Avi Service Engine Virtual Machines for consistent and deterministic performance.
Use compact mode in Avi Service Engine Group settings for virtual service placements on SEs. This ensures Avi Vantage uses the minimum number of SEs required for virtual service placement. This helps in saving the cost in case of public cloud use cases.
The following are the dispatcher configurations:
dedicated_dispatcheris set to False by default, at the SE group level. This configuration is optimal for Service Engines with smaller computer capacities, such as 1 core and 2 cores.
Avi Vantage recommends
dedicated_dispatcherset to True for Service Engine size greater than 2 cores.
GRO and TSO Configurations
The following are the GRO and TSO configurations:
The default settings for GRO is disabled and TSO is enabled. This configuration works fine for most of the workloads.
GRO can be enabled whenever there are enough dispatchers (> = 4) and their utilization is low.
Starting with NSX Advanced Load Balancer version 22.1.2, if the SE group has SEs with greater than or equal to 8 vCPUs, GRO will be enabled.
Starting with NSX Advanced Load Balancer 22.1.3 version, LRO is enabled on
serviceenginegroup by default for supported environments, such as, Vmware/ NSX-T Cloud.
For more details on LRO Configurations, refer TSO LRO/ GRO RSS Configuration Guide guide.
Receive Side Scaling (RSS) Configurations
The following are the RSS configurations:
You can enable RSS (Receive Side Scaling) for better performances. RSS can realize better PPS with more dispatchers and queues per NIC.
The number of dispatchers can only be set in the power of 2, i.e., number of dispatchers can be 1, 2, 4, 8 and so on.
Default value of
max_queues_per_vnicis 1. Setting this value to 0 will automatically decide the number of queues based on the dispatcher count configured. You can set this value as per the requirements.
If the number of queues available per NIC is lesser than the dispatcher, then the number of dispatcher is floored to the number of queues. The recommendation is to have number of dispatcher greater than the number of queues available.
You can enable Service Engine datapath isolation for latency and jitter sensitive applications. The feature creates two independent CPU sets for datapath and control plane SE functions.
Recommendations for Auto RSS on Public Clouds
All NSX Advanced Load Balancer supported public cloud instances can enable auto RSS by setting the appropriate values of
num_dispatcher_cores knob in the SE group. For more details on configuring auto RSS, refer to TSO GRO RSS Configuration Guide.
It is recommended to set dedicated dispatcher via
dedicated_dispatcher_core knob in the SE Group for all instances where RSS is enabled and the number of vCPU of the instance is greater or equal to 8. Dedicated dispatcher is a runtime property and does not require Reboot.
Recommendations for different Workloads
The following are the recommendations for different workloads:
High PPS load such as high connections per seconds with small file GETs should have more dispatchers to do higher PPS.
Workloads with high SSL transactions will be proxy heavy and will benefit from high count of proxy cores.
Default settings are recommended for 1 core and 2 core Service Engines.
The following examples will explain the configuration recommendation for a 6 core Service Engine running on vCenter full access cloud.
Example 1 – PPS heavy traffic profile
Let us assume 100 layer 4 virtual services with TCP and doing average of 1000 new TCP connections per second with each connection lasting 3 seconds and downloading a single small file over single GET request.
Considering 18 to 20 packets for each TCP transaction for both frontend and backend, this requirement translates to nearly 1 million packets per second for new TCP connections. Given the volume of packets, Avi Service Engine should be configured with the following configuration:
Dedicated dispatcher: True
Number of dispatchers: 2
Number of proxy cores: 4
Number of queues per NIC: 2
Example 2 – SSL throughput and TPS heavy traffic profile
Let us assume multiple SSL applications doing total of 2000 ECC transactions per second and 2 Gbps of SSL throughput of GET.
For above requirements the dispatcher cores will not be busy as the packets per second will not be very high and SSL processing will be consuming proxy cores for doing ECC transactions and throughput. RSS will not help in this use case and recommendations for this work load will be the following:
Dedicated dispatcher: True
Number of dispatchers: 1
Number of proxy cores: 5
Number of queues per NIC: 1
Example 3 – HTTP workloads with 50% of IP routing traffic
Multiple L7 applications doing nearly 5 – 6 Gbps with nearly 1.5 Million packets per second with single service engine with 50% of IP routing traffic. Applications running are latency and jitter sensitive.
To achieve the above requirements, Avi Vantage recommends dedicating one of the Service Engine core for non data-path tasks. It can be achieved with the following configuration:
Dedicated Dispatcher: True
se_dpisolation mode: True
Number of non-dp cores: 1
Number of dispatcher cores: 2
Number of queues per NIC: 2
Number of proxy cores: 3