Recommendation for better performance of Avi Service Engines


Depending on the real-life workloads, the Avi Service Engine settings can be tweaked by using configurations at Service Engine Group level.

The following are the some guidelines to follow while planning capacity for Avi Service Engines:

General Guidelines

The following are the general guidelines:

  • CPU and memory reservations are recommended for Avi Service Engine Virtual Machines for consistent and deterministic performance.

  • Use compact mode in Avi Service Engine Group settings for virtual service placements on SEs. This ensures Avi Vantage uses the minimum number of SEs required for virtual service placement. This helps in saving the cost in case of public cloud use cases.

Dispatcher Configurations

The following are the dispatcher configurations:

  • The dedicated_dispatcher is set to False by default, at the SE group level. This configuration is optimal for Service Engines with smaller computer capacities, such as 1 core and 2 cores.

  • Avi Vantage recommends dedicated_dispatcher set to True for Service Engine size greater than 2 cores.

GRO and TSO Configurations

The following are the GRO and TSO configurations:

  • The default settings for GRO is disabled and TSO is enabled. This configuration works fine for most of the workloads.

  • GRO can be enabled whenever there are enough dispatchers (> = 4) and their utilization is low.

Receive Side Scaling (RSS) Configurations

The following are the RSS configurations:

  • You can enable RSS (Receive Side Scaling) for better performances. RSS can realize better PPS with more dispatchers and queues per NIC.

  • The number of dispatchers can only be set in the power of 2, i.e., number of dispatchers can be 1, 2, 4, 8 and so on.

  • Default value of max_queues_per_vnic is 1. Setting this value to 0 will automatically decide the number of queues based on the dispatcher count configured. You can set this value as per the requirements.

  • If the number of queues available per NIC is lesser than the dispatcher, then the number of dispatcher is floored to the number of queues. The recommendation is to have number of dispatcher greater than the number of queues available.

Datapath Isolation

You can enable Service Engine datapath isolation for latency and jitter sensitive applications. The feature creates two independent CPU sets for datapath and control plane SE functions.

Recommendations for different Workloads

The following are the recommendations for different workloads:

  • High PPS load such as high connections per seconds with small file GETs should have more dispatchers to do higher PPS.

  • Workloads with high SSL transactions will be proxy heavy and will benefit from high count of proxy cores.

  • Default settings are recommended for 1 core and 2 core Service Engines.

The following examples will explain the configuration recommendation for a 6 core Service Engine running on vCenter full access cloud.

Example 1 – PPS heavy traffic profile

Let us assume 100 layer 4 virtual services with TCP and doing average of 1000 new TCP connections per second with each connection lasting 3 seconds and downloading a single small file over single GET request.

Considering 18 to 20 packets for each TCP transaction for both frontend and backend, this requirement translates to nearly 1 million packets per second for new TCP connections. Given the volume of packets, Avi Service Engine should be configured with the following configuration:

  • Dedicated dispatcher: True

  • Number of dispatchers: 2

  • Number of proxy cores: 4

  • Number of queues per NIC: 2

Example 2 – SSL throughput and TPS heavy traffic profile

Let us assume multiple SSL applications doing total of 2000 ECC transactions per second and 2 Gbps of SSL throughput of GET.

For above requirements the dispatcher cores will not be busy as the packets per second will not be very high and SSL processing will be consuming proxy cores for doing ECC transactions and throughput. RSS will not help in this use case and recommendations for this work load will be the following:

  • Dedicated dispatcher: True

  • Number of dispatchers: 1

  • Number of proxy cores: 5

  • Number of queues per NIC: 1

Example 3 – HTTP workloads with 50% of IP routing traffic

Multiple L7 applications doing nearly 5 – 6 Gbps with nearly 1.5 Million packets per second with single service engine with 50% of IP routing traffic. Applications running are latency and jitter sensitive.

To achieve the above requirements, Avi Vantage recommends dedicating one of the Service Engine core for non data-path tasks. It can be achieved with the following configuration:

  • Dedicated Dispatcher: True

  • se_dp isolation mode: True

  • Number of non-dp cores: 1

  • Number of dispatcher cores: 2

  • Number of queues per NIC: 2

  • Number of proxy cores: 3

Additional References