Preserve Client IP for NSX-T Overlay

Overview

Load balancing solutions for some applications (particularly in Layer 4 mode) require Client IP address to be presented as the source IP address when the packet lands on the backend pool server member. NSX Advanced Load Balancer (formerly known as Avi Vantage) supports this functionality as Preserve Client IP.

This solution is deployed generally by making the default gateway of the backend server to point to a floating IP. The floating IP is hosted on the active Service Engine’s backend interface. However, in this mode, the servers’ default gateway has to be modified/updated to point to the floating IP. The NSX-T overlay deployment model being in layer 3 mode has complications for preserving the client IP.

Service Insertion Framework

VMware NSX-T provides the service insertion, framework which has the ability to redirect traffic. The NSX-T service insertion framework has been utilized to achieve the return traffic from backend server to the floating IP of Active Service Engine without needing to change the default gateway of the backend server.

Note: Starting with NSX Advanced Load Balancer version 22.1.6, Service Insertion with IPv6 is supported (Tech Preview).

Preserve Client IP

Notes:

  • Preserve client IP/Service Insertion will not work with port translation disabled (From the NSX Advanced Load Balancer UI, navigate to Pool > Advanced Setting > Disable Port Translation option) on NSX-T overlay, as redirect rule to attract return traffic from server is based on configured server ip: server port.
  • Since the Service Insertion rule is based on server ip and server port to redirect the return traffic from server to SE. So if there are multiple ports on front-end and there is no matching server port entry in pool then redirect rule will not work if we did not translate the port as the rules are specific to server ip:server port only.
  • Set Use_service_port to false (i.e. enable port translation) if Preserve client IP/Service Insertion is used with NSX-T overlay.

Preserve client IP for NSX-T overlay deployments with respect to configuration at virtual service, Service Engine group, and network service will remain the same as per other supported clouds. There are additional pre-requisites and limitations which are discussed in this article.

Pre-requisites

  1. The Service Engine HA mode should be Legacy HA mode (Active/Standby).

    Preserve Client IP

  2. The NSX-T user for configuring NSX-T cloud should have additional permissions of Netx Partner Admin and Security Admin for the preserve client IP functionality apart from the Network Admin requirement for other use cases.

    Preserve Client IP

  3. Set URPF Mode to None for the VIP data segments in which the preserve client IP feature will be enabled.

    Preserve Client IP

    Note: If uRPF Mode is not set to None, then an event with error will be generated, though the status of the virtual service is up, as shown below:

    Preserve Client IP

  4. Configure the virtual service (for which preserve client IP has to configured), and pool server as Network Security Groups. Individual IP address or range, DNS Name, and IP group are not allowed.

Configuring Preserve Client IP (IPv4)

Configure the Floating Interface IP (FIP) in the network service and attach it to the appropriate Avi Service Engine group, VRF, and cloud reference which is hosting the virtual service that require the preserve client IP feature.


    [admin:10-170-67-140]: > show nsxt segment London_ALB_DATA_SEGMENT
    +-------------------+-----------------------------------------+
    | Field             | Value                                   |
    +-------------------+-----------------------------------------+
    | uuid              | segmentruntime-ab75a213243b             |
    | segment_id        | /infra/segments/London_ALB_DATA_SEGMENT |
    | name              | London_ALB_DATA_SEGMENT                 |
    | subnet            | 192.168.100.0/24                        |
    | dhcp_enabled      | True                                    |
    | nw_ref            | London_ALB_DATA_SEGMENT                 |
    | nw_name           | London_ALB_DATA_SEGMENT                 |
    | vrf_context_ref   | London_Tier1Gateway1                    |
    | tier1_id          | /infra/tier-1s/London_Tier1Gateway1     |
    | opaque_network_id | 9cbf6823-3bb8-4935-a675-e07872e7935f    |
    | segment_gw        | 192.168.100.1/24                        |
    | dhcp_ranges[1]    | 192.168.100.170-192.168.100.180         |
    | segname           | London_ALB_DATA_SEGMENT                 |
    | tenant_ref        | admin                                   |
    | cloud_ref         | nsxt_cloud_overlay                      |
    +-------------------+-----------------------------------------+

From the configuration, note that the DHCP range is 192.168.100.170-192.168.100.180 Preserve Client IP

The preserve client IP is configured as shown below:


 [admin:10-170-67-140]: > show networkservice nsxt_preserveIP_ns
 +--------------------------------+-----------------------------------------------------+
 | Field                          | Value                                               |
 +--------------------------------+-----------------------------------------------------+
 | uuid                           | networkservice-55e0f033-02e1-4a6b-99a1-b3a0f674f380 |
 | name                           | nsxt_preserveIP_ns                                  |
 | se_group_ref                   | Default-Group                                       |
 | vrf_ref                        | London_Tier1Gateway1                                |
 | service_type                   | ROUTING_SERVICE                                     |
 | routing_service                |                                                     |
 |   enable_routing               | False                                               |
 |   routing_by_linux_ipstack     | False                                               |
 |   floating_intf_ip[1]          | 192.168.100.150                                     |
 |   enable_vmac                  | False                                               |
 |   enable_vip_on_all_interfaces | True                                                |
 |   advertise_backend_networks   | False                                               |
 |   graceful_restart             | False                                               |
 |   enable_auto_gateway          | False                                               |
 | tenant_ref                     | admin                                               |
 | cloud_ref                      | nsxt_cloud_overlay                                  |
 +--------------------------------+-----------------------------------------------------+

The floating IP is 192.168.100.150, which is outside the DHCP range:

Preserve Client IP

Notes:

  • Ensure that the FIP is from the same segment where the Service Engine’s data segment is configured.
  • Ensure that the FIP does not fall in the DHCP/static range of the data segment.

Configuring Preserve Client IP (IPv4, L7 Virtual Service)

To configure preserve client IP for the L7 virtual service,

  1. In the HTTP profile, enable Preserve Client IP as shown below:

    Preserve Client IP

    Note: Connection Multiplex cannot be enabled while using Preserve Client IP.

  2. Bind the Application Profile to the L7 virtual service as shown below:

    Preserve Client IP

Configuring Preserve Client IP (IPv4, L4 Virtual Service)

To configure preserve client IP for the L4 virtual service,

  1. In the L4 Application Profile, enable Preserve Client IP as shown below:

    Preserve Client IP

  2. Bind the L4 Application Profile to the L7 virtual service as shown below:

    Preserve Client IP

Configuring Tier-1 Locale Service

To configure a non-default customized Tier-1 locale-service for hosting the redirect service, configure the field locale_service.

Note: This option is available in the CLI (Configure cloud > nsxt_configuration > data_network_config > tier1_segment_config > manual > tier1_lrs).


[admin:10-170-67-140]: cloud:nsxt_configuration:data_network_config:tier1_segment_config:manual:tier1_lrs>
[admin:10-170-67-140]: cloud:nsxt_configuration:data_network_config:tier1_segment_config:manual:tier1_lrs> locale_service /infra/tier-1s/London_Tier1Gateway1/locale-services/London_Tier1LocalServices-1
[admin:10-170-67-140]: cloud:nsxt_configuration:data_network_config:tier1_segment_config:manual:tier1_lrs> where
Tenant: admin
Cloud: Default-Cloud
+----------------+---------------------------------------------------------------------------------+
| Field          | Value                                                                           |
+----------------+---------------------------------------------------------------------------------+
| tier1_lr_id    | London_Tier1Gateway1                                                            |
| segment_id     | /infra/segments/London_ALB_DATA_SEGMENT                                         |
| locale_service | /infra/tier-1s/London_Tier1Gateway1/locale-services/London_Tier1LocalServices-1 |
+----------------+---------------------------------------------------------------------------------+
[admin:10-170-67-140]: cloud:nsxt_configuration:data_network_config:tier1_segment_config:manual:tier1_lrs>

Preserve Client IP

Configuring Preserve Client IP (IPv6)

The implementation of preserve client IPv6 addresses is similar to that of IPv4. It involves applying the Application Profile configured with the Preserve Client IP Address option for both Layer 4 or Layer 7 Virtual Services.
This new implementation for IPv6 introduces the FIP IPv6 capability in the Network Service configuration using the field floating_intf_ip6_addresses.

Preserve Client IP

Configuring Preserve Client IP (IPv6, L4 and L7 Virtual Services)

  1. Create a Network Service with a Floating IPv6 address, along with appropriate configurations for Service Engine group (se_group_ref), VRF (vrf_ref), and the Cloud (cloud_ref) of the Virtual Service requiring the preserving of the client’s IP address.

  2. From Templates > Profiles > Application. In the Application profile,
    • In case of a HTTP profile, deactivate Connection Multiplex and enable Preserve Client IP as shown below: Preserve Client IP
    • In case of an L4 profile, enable Preserve Client IP as shown below: Preserve Client IP
  3. Navigate to Infrastructure > Cloud Resources > Service Engine Group. Edit the Service Engine group for which the virtual service needs the Preserve Client IP functionality. Disable Enable Service Engine Self-Election as shown below: Preserve Client IP
  4. Navigate to Applications > Virtual Services and create a virtual service with IPv6 VIP and IPv6 Pool NS-group configuration. Preserve Client IP
    Preserve Client IP

Caveats for Preserve Client IP (IPv6)

  • Disabling the option Enable Service Engine Self-Election on the SE group configuration can cause health monitor failures for the pool members if the pools are not configured correctly.
  • In scale environments, there might be delay on the CRUD (Create, Read, Update, Delete) operations of the virtual service like Disable virtual service, Delete virtual service, enable the virtual service after disabling it) since this involves updating the underlying network configurations like Static Address and Service Insertion rules.

Notes:

  • Preserve Client IP (IPv6) is supported with NSX version 4.1.2.1.1.
  • Preserve Client IPv6 is only supported for end-to-end IPv6 (i.e., VIP, Pool members, FIP should be all IPv6). Mixed mode of v4 and v6 is not supported.
  • Service Engine Data NIC and Pool member network segment should belong to the same NSX Tier-1.
  • NSX API rate limit might get hit in scale environments, it is recommended to increase or set appropriate limit as per the NSX API Guide.

Convergence Metrics

  • Switching the Virtual Service Application Profile to a Preserve Client IP-based Profile, causes traffic loss of around 120 seconds.
  • After changing the NS group, it takes approximately 120 seconds for traffic redirection to occur and route to the new pool servers.
  • After VS disable, it takes around 180 seconds to delete the Service Insertion rule.
  • After VS enable following the disable, it takes around 5 min for VIP to be Operational and in OPER_UP state.

Caveats For Preserve Client IP (IPv4 and IPv6)

  • Only Inline Topologies are supported for preserving the client IP. One-Arm Topology is not supported, similar to legacy NSX-T load balancer
  • If the same application NSG is used in more than one virtual services (across SE group/across the cloud), each pool must have a different service port.

  • The Tier-1, hosting the VIP data segment,should have the NSX Edge cluster selected as redirect service framework requires Tier-1’s Service Router component

    Note: Ensure that the sizing of the Edge cluster considers the traffic needs of the preserve client IP-enabled virtual service.

  • Updating the redirect rule (change of port number of the pool or changing the FIP IP in the SE group) will cause traffic loss for around 90 seconds
  • Using the same pool server and port for preserve client IP and non-preserve client IP virtual service across SE groups causes the non-preserve client IP virtual service to be marked down due to health monitor traffic failing
  • Distributing load & auto-redistributing load properties of SE-group are not supported.
  • Preserve client IP does not work With SNAT rule for the pool server applied on the same Tier-1 Gateway.

Document Revision History

Date Change Summary
February 20, 2024 Updated the support for IPv6 in Tech Preview (version 22.1.6)
April 11, 2022 Published the feature KB for for Preserve Client IP in NSX-T Overlay