Route Aggregation for Google Cloud Platform

This article discusses route aggregation configuration for provisioning a large number of virtual services on an Avi Controller operating in Google Cloud Platform (Cloud Platform).

In this document, Google Cloud Platform, also referred to as Cloud Platform, will be abbreviated as GCP.

Overview

Route aggregation is a method used to minimize the number of route entries required in an IP network. Unlike flat routing, wherein each routing table contains a unique entry for every route, route aggregation consolidates multiple routes into a single route advertisement. The route aggregation feature introduced in Avi Vantage 17.1 aggregates the /32 routes into mask-based routes to reduce the number of route entries in GCP.

For Avi Vantage deployed in GCP, Avi Controller creates a specific /32 route for each virtual service, so that the incoming traffic is rightly forwarded to the specified virtual service. Depending on the network, the number of virtual services can scale to large numbers and with this the route entries would also increase. However, the number of route entries supported in GCP is limited (about 2000). To address this limitation, the routes that belong to a specific subnet are aggregated to create a mask-route, so that the number of route entries in GCP is reduced.

The steps to configure this feature are discussed in the following sections.

Using Route Integration with Avi Vantage in GCP

To enable aggregation of GCP routes corresponding to multiple virtual services, the virtual IP (VIP) allocated to virtual services should fall within the same subnet range. In case of auto-allocation, the VIP gets allocated from a subnet specified in the Usable Networks of IPAM configuration.

The following are a few examples to demonstrate route aggregation feature:

Example 1: For a single Service Engine.

Without route aggregation:

Virtual service Virtual IP Route programmed in GCP
vs1 10.1.1.1 10.1.1.1/32, next hop=SE1
vs2 10.1.1.2 10.1.1.2/32, next hop=SE1
vs3 10.1.1.3 10.1.1.3/32, next hop=SE1
vs4 10.1.1.4 10.1.1.4/32, next hop=SE1

With route aggregation enabled: Note that there is just a single route entry (10.1.1.0/29) in GCP.

Virtual service Virtual IP Route programmed in GCP
vs1 10.1.1.1
10.1.1.0/29, next hop=SE1
vs2 10.1.1.2
vs3 10.1.1.3
vs4 10.1.1.4

As there is a single aggregated route programmed for multiple virtual services, these virtual services need to be part of the same Service Engine group, so that the next hop which is the Service Engine for these virtual services can be the same.

Example 2: Virtual service scaled out to two Service Engines within a SE group.

Without route aggregation:

Virtual service Virtual IP Route programmed in GCP
vs1 10.1.1.1 10.1.1.1/32, next hop=SE1
10.1.1.1/32, next hop=SE2
vs2 10.1.1.2 10.1.1.2/32, next hop=SE1
10.1.1.2/32, next hop=SE2
vs3 10.1.1.3 10.1.1.3/32, next hop=SE1
10.1.1.3/32, next hop=SE2
vs4 10.1.1.4 10.1.1.4/32, next hop=SE1
10.1.1.4/32, next hop=SE2

With route aggregation enabled:

Virtual service Virtual IP Route programmed in GCP
vs1 10.1.1.1
10.1.1.0/29, next hop=SE1
10.1.1.0/29, next hop=SE2
vs2 10.1.1.2
vs3 10.1.1.3
vs4 10.1.1.4

The subnet mask used for aggregating the route should be configured as a part of the SE group using service_ip_subnet configuration option, as explained later. This subnet should be derived from one of the Usable Networks configured in the GCP IPAM, from which the VIP is allocated.

Note: With multiple SEs, it is important that all the virtual services in that SE group are scaled out to the same set of SEs. This can be achieved by setting the HA mode of SE group to active/active. Hence, the route aggregation feature is currently limited only to active/active HA mode of the SE group.

Example 3: Virtual services scaled out to multiple SE groups.

While configuring a large number of virtual services, the limit of virtual services per SE may hit the maximum, based on the size of the Service Engine. In order to overcome this, virtual services will need to be distributed across multiple SE groups, so that the groups of virtual services within each SE group can be aggregated through route aggregation.

On enabling route aggregation and configuring service_ip_subnet for individual SE groups, Avi Controller automatically places the virtual service to one of the SE group, while the VIP for the virtual services gets allocated from within the service_ip_subnet configured on that SE group.

Consider an instance with eight virtual services. Based on the expected throughput and the size of selected SE VM, the maximum number of virtual services per SE needs to be restricted to two. To configure eight virtual services, four SE groups will have to be created. Each SE group will program two aggregated routes (two SEs in each SE group). Thus, four subnets will need to be configured. This needs to be carved out from the larger IPAM subnet.

Calculation Details:

  • Configured GCP IPAM subnet = 10.1.1.0/24
  • Configured maximum number of virtual service per SE = 2
  • Maximum number of virtual services expected = 8
  • Number of SE groups required = 8/2 = 4
  • Number of service_ip_subnets to be configured = 4

The IPAM subnet 10.1.1.0/24 needs to be divided into four subnets, and each should be assigned to one SE group as shown below.

Virtual service SE group service_ip_subnet Virtual IP Route programmed in GCP
vs1 SE Group 1 10.1.1.0/26 10.1.1.1
10.1.1.0/26, next hop = SE1
10.1.1.0/26, next hop = SE2
vs2 SE Group 1 10.1.1.0/26 10.1.1.2
vs3 SE Group 2 10.1.1.64/26 10.1.1.65
10.1.1.64/26, next hop = SE3
10.1.1.64/26, next hop = SE4
vs4 SE Group 2 10.1.1.64/26 10.1.1.66
vs5 SE Group 3 10.1.1.128/26 10.1.1.129
10.1.1.128/26, next hop = SE5
10.1.1.128/26, next hop = SE6
vs6 SE Group 3 10.1.1.128/26 10.1.1.130
vs7 SE Group 4 10.1.1.192/26 10.1.1.193
10.1.1.192/26, next hop = SE7
10.1.1.192/26, next hop = SE8
vs8 SE Group 4 10.1.1.192/26 10.1.1.194

Example 4: Multiple clouds and IPAM subnets.

The figure below depicts a complex configuration setup which would follow the same principle of route aggregation.

behind_the_scenes.jpg

Configuration Steps

This article assumes that the Avi Vantage Controller, SEs, cloud, IPAM, and networks have been set up as per Avi Deployment Guide for Google Cloud Platform. Ensure all steps in the document have been completed before proceeding with the following steps for configuring route aggregation.

Step 1: Enable the match_se_group_subnet field in GCP IPAM.

This field is currently not available on the Avi UI, but can be configured via the CLI.


[admin:10-146-14-2]: > configure ipamdnsproviderprofile route-aggregation-ipam

[admin:10-146-14-2]: ipamdnsproviderprofile> type ipamdns_type_gcp

[admin:10-146-14-2]: ipamdnsproviderprofile> gcp_profile

[admin:10-146-14-2]: ipamdnsproviderprofile:gcp_profile> usable_network_refs route-aggregation

[admin:10-146-14-2]: ipamdnsproviderprofile:gcp_profile> match_se_group_subnet

[admin:10-146-14-2]: ipamdnsproviderprofile> save

+--------------------------+-------------------------------------------------------------+
| Field                    | Value                                                       |
+--------------------------+-------------------------------------------------------------+
| uuid                     | ipamdnsproviderprofile-68281eed-cb88-44bb-9d1c-95e7c0afdc85 |
| name                     | Route-Aggregation-IPAM                                      |
| type                     | IPAMDNS_TYPE_GCP                                            |
| internal_profile         |                                                             |
|   ttl                    | 30 sec                                                      |
|   usable_network_refs[1] | Route-Aggregation                                           |
| gcp_profile              |                                                             |
|   usable_network_refs[1] | Route-Aggregation                                           |
|   match_se_group_subnet  | True                                                        |
| tenant_ref               | admin                                                       |
+--------------------------+-------------------------------------------------------------+
[admin:10-146-14-2]: >


Use the following UI screens to verify the IPAM profile and cloud properties:

Figure_1.png

Figure 1. IPAM creation

Figure_2.png

Figure 1. IPAM reference under cloud

Step 2: Plan the number of SEs and SE groups required.

The minimum number of SE groups required can be calculated as follows:

  • Estimate the total number of virtual services to be provisioned. Call that V.
  • The maximum number of virtual services a single SE can host will be called M. For more information, refer to CPU Memory Allocations.
  • The total number of SE groups will be V/M; let that be represented by N.
  • Divide the IPAM network into N smaller subnets, such that each subnet has a maximum of M IP addresses. The Avi Controller uses these subnets to program the routes in GCP, instead of individual /32 routes corresponding to each VIP and SE(s) on which the VIP is placed.

The following are a few examples for the above mentioned calculation:

Example 1:
(i) The total number of virtual services to be provisioned is 256.
(ii) The maximum number of virtual services that can be hosted on a single SE is 64.
(iii) Therefore, the number of SE groups needing to be created is 256/64 = 4.
(iv) The IPAM network created for virtual service is 10.10.0.0/24, which has 254 IP addresses.
(v) Divide the IPAM network into four smaller subnets of 64 IP addresses each: 10.10.0.0/26, 10.10.0.64/26, 10.10.0.128/26 and 10.10.0.192/26.

Example 2:
(i) The total number of virtual services to be provisioned is 512.
(ii) The maximum number of virtual services that can be hosted on a single SE is 32.
(iii) Therefore, the number of SE groups needing to be created is 512/32 = 16.
(iv) The IPAM network created for virtual service is 10.10.0.0/23, which has 510 IP addresses.
(v) Divide the IPAM network into 16 smaller subnets of 32 IP addresses each: 10.10.0.0/27, 10.10.0.32/27, 10.10.0.64/27 till 10.10.1.224/27.

Step 3: Set up SE groups.

(i) The high availability (HA) mode should be set to active/active. If the route aggregation feature is enabled, then neither N+M buffer nor active/standby mode is supported. Refer to the Service Engine Group article for more information on SE HA.
(ii) Configure service-IP subnets for each SE group. Service-IP subnets will provide IP addresses to the respective virtual services in their respective SE group. For this, configure a service-IP subnet object in the SE group configuration. The service-IP subnet should be set to smaller subnets, calculated as shown in Step 2(iv). This field can only be set via the CLI.

Note: Once the field service-IP subnet is used, virtual service IPs are allocated only from the specified SE group. If the subnet is not set, then you must program a /32 route for the virtual service IPs.

(iii) Configure min_scaleout_per_vs and max_scaleout_per_vs values to the number of SEs in the SE group as determined in Step 2 Example 1 (ii).
(iv) Configure max_vs_per_se value to the maximum number of virtual services that a single SE can host (64 in this case) as determined in Step 2 Example 1 (ii).


[admin:10-146-14-2]: > configure serviceenginegroup gcp-route-aggregation
[admin:10-146-14-2]: serviceenginegroup> ha_mode ha_mode_shared_pair
[admin:10-146-14-2]: serviceenginegroup> max_vs_per_se 64
[admin:10-146-14-2]: serviceenginegroup> min_scaleout_per_vs 2
[admin:10-146-14-2]: serviceenginegroup> max_scaleout_per_vs 2
[admin:10-146-14-2]: serviceenginegroup> service_ip_subnets 10.10.0.0/26
[admin:10-146-14-2]: serviceenginegroup> save
+---------------------------------------+---------------------------------------------------------+
| Field                                 | Value                                                   |
+---------------------------------------+---------------------------------------------------------+
| uuid                                  | serviceenginegroup-fbd33752-996d-48aa-9534-08372ef10095 |
| name                                  | gcp-route-aggregation                                   |
| max_vs_per_se                         | 64                                                      |
| min_scaleout_per_vs                   | 2                                                       |
| max_scaleout_per_vs                   | 2                                                       |
………
| ha_mode                               | HA_MODE_SHARED_PAIR                                     |
……….
| service_ip_subnets[1]                 | 10.10.0.0/27                                            |
………..
+---------------------------------------+---------------------------------------------------------+


Note: All SE groups in a cloud should have similar configuration. Only the service-IP subnets will be different for each SE group.

Figure_3.png

Figure 3. Creating Service Engine groups

Step 4: Add SEs to the SE groups.
The number of SEs in each SE group should be equal to max_scaleout_per_vs value as discussed in Step 3 (iv).

Figure_4.png

Figure 4. Adding the requisite SE to the respective SE group

Figure_5.png

Figure 5. SEs reflecting the respective SE group

Step 5: Create a virtual service.

Create a virtual service using auto-allocation, as explained in Avi Deployment Guide for Google Cloud Platform. Note that all virtual services should use auto-allocation with this feature. The SE group is automatically selected by the Avi Controller to distribute virtual services across different SE groups. Any manual SE-group selection will be overridden by the IPAM, based on the auto-allocated VIP. For more information on virtual service placement, refer to Virtual Service Placement Policy.

Figure_6.png

Figure 6. Creating virtual service using the created IPAM profile

Figure_7.png

Figure 7. Aggregated routes

Figure_8.png

Figure 8. Post virtual service creation: SE group is automatically updated under Advanced Settings

Troubleshooting

  • Scenario 1: Selecting different IPAM profile.

As explained in Step 3 (iv), the service_ip_subnet field should be configured for an SE group that will match the respective IPAM profile. If a different IPAM profile is chosen, the following error will result:

Troubleshooting_3.png


  • Scenario 2: Static IP address pool not configured.

Under network settings, if a static IP address pool has not been configured for an IP subnet, the following error will result.

Troubleshooting_2.png

STroubleshooting_1.png

This can be fixed by choosing a static range, as displayed in earlier examples.

Troubleshooting_4.JPG