Upgrades in an Avi GSLB Environment

This article explains how infrastructure administrators and application owners may approach their respective upgrade scenarios in an Avi Vantage deployment in which GSLB has been configured. Some circumstances may mandate an immediate cessation of service; others may permit a graceful upgrade, with no interruption of service. Avi Vantage’s GSLB implementation supports graceful upgrades, even when challenged by actions such as:

  1. A new version of Avi Vantage — and potentially the GSLB code itself — needs to be installed on active and passive GSLB sites.
  2. A site-wide change must be made.
  3. An application server (or servers) must be replaced or its (their) configuration changed.
  4. A new version of a global application’s executable needs to be installed on all site servers currently running it.

Avi Vantage Upgrades

Notes:

  1. To avoid service disruption due to site upgrade, it is recommended that the DNS virtual service be scaled out to at least two Service Engines within the SE group.
  2. For general information on the Avi Vantage upgrade process, refer to How to Upgrade Avi Vantage Software.

Upgrade Steps

Step 1: Pre-upgrade

On the GSLB leader: Enable GSLB maintenance mode via the REST API or the following CLI command:

gslb maintenancemode enabled

As a result, Avi Vantage takes the following steps:

  • GSLB configuration changes are blocked. We do not want any GSLB configuration changes applied at the leader site while follower sites are getting upgraded.
  • Health-monitor probe frequency is reduced by changing the send_interval to 30 minutes.
  • The cached states of remote sites are not flushed for time T, according to the calculation:

    T = gslb_cfg.send_interval x gslb_cfg.clear_on_max_retries

    The remote site is not declared down during this interval.

Step 2: Upgrade the Site

Upgrade the site to the new version of Avi Vantage. Typically, depending on configuration, the number of virtual services, etc., this may take anywhere from 15 - 45 minutes for the entire site (Controllers and SEs) to be upgraded. Followers are upgraded first; the leader is the last to be upgraded.

Upgrade status can be checked via the CLI command show upgrade status.

Step 3: Post-upgrade

  • Go to the leader site to disable GSLB maintenance mode via the CLI command gslb no maintenancemode so that the newly upgraded remote site can build its runtime states from the rest of the GSLB ecosystem.
  • On the other hand, if ready to upgrade the next site, you may skip step 3.

Repeat steps 1 through 3 for all followers and finally the leader itself.

Note: After all sites have been upgraded, Ansible modules should be migrated to the latest version.

Error Scenarios

  • Follower Upgrade Failure: If the upgrade of a follower site fails, it will end in a GSLB disruption. The site will rollback to the previous version and come up. On the GSLB leader, disable maintenance mode. After determining the reason for failure and addressing it, resume the upgrade process.

  • Leader Upgrade Failure: On the leader, disable maintenance mode. Determine the reason and address it. In the meantime, no new features can be turned on.

Mixing Avi Vantage Versions

Avi recommends that all sites participating in GSLB run at the same Avi Vantage version and maintenance release.
During upgrade cycles, it may be not be possible or desirable to upgrade all sites to the same software version in a single maintenance window. In this situation, it is supported to mix sites running different versions of Avi Vantage for a period of time with the following important caveats:

  1. The GSLB leader site must be the last to be upgraded. It is explicitly not supported for the leader site to be running a later software version or maintenance release than any of the GSLB follower sites.
  2. GSLB configuration changes can be carried out on the leader during the period that sites are running mixed software versions with the caveat that new GSLB features in the later software version cannot be turned on until all sites have been upgraded.

Other Upgrades (not upgrading Avi Vantage itself)

In this section scenarios are illustrated via the Avi UI. The same operations are possible from the CLI and REST API.

CAVEAT: Upgrade actions described below are GSLB operation actions, not GSLB-configuration-changing options. Therefore, there is no guarantee (for a variety of reasons) that clients will “stop knocking at the door” of member virtual services previously known to them. In-flight connections to a VIP/VS at a site are not affected. This could be potentially addressed by changing the DNS TTL, but some intermediate DNS cache may not honor the TTL; traffic may continue arrive to the erstwhile VIP.

Upgrade a Global App on Specific Servers — The Member Disable-Enable Option

When it is time to upgrade a generally sound global application to fix bugs and/or add features, we use this option. It is graceful and non-disruptive. The fact that multiple sites are already delivering the service suggests that one site’s service can be turned off and upgraded while traditional service from N-1 remaining sites continues. This is the classical rolling upgrade scenario.

Refer to figure 1. On the GSLB leader, on a site-by-site basis, use the GSLB pool editor to uncheck the Enabled option of the relevant pool member (gs11 in the case of figure 1).

Note: We prefer that a member VS be upgraded after it has reached a quiescent state.

Next, make required changes to the application on the servers within that virtual service’s server pool(s), re-launch the application, and then check Enabled back on.

GSLB pool editor Figure 1. Pool member is enabled

Behind the scenes, disabling the GSLB pool member causes Avi Vantage to change the global DNS (the member’s IP address is withdrawn) such that traffic to the member virtual service (gs11 in this case) ceases. Traffic remains distributed across the remaining sites that host the application.

Upgrade a Global App Pervasively — The Global Service Disable-Enable Option

In some cases, a global application needs to be suspended immediately and pervasively, for example, until some bug or security breach has been addressed at all participating sites. In this case, rather than disable a member at a time as described above, we want all members to stop as a set. The global-service-disable option is the option.

Refer to figure 2. On the GSLB leader, navigate to Applications > GSLB Services. Click the box at the left of the application’s row. Then click on the Disable button. Do not click on Delete, as that would completely permanently remove the global service from the system configuration, which is not the goal.

GSLB services list Figure 2. GSLB services list

Behind the scenes, Avi Vantage changes the global DNS such that the FQDN is withdrawn.

Upgrade All Global Apps at a Specific Site — The Site Disable-Enable Option

Imagine a bank of servers housing all of the enterprise’s virtualized global applications. If data center space is tight, upgrading them to next-generation servers demands a fork-lift upgrade. Running these global apps simultaneously on the old bank while also launching them on the new bank is not feasible. Downtime at the site is unavoidable, and during it, client load must served by the remaining GSLB sites. The term “DC drain” is often used to describe the first phase of such an upgrade. The site-disable option is the answer.

Refer to figure 3. On the GSLB leader, navigate to Infrastructure > GSLB > Site Configuration. Check the site(s) on which you want all GSLB apps disabled. Then click the Disable button. Do not click Delete, as doing so would permanently remove the identified site(s) from the GSLB configuration.

GSLB sites list Figure 3. GSLB sites list

Behind the scenes, Avi Vantage changes the global DNS such that the IP addresses of all virtual services at the identified site(s) participating in all global applications are withdrawn. Local applications are not affected. Responses to requests for the FQDN of global applications will contain the IP addresses of virtual services running on N-1 remaining GSLB sites.

Notes:

  1. We do not allow the leader site to be disabled. To overcome this limitation, leadership should be passed over to a site that has already been upgraded. From its vantage point the former leader can then be disabled.
  2. Once a site is re-enabled, Avi Vantage resynchronizes all relevant information to the newly enabled site.