Avi GSLB Site Configuration and Operations

This article describes Avi GSLB GUI- and CLI-based GSLB site configuration, as well as the ability to cope with various failure modes.

Prerequisite Reading

Avi GSLB Sites

Let’s begin by reviewing some some of the content of the Avi GSLB Architecture article.

GSLB sites fall into two broad categories — Avi sites and external sites (external sites run third-party ADCs from vendors such as F5 and Citrix or ADC-less sites hosting a solitary server). This article focuses on Avi sites.

Each Avi site is characterized as either active or a passive. Active sites synchronize the GSLB site configuration among themselves. They also query all sites to ascertain the health of those sites. Passive sites host local virtual services, but do not host any GSLB service configuration. Neither do passive sites monitor the health of other sites.

Active sites are further classified into two types — GSLB leader and followers. Exactly one of the active sites — the one from which the initial GSLB site configuration is performed, is the designated GSLB leader. GSLB configuration changes are permitted only by logging into the leader, which propagates those changes to all accessible followers. The only way to switch the leadership role to a follower is by overriding the configuration of the leader from a follower site. This override can be invoked in the case of site failures or for maintenance.

Centralized analytics are only available from the GSLB leader site. Localized metrics and logs are available for the DNS services hosting the GSLB records.

Active/Passive Site

In the figure above, Santa Clara, Chicago, and NY-1 are active sites, and indicated as such by an orange oval around their Controller icons. Boston, Austin, and NY-2 are passive sites, as indicated by grey ovals. Santa Clara is the GSLB leader, and indicated as such by a black oval on the Controller’s icon; all other active sites are followers. To simplify this and other figures, we’ve used a single Controller icon to depict what in production would be a 3-Controller cluster.

Points to Consider

  • All active sites have full-mesh connectivity between them at all times. This includes connectivity from leader-to-follower, follower-to-follower, etc.
  • All active sites in the group initiate connectivity in the direction of the passive site from the active site.
  • The connections between all sites are persistent.
  • Any connectivity issue between the sites is addressed using retries. The clear_on_max_retries parameter is configured for GSLB; it defines the maximum number of connection retries permitted. If Avi Vantage is not able to connect to the remote site within the configured retry count, the initiating site clears all the cached states and the remote site is declared down. Following this, the initiating site attempts to connect to the remote site on a periodic basis, based on the configured send_interval parameter.

Failure Scenarios

Avi Vantage’s behavior after a failure in a GSLB deployment depends on whether the failure is 

  1. At a leader site or one of the followers, and
  2. Of the entire site or just the Avi Controller.

Follower Site Failures

Note: In our follower-site failure examples we focus on infrastructure deployed in Santa Clara, Chicago and New York.

Full-Site Failure

A full-site failure occurs at the NY-1 follower site, as depicted below.

Follower site failure handling

  1. The leader (in Santa Clara) and Chicago are active sites, and therefore detect the failure.
  2. Administrative changes to the GSLB configuration continue to be possible on the leader, but they will not make it to the NY-1 site. Central analytics continue to be accessible, albeit without any contribution from NY-1.
  3. Both control-plane and data-plane health monitors will mark NY-1's GS members DOWN. [Read more about control- and data-plane health in Avi GSLB Service and Health Monitors.)
  4. DNS service for the GSLB configuration remains operational at the two surviving sites.
  5. Global application service will continue on the surviving sites (Santa Clara, Chicago, and NY-2).
Partial-site Failure

If only the Avi Controller at the NY-1 site fails, Avi Vantage’s SEs continue to serve applications in ”headless” mode.

  1. The leader and Chicago Controllers detect the failure via their control-plane monitors. Central analytics continue to be accessible, albeit without any contribution from NY-1.
  2. Any administrative changes made on the leader do not propagate to the NY-1 site.
  3. Data-plane health monitors running in Santa Clara and Chicago continue to perceive NY-1's members as UP.
  4. DNS service for the GSLB configuration remains operational at all three sites (because it comes from SEs, none of which have failed).
  5. Global application service continues on all four sites (Santa Clara, Chicago, NY-1, and NY-2.
Follower Site Recovery

The following holds true for either full- or partial-site failures.

  1. The leader Controller in Santa Clara detects connectivity to the (newly rebooted) follower Controller at NY-1. The latest GSLB configuration is pushed to it.
  2. Other active sites likewise detect successful connectivity to the NY-1 follower Controller as a result of their control-plane health monitors.
  3. If the data-plane never went down (partial-site failure), no more need be done.
  4. If data-plane monitors for NY-1's GS members had been configured and previously marked NY-1's GS members as DOWN, NY-1's members will be marked UP and traffic to them will resume only after those data-plane monitors once again perceive good health.

Leader Site Failures

Full-Site Failure

A full-site failure occurs at the Santa Clara leader site, as depicted below.

Avi GSLB leader site failure

  1. As they are active sites, both Chicago and NY-1 detect the failure.
  2. No administrative changes to the GSLB configuration can be made. There is no access to central analytics.
  3. Both control-plane and data-plane health monitors mark Santa Clara's GS members DOWN.
  4. DNS service for the GSLB configuration remains operational at the two surviving active sites (Chicago and NY-1).
  5. Global application service continues on the three surviving sites (Chicago, NY-1, and NY-2).
Partial-Site Failure

If only the Avi Controller at the Santa Clara site fails, The site’s SEs continue to serve applications in ”headless” mode.

  1. As they are active sites, both Chicago and NY detect the Controller failure via their control-plane health monitors.
  2. No administrative changes to the GSLB configuration can be made. There is no access to central analytics.
  3. Data-plane health monitors running in Chicago and NY continue to perceive Santa Clara's members as UP.
  4. DNS service for the GSLB configuration remains operational at Santa Clara, Chicago and NY-1.
  5. Global application service will continue on all sites.

Leader Site Change

In neither of the above leader-site failure scenarios is a new leader designated; the re-election process is not automatic. Instead, an optional and manual operation can be initiated at any active follower site, either to restore the ability to lead, or for the sake of site maintenance. Both Chicago and NY-1 qualify as potential leaders if Santa Clara’s Controller is down. From either follower site the steps would be:

  1. A GSLB administrator logs into the follower site's Controller and commands it to become the new leader. Until it becomes the leader, it is the so-called "leader-designate."
  2. A take-over message is propagated to all other Avi sites, apprising them of the change in command.
  3. Should the prior leader (Santa Clara) come back up, it assumes the role of a follower as a result of the take-over message that was queued for transmission while it was down.

Change gslb leader

GSLB followers trust all configuration commands from their leader. Although the command to become a leader might be thought of as a configuration command, it is highly privileged and never sent from another site. Rather, it requires an admin with the appropriate credentials to actually log into the Controller being promoted to the leadership role.

For more details, read How to select a leader site manually.

Network Partitioning

Network partitioning (a kind of ”split brain”) occurs due to failures or outages in the Internet or VPN infrastructure between the sites. In the case of network failure, each site updates the GS member state based on control-plane and data-plane health monitors. Both parts of the network act as independent and exclusive subnetworks.

Network partitioning

Hence, each site responds to DNS queries using only the member virtual services to which it has connectivity. In the above example, DNS queries to Santa Clara would be resolved to steer clients to vip-1, whereas DNS queries to Chicago or NY-1 would steer clients to either vip-3 or vip-6.

Santa Clara remains the leader during the network outage, notwithstanding its inaccessibility to access the other two Avi sites. No new leader is automatically elected on the other network partition, nor is it required that there be a leader.

If and when an active site within such a leaderless partition is promoted to the leadership role, the UUID of that site’s Controller plus the Gslb.view_id parameter that gets generated are shared with other followers within the partition. Should a network repair cause a leader outside the partition to re-attempt GSLB configuration changes on sites within the partition, those attempts will be rejected as a result of the view_id clash. This prevents the confusion that would otherwise arise from followers obeying configuration commands from more than one leader.

Configuration of GSLB Sites

A given Avi Controller participates in the GSLB deployment or not. If not, the first time it is approached to turn on GSLB, it assumes a leadership role for GSLB functionality. Multiple GSLB configurations on a given Controller cluster is not supported.

Note: Avi recommends that all sites participating in GSLB run the same software version and maintenance release. The Controller that assumes the GSLB leader role must not run a later software release or maintenance version than any of its GSLB follower sites. This restriction applies both during the initial configuration of GSLB and during subsequent upgrades of Avi Vantage (i.e., the leader site must be upgraded after all its follower sites have been upgraded).

1. Set up the individual Avi Controller clusters

Create two or more Controller clusters (two in this example), and run through the initial system configuration steps. In the below scenario, the two Controllers will be Santa Clara (10.10.25.10) and Boston (10.160.0.20). Each of the Controller clusters could be a 1-node (test & development) or a 3-node (production) cluster. 

Note: For better audit trails, the recommended practice is to set up an admin user account for GSLB configuration. Create a user named ‘gslb’ and assign it admin roles in all the Controller clusters.

2. Configure a local DNS virtual service on all active sites that host DNS

Configure a local DNS virtual service on all the clusters where the DNS service needs to be hosted, bound to the the local g-dns SE group.

As a best practice, a DNS for GSLB should exclusively allocated its own Service Engine group. That is, place no other virtual services (DNS or other application types) on it.

For each Controller cluster, configure a Service Engine group to host the DNS virtual service (named g-dns in this example). This configuration is done by navigating to Infrastructure > Cloud > Service Engine Group.

Notes:

  • The virtual service and SE group names need not be identical across all GSLB sites. That said, you may find it helpful to embrace a naming convention that is descriptive of the relationship these entities have to each other. 
  • There is no attribute associated with a DNS virtual service object to define it as a “GSLB” DNS. Therefore, if you wish to readily distinguish such a DNS from one that is not part of any GSLB service definition, consider a naming convention that suggests your intention. In the below screenshot, the “g” in “g-dns” suggest GSLB is the intended purpose of the entity.Note: In the below screenshot, the default of 10 virtual services per SE has been left intact. Setting it to 1 would serve to enforce the best practice mentioned above.
    configure local DNS

In Santa Clara (10.10.25.10):

Configure a DNS virtual service on all the clusters where the DNS service needs to be hosted, bound to the g-dns SE group:

Use Advanced setup:

advanced configurations

Configure a DNS virtual service. Be sure to select an application profile of System-DNS. Accept the default for the TCP/UDP Profile field (System-UDP-Per-Pkt). No pool is required; the DNS service will run within the SE’s virtual machine.

advance setup-stepTwo

Network security rules may be established for this DNS VS, but not required.

configure local DNS_stepTwo

Click on Next to proceed to Step 3 Analytics.

Accept the defaults for analytics or change them in the below-depicted Step 3 of the wizard.

configure local DNS_stepThree

Click on Next to proceed to Step 4 Advanced.

Be sure to identify the SE group you’ve created to host this DNS virtual service.

configure local DNS_stepFour

Optionally create static DNS records.

configure local DNS_stepFive

Click on Save to complete the process of defining the DNS virtual service for the Santa Clara site.

On 10.160.0.20 (Boston) repeat the above process to create a DNS VS named colo-dns with VIP = 10.160.110.100.

3. Configure local application virtual services

Create application virtual services normally. For example, create an HTTP virtual service vs-1 in Controller cluster 1, and virtual service vs-2 in Controller cluster 2.

See Configuring Virtual Services for more details: 

On 10.10.25.10 (Santa Clara):

local Application VS-1 Santa Clara

On 10.160.0.20 (Boston):

localApplicationVS-2 Boston

4. Designate the GSLB leader Controller, and add site configuration

Choose one of the Controller clusters as the leader, and perform the GSLB configuration on it. In the sample topology, the Santa Clara site (10.10.25.10) is chosen as the GSLB leader.

  1. Go to Infrastructure -> GSLB
    Screen Shot 2017-01-30 at 3.56.31 PM
  2. Edit and create the GSLB leader site. Note how Avi Vantage correctly assumes that when GSLB is first enabled, the Controller will become an active member. In particular, the leader member.
    Screen Shot 2017-01-30 at 4.11.04 PM

    After successful configuration, the following screen appears. In the Type column, "Owner" should be interpreted as "Leader."GSLB_three
  3. Add the second site by clicking ‘Add New’
  1. Go to Infrastructure -> GSLB
    Screen Shot 2017-01-30 at 3.56.31 PM

  2. Edit and create the GSLB leader site. Note how Avi Vantage correctly assumes that when GSLB is first enabled, the Controller will become an active member. In particular, the leader member.
    Screen Shot 2017-01-30 at 4.11.04 PM
    Under Advanced Settings, you can configure the Client Group IP Address Type and Health Monitor Proxy. A user configured Geo Location Source can be configured, by filling out the relevant fields - Name, Tag, Latitude, and Longitude. Latitude and Longitude are represented as degrees.minutes. The latitude range is from -90.0 (south) to +90.0 (north) and the longitude range is from -180.0 (west) to +180.0 (east). The precision of the entered value is limited to four decimal digits.
    screenshot_advanced
    After successful configuration, the following screen appears: GSLB_three Note: The Type is marked as “Owner”. This should be interpreted as Leader.

  3. Add the second site by clicking Add New Site
  4. The New GSLB Site screen appears. Enter the details as shown below:
    GSLB_four

  5. To indicate the site is active, ensure that the check box Active Member is selected.
  6. In case of active sites that have a DNS virtual service, click Save and Set DNS Virtual Services. In case of passive sites that do not have a DNS virtual service, click Save to save the site configuration.
    Save and Set DNS Virtual Services
    Screen Shot 2017-01-30 at 4.25.01 PM

At this point, the two sites are talking to each other, and configuration synchronization is enabled.

Site Configuration Errors

Errors in site configuration — having to do with IP address, credentials, and the like — show up when the site information is saved. Some sample error screens follow.

Authentication Failure

Remember, the username and password for the admin of Boston may be unique to that site, or the same credentials used at all Avi GSLB sites — it’s up to the user.

Authentication failed

 

Max Retry Login Failure

Appropriately authenticated individuals log into a leader to perform GSLB-related functions, such as to read a GSLB configuration or to make changes to it. In addition, behind the scenes, the leader GSLB site will robotically log into a follower GSLB site to pass on configuration changes that can only be initiated from the leader. In both cases, a login attempt “lockout” rule may be in force, whereby a certain number of failures results in locking out the administrative account for some specified number of minutes (default = 30 minutes).

MaxRetryError

HTTP 400 Error

There are several GSLB contexts in which a “400” error may occur. This particular example illustrates an understandable restriction: An Avi site can participate exactly in one GSLB configuration. Invitations to join a second are rejected.

HTTP 400 error

Validating the GSLB Configuration

In the secondary site (Boston), check the GSLB page, and make sure that the site configuration shows up.

https://10.160.0.20/#/authenticated/administration/gslb
Screen Shot 2017-01-30 at 4.46.23 PM

Enable DNS SE Network Access to All Virtual Services Being Health-Checked

The DNS Service Engine monitors the health of the GSLB service members (application virtual services). Add static routes (or default gateway) to ensure the members are reachable.

local vs

For example, on 10.10.25.10 (Santa Clara):

Santa Clara static route

On 10.160.0.20 (Boston):

Boston settings

CLI-Based Configuration

1. Set up the Controller clusters

Current limitations:

All member Controller clusters have to be set up completely, before starting any GSLB configuration. If GSLB configuration is made, and a new Controller is added, the configuration is not [yet] synced to the new Controller.

2. Designate GSLB leader Controller, and create global configuration

Create GSLB global configuration:

Example: two Controller clusters (10.10.25.10 and 10.160.0.25)

10.10.25.10 is the designated GSLB leader. So, create the configuration on the GSLB leader.

Find the cluster UUIDs of both Controller clusters.

On 10.10.25.10:

: > show cluster

+---------------+----------------------------------------------+
| Field         | Value                                        |
+---------------+----------------------------------------------+
| uuid          | cluster-42301dd3-0529-ada4-ec02-69a2c593df6d |
: > configure gslb glb
: gslb> dns_configs
New object being created

: gslb:dns_configs> domain_name avi.com
: gslb:dns_configs>
: gslb> site_controller_clusters
New object being created

: gslb:site_controller_clusters> ip_addresses 10.10.25.10
: gslb:site_controller_clusters> cluster_uuid cluster-42301dd3-0529-ada4-ec02-69a2c593df6d
: gslb:site_controller_clusters> username admin
: gslb:site_controller_clusters> password admin
: gslb:site_controller_clusters> name SantaClara
: gslb:site_controller_cluster> save
: gslb> site_controller_clusters
New object being created

: gslb:site_controller_clusters> ip_addresses 10.160.0.20
: gslb:site_controller_clusters> cluster_uuid cluster-42215c91-6280-6016-31f6-7416a1f4c4ad
: gslb:site_controller_clusters> username admin
: gslb:site_controller_clusters> password admin
: gslb:site_controller_clusters> name Boston
: gslb:site_controller_clusters> save
: gslb> save
+------------------------------+-----------------------------------------------+
| Field                        | Value                                         |
+------------------------------+-----------------------------------------------+
| uuid                         | gslb-cafe8f98-c411-47cd-96d2-1a6d4e3bad74 |
| name                         | glb                                           |
| dns_configs[1]               |                                               |
|   domain_name                | avi.com                                       |
| site_controller_clusters[1]  |                                               |
|   cluster_ref                | cluster-42301dd3-0529-ada4-ec02-69a2c593df6d  |
|   name                       | SantaClara                                    |
|   ip_addresses[1]            | 10.10.25.10                                   |
|   port                       | 443                                           |
|   username                   | admin                                         |
| site_controller_clusters[2]  |                                               |
|   cluster_ref                | cluster-42215c91-6280-6016-31f6-7416a1f4c4ad  |
|   name                       | Boston                                        |
|   ip_addresses[1]            | 10.160.0.20                                   |
|   port                       | 443                                           |
|   username                   | admin                                         |
| owner_controller_cluster_ref | cluster-42301dd3-0529-ada4-ec02-69a2c593df6d  |
| tenant_ref                   | admin                                         |
+------------------------------+-----------------------------------------------+
: >

Now, the synchronization is set up.

Validation

Go to the secondary site, and try the show command. The GSLB leader and follower UUIDs will match.

: > show gslb
+------+-----------------------------------------------+
| Name | UUID                                          |
+------+-----------------------------------------------+
| glb  | gslb-cafe8f98-c411-47cd-96d2-1a6d4e3bad74 |
+------+-----------------------------------------------+

 

3. Configure local DNS virtual service

Configure a new SE group to host the DNS virtual service (referred to as g-dns SE group), on both Controller clusters.

Configure a DNS virtual service on all the clusters where the DNS service is hosted, bound to the g-dns SE group:

  • Configure domain names hosted by the DNS virtual service (optional)

From the CLI, create an application profile that selects the domain names hosted by this virtual service (on all Controller clusters).

: > configure applicationprofile dns
: applicationprofile> type application_profile_type_dns
: applicationprofile> dns_service_profile
: applicationprofile:dns_service_profile> domain_names avi.com
: applicationprofile:dns_service_profile> save
: applicationprofile> save

From the UI or CLI, create an application profile that selects the domain names hosted by this virtual service.

4. Configure local application virtual services

Create application virtual services normally. For example, create an HTTP virtual service vs-1 in Controller cluster 1, and virtual service vs-2 in Controller cluster 2.

See Configuring Virtual Services for more details.

5. Configure health monitor for GSLB Services

Only on the GSLB leader (Santa Clara / 10.10.25.10):

: > configure gslbhealthmonitor global-http-hm
: gslbhealthmonitor> type health_monitor_http
: gslbhealthmonitor> monitor_port 80
: gslbhealthmonitor> save

6. Configure routes to ensure that DNS virtual service has access to local virtual services

The DNS Service Engine monitors the health of the GSLB service members. Add static routes (or default gateway) to make sure that the members are reachable.

For example, on 10.10.25.10 (Santa Clara):

: > configure vrfcontext global
Updating an existing object. Currently, the object is:

+----------------+-------------------------------------------------+
| Field          | Value                                           |
+----------------+-------------------------------------------------+
| uuid           | vrfcontext-fde3b826-b19c-449c-8dec-ddeb119f2498 |
| name           | global                                          |
| system_default | True                                            |
| tenant_ref     | admin                                           |
| cloud_ref      | Default-Cloud                                   |
+----------------+-------------------------------------------------+
: vrfcontext> static_routes
: vrfcontext:static_routes> prefix 10.0.0.0/8 next_hop 10.90.12.1
: vrfcontext:static_routes> save
: vrfcontext> save

+------------------+-------------------------------------------------+
| Field            | Value                                           |
+------------------+-------------------------------------------------+
| uuid             | vrfcontext-fde3b826-b19c-449c-8dec-ddeb119f2498 |
| name             | global                                          |
| static_routes[1] |                                                 |
|   prefix         | 10.0.0.0/8                                      |
|   next_hop       | 10.90.12.1                                      |
|   route_id       | 1                                               |
| system_default   | True                                            |
| tenant_ref       | admin                                           |
| cloud_ref        | Default-Cloud                                   |
+------------------+-------------------------------------------------+
: >

On 10.160.0.20 (Boston):

7. Configure GSLB services

: > configure gslbservice view
: gslbservice> domain_names view.avi.com
: gslbservice> health_monitor_refs global-http-hm
: gslbservice> num_dns_ip 1
: gslbservice> groups
New object being created

: gslbservice:groups> algorithm gslb_algorithm_round_robin
: gslbservice:groups> name active-sc
: gslbservice:groups> priority 10
: gslbservice:groups> members
New object being created

: gslbservice:groups:members> ip 10.90.12.100
: gslbservice:groups:members> save
: gslbservice:groups> save
: gslbservice> groups
: gslbservice:groups:members> ip 10.160.110.200
: gslbservice:groups:members> save
: gslbservice:groups> save
: gslbservice> save
+----------------------------------+----------------------------------------------------+
| Field                            | Value                                              |
+----------------------------------+----------------------------------------------------+
| uuid                             | gslbservice-3f359566-f534-47d9-a735-10105fa53bfb |
| name                             | view                                               |
| domain_names[1]                  | view.avi.com                                       |
| groups[1]                        |                                                    |
|   name                           | active-sc                                          |
|   priority                       | 10                                                 |
|   algorithm                      | GSLB_ALGORITHM_ROUND_ROBIN                         |
|   members[1]                     |                                                    |
|     ip                           | 10.90.12.100                                       |
|     ratio                        | 1                                                  |
|     enabled                      | True                                               |
| groups[2]                        |                                                    |
|   name                           | standby-boston                                     |
|   priority                       | 7                                                  |
|   algorithm                      | GSLB_ALGORITHM_ROUND_ROBIN                         |
|   members[1]                     |                                                    |
|     ip                           | 10.160.110.200                                     |
|     ratio                        | 1                                                  |
|     enabled                      | True                                               |
| num_dns_ip                       | 1 count                                            |
| health_monitor_refs[1]           | global-http-hm                                     |
| site_persistence_type            | SITE_PERSISTENCE_NONE                              |
| site_persistence_profile_timeout | 5 mins                                             |
| tenant_ref                       | admin                                              |
+----------------------------------+----------------------------------------------------+

8. Configure pass-through server

If there’s an FQDN miss on a DNS virtual service, Avi can pass this request through (load-balance) to one or more back-end DNS servers. To enable this, configure a pool containing these servers, and attach this to the DNS virtual service.

If a domain filter is configured in the application filter of the VS, then the pass-through is performed only for FQDNs falling within this subdomain. All other queries are dropped.

Unsupported queries are also forwarded to the pass-through server.

Configure Domain Names hosted by the DNS virtual service (optional)

From the CLI, create an application profile that selects the domain names hosted by this virtual service (on all Controller clusters).

: > configure applicationprofile dns
: applicationprofile> type application_profile_type_dns
: applicationprofile> dns_service_profile
: applicationprofile:dns_service_profile> domain_names avi.com
: applicationprofile:dns_service_profile> save
: applicationprofile> save

From the UI or CLI, create an application profile that selects the domain names hosted by this virtual service.

9. Configure corporate/external DNS server to delegate subdomain to the Avi DNS service.

Delegate avi.com to the Avi GSLB.

To try this out in the lab, dnsmasq was installed on the clients, and the following entries added:

On client 1:

server=/avi.com/10.10.25.10

server=/avi.com/10.160.110.100

dig pay.avi.com ..

On client 2:

server=/avi.com/10.160.110.100

server=/avi.com/10.10.25.10

10. Troubleshooting

: > show virtualservice colo-dns dnstable
Sub Domains Serviced:
avi.com
+----------------+-----+---------+-------------------------------------------------------------+---------+--------+
| FQDN           | TTL | Num-IPS | IP Addresses                                                | Service | Tenant |
+----------------+-----+---------+-------------------------------------------------------------+---------+--------+
| cloud7.avi.com | 60  | 1       | de:10.40.10.10, de:10.40.10.1, be:10.40.10.3, uk:10.40.10.2 | gs-4    | admin  |
| cloud8.avi.com | 60  | 1       | de:10.40.10.10, de:10.40.10.1, be:10.40.10.3, uk:10.40.10.2 | gs-4    | admin  |
...

: > show virtualservice colo-dns globalserviceruntime filter gs_uuid globalservice-c3a4785b-a722-476b-906a-6869ed7e2cae
: > show virtualservice colo-dns globalserviceinternal filter gs_uuid globalservice-c3a4785b-a722-476b-906a-6869ed7e2cae
: > show virtualservice colo-dns globalservicehmonstat filter gs_uuid globalservice-c3a4785b-a722-476b-906a-6869ed7e2cae

Connection logs provide information about the FQDN and the response IP addresses provided in the DNS query response.

Site Configuration Errors

Errors in site configuration — having to do with IP address, credentials, and the like — show up when the site information is saved. Some sample error screens follow.

Authentication Failure

Remember, the username and password for the admin of Boston may be unique to that site, or the same credentials used at all Avi GSLB sites — it’s up to the user.

Authentication failed

 

Max Retry Login Failure

Appropriately authenticated individuals log into a leader to perform GSLB-related functions, such as to read a GSLB configuration or to make changes to it. In addition, behind the scenes, a leader GSLB site will robotically log into a follower GSLB site to pass on configuration changes that can only be initiated from the leader. In both cases, a login attempt “lockout” rule may be in force, whereby a certain number of failures results in locking out the administrative account for some specified number of minutes (default = 30 minutes).

MaxRetryError

HTTP 400 Error

There are several contexts in which a “400” error may occur in the GSLB context. This particular example illustrates an understandable restriction: An Avi site can participate exactly in one GSLB configuration. Invitations to join a second are rejected.

HTTP 400 error

Error in Overall Member Status

The GSLB Health monitor displays an exception in the form of a red circle with an exclamation point under Overall Member. The State is Unknown and the Reason is Info Unavailable, as shown in the image below:

GSLB_Troubleshooting_Status

There is no record of the GSLB service status being marked Up or Down in the event logs. However, manual checks from the SE DP namespace are successful.

Reason:
The active GSLB site does not have a DNS virtual service set.
Resolution:
In this case, set the DNS Virtual Service in the GSLB site.
Refer to Configure a local DNS virtual service on all active sites that host DNS under Configuration of GSLB Sites

Validating the GSLB Configuration

In the secondary site (Boston), check the GSLB page, and make sure that the site configuration shows up.

https://10.160.0.20/#/authenticated/administration/gslb
Screen Shot 2017-01-30 at 4.46.23 PM

Enable DNS SE Network Access to All Virtual Services Being Health-Checked

The DNS Service Engine monitors the health of the GSLB service members (application virtual services). Add static routes (or default gateway) to ensure the members are reachable.

local vs

For example, on 10.10.25.10 (Santa Clara):

Santa Clara static route

On 10.160.0.20 (Boston):

Boston settings