Unable to Detach GSLB Site from the Unresponsive Leader site

Issue & Symptoms

Unable to detach GSLB follower site from the unresponsive leader site. None of the configuration changes can be done as the configuration changes are done only on the leader site.

Resolution

Manual election of GSLB leader site is also required in a network partitioning scenario. Network partitioning is similar to the “split-brain” condition, which occurs due to failures or outages in the internet or VPN infrastructure between the sites. In case of a network failure, each site updates the GSLB member state based on control-plane and data-plane health monitors. Both parts of the network act as independent and exclusive subnetworks. No new leader is automatically elected on the other network partition. A follower site in a GSLB deployment can be manually changed to the leader site when the previous leader node is down. Follow the steps mentioned below to change the GSLB leader manually.

  • Login to the shell prompt of the follower site, which should become the leader site. Use the show cluster command and note the value of UUID (vm_uuid) from the below output. The value of UUID from the below output is 005056b0333f.

> show cluster
-------------------------------------+

Field	Value
-------------------------------------+

uuid	cluster-005056b0333f
name	cluster-0-1
nodes[1]	 
name	10.10.30.55
ip	10.10.30.55
vm_uuid	005056b0333f
vm_mor	vm-118231
vm_hostname	node1.controller.local
-------------------------------------+

2) Run the gslb changeleader new_leader cluster-UUID command on the follower node, which will be the new leader site. Use the cluster UUID (005056b0333f) as captured from the previous step:


> gslb changeleader new_leader cluster-005056b0333f

Sample output is shown below:


> gslb changeleader new_leader cluster-005056b0333f
--------------------------------------------------------+

Field	Value
--------------------------------------------------------+

new_leader	cluster-005056b0333f
view_id	1525803021
details[1]	Review event-logs for additional information
--------------------------------------------------------+

Note: Once the steps mentioned above are complete, the follower site will become the new leader site. The new leader site would start synchronizing the configuration to the follower site. As the other site (the old leader site) is not reachable, so the new leader site will try max-retries and stop synchronizing information.

Changes on the previous leader site when it comes up

It is recommended to recover the previous leader site so that GSLB set-up is fully functional between the both sites. When the network partition condition is recovered, both the nodes can communicate to each other. To avoid any conflict issues, configuration changes are required to change the previous leader site to become the member (the follower site). The previous leader site should refer to new leader site as the only actual leader site.

Execute the below commands to make the previous leader site as the follower site to avoid any conflict issues:

  • Run the show gslb Default | grep -i view command on the new leader site and note down the view_id as shown below:

> show gslb Default | grep -i view

view_id	1525801573

Also execute the show cluster command on the new leader site and make a note of UUID (005056b0333f).


> show cluster
-------------------------------------+

Field	Value
-------------------------------------+

uuid	cluster-005056b0333f
name	cluster-0-1
nodes[1]	 
name	10.10.30.55
ip	10.10.30.55
vm_uuid	005056b0333f
vm_mor	vm-118231
vm_hostname	node1.controller.local
-------------------------------------+
  • Login to the shell prompt of the old leader site which should act as the follower site. Execute gslb changeleader new_leader cluster-UUIS view_id <view-id>.

> gslb changeleader new_leader cluster-005056b0333f view_id 1525801573

Note: In the previous example, the cluster-uuid and view_id are the values from the new leader site.

The above steps make sure that there is only one leader site in the GSLB set-up. The configuration is pushed once again to new follower site from the new leader site.

Observations

The following are the few points that should be considered while making this change, as the GSLB site may show some inconsistencies:

  • There might be configuration synchronization issues between the two sites if the leader is changed manually.
  • There will be a disruption in the traffic while doing these changes as changing the leader site will initiate the configuration synchronization from the new leader site to the old leader site.
  • DNS records may also have inconsistencies as the new leader might synchronize fewer records to the previous leader site(the new follower).

Additional Information