How to Change a Follower Site to the New Leader Site

Background

This article discusses how to manually change a follower site to the new leader site. The following are the scenarios when the manual change of the GSLB site is required:

  • Unable to detach GSLB follower site from the unresponsive leader site. Configuration changes can not be done as the configuration changes are done only on the leader site.
  • Manual election of GSLB leader site is also required in a network partitioning scenario. Network partitioning is similar to the split-brain condition, which occurs due to failures or outages in the internet or VPN infrastructure between the sites. In case of a network failure, each site updates the GSLB member state based on control-plane and data-plane health monitors. Both parts of the network act as independent and exclusive subnetworks. A new leader is not automatically elected (as per design) on the other network partition.
  • When there is a requirement to move leadership to another to another site to perform updates or maintenance work on the current leader.

When the GSLB leader site goes down, and the other follower sites are still running, traffic is handled without any issues by the Service Engines in all the sites. If there is no GSLB configuration change needed, changing of the leader site should be avoided. It is recommended to avoid changing the leader site , if you are experiencing a temporary outage, and expect to restore the leader in a short period of time. Please change the leader site only if:

  • Configuration updates are needed.
  • The leader site will be down for a considerable period of time.
  • In the long term, you truly need to change the leader to the new site.

Instructions

A follower site in a GSLB deployment can be manually promoted to the leader role by following the below steps.

  • Login to the shell prompt of the follower site which is to become the leader site. Use the show cluster command, noting the value of the UUID (vm_uuid) from the below output. The value of UUID in the below output is 005056b0333f.

> show cluster
-------------------------------------+

Field	Value
-------------------------------------+

uuid	cluster-005056b0333f
name	cluster-0-1
nodes[1]	 
name	10.10.30.55
ip	10.10.30.55
vm_uuid	005056b0333f
vm_mor	vm-118231
vm_hostname	node1.controller.local
-------------------------------------+

2) Run the gslb changeleader new_leader cluster-UUID command on the follower node which is to become the new leader site. Use the cluster UUID (005056b0333f) as captured from the previous step:


> gslb changeleader new_leader cluster-005056b0333f

Sample output is shown below:


> gslb changeleader new_leader cluster-005056b0333f
--------------------------------------------------------+

Field	Value
--------------------------------------------------------+

new_leader	cluster-005056b0333f
view_id	1525803021
details[1]	Review event-logs for additional information
--------------------------------------------------------+

Note: After the above steps mentioned are complete, the follower site starts synchronizing the GSLB configuration to itself. Since the other site (the old leader site) is not reachable, the new leader site stops its attempt to synchronize information after max-retries.

Changes on the previous leader site when it comes up

It is recommended to recover the previous leader site so that GSLB set-up is fully functional between the both sites. When the network partition condition is recovered, both the nodes can communicate to each other. To avoid any conflict issues, configuration changes are required to change the previous leader site to become afollower site. The previous leader site should refer to new leader site as the only actual leader site.

To avoid any conflicts, execute the below commands to make the previous leader site a follower:

  • Run the show gslb Default | grep -i view command on the new leader site and note down the view_id as shown below:

> show gslb Default | grep -i view

view_id	1525801573

Also execute the show cluster command on the new leader site and make a note of UUID (005056b0333f).


> show cluster
-------------------------------------+

Field	Value
-------------------------------------+

uuid	cluster-005056b0333f
name	cluster-0-1
nodes[1]	 
name	10.10.30.55
ip	10.10.30.55
vm_uuid	005056b0333f
vm_mor	vm-118231
vm_hostname	node1.controller.local
-------------------------------------+
  • Login to the shell prompt of the old leader site (which how must act as the follower site), and execute the gslb changeleader new_leader cluster-UUIS view_id <view-id> command.

gslb changeleader new_leader cluster-005056b0333f view_id 1525801573

Note: In the previous example, the cluster-uuid and view_id are the values from the new leader site.

The above steps make sure that there is only one leader site in the GSLB set-up. The configuration is pushed once again to new follower site from the new leader site.

Confirming the changes

To check whether the GSLB leader election has succeeded or not, use the following command on all the sites:
show gslb runtime</code>
The output will indicate the new leader site.

Observations

The following are the few points that should be considered while making these changes, as the GSLB site may show some inconsistencies:

  • There might be configuration synchronization issues between the two sites if the leader is changed manually.
  • There will be a disruption in traffic while doing these changes, since changing the leader site initiates the configuration synchronization from the new leader site to the old leader site.
  • DNS records may also have inconsistencies, since the new leader might synchronize fewer records to the previous leader site (the new follower).

Additional Information