Install and Configure Avi Multi-Cluster Kubernetes Operator

Overview

The Avi Multi-Cluster Kubernetes Operator (AMKO) facilitates application deployments across multiple OpenShift/Kubernetes clusters.

Note: Currently, AMKO is under Tech Preview.

AMKO is aware of the following object types:

Kubernetes:
- Ingress
- Service type load balancer
OpenShift:
- Routes
- Service type load balancer

AMKO is a Kubernetes pod that will run in a separate namespace called avi-system. The Avi Kubernetes Operator (AKO) also runs in the same namespace.

The installation of AMKO is done via Helm which applies the following permissions via a Kubernetes serviceaccount:

READ GSLBConfig and GlobalDeploymentPolicy instances
UPDATE GSLBConfig and GlobalDeploymentPolicy status fields

AMKO also requires permissions to read the ingress and service objects for all the member clusters. For this, a separate kubeconfig file is created with all the required permissions from all the clusters and is passed to AMKO via an OpenShift/Kubernetes secret object.

To know more, refer to the following articles:

Requirements

This section explains the minimum version requirements to use AMKO in Kubernetes clusters and OpenShift clusters:

For Kubernetes clusters:

Components	Version
Kubernetes	1.16+
AKO	0.9.1+
Avi Controller	18.2.9-4p1

For OpenShift Clusters:

Components	Version
OpenShift	4.4+
AKO	1.2.1-beta
Avi Controller	118.2.9-4p1

Pre-Installation Checklist

Ensure that the following tasks are completed before commencing AMKO installation:

At least one OpenShift/ Kubernetes cluster
At least one Avi Controller
AMKO assumes that it has connectivity to all the member clusters’ OpenShift/Kubernetes API servers. Without this, AMKO will not be able to watch over the Kubernetes and OpenShift resources in the member clusters.
Choose one of the Kubernetes/OpenShift clusters where AMKO should be deployed. All the configuration for AMKO will be added to this cluster. For example, cluster-amko.
Create a namespace avi-system in cluster-amko:


kubectl create ns avi-system

Create a kubeconfig file with the permissions to read the service and the ingress/route objects for all the member clusters.


    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: <base64 encoded ca.crt data>
        server: https://10.10.10.10:6443
      name: cluster1
    - cluster:
        certificate-authority-data: <base64 encoded ca.crt data>
        server: https://10.10.10.11:6443
      name: cluster2
    
    contexts:
    - context:
        cluster: cluster1
        namespace: default
        user: admin1
      name: cluster1-admin
    
    - context:
        cluster: cluster2
        namespace: default
        user: admin2
      name: cluster2-admin
    
    kind: Config
    preferences: {}
    users:
    - name: admin1
      user:
        client-certificate-data: <client.crt>
        client-key-data: <client.key>
    - name: admin2
      user:
        client-certificate-data: <client.crt>
        client-key-data: <client.key>

To know more refer to the article Creating a kubeconfig file for multi cluster access.

Name this file gslb-members and generate a secret with the kubeconfig file in cluster-amko as shown below:


kubectl create secret generic gslb-config-secret --from-file gslb-members -n avi-system

The permissions provided in the kubeconfig file for all the clusters must have at least the permissions to [get, list, watch] on:

Kubernetes ingress and service type load balancers.
OpenShift routes and service type load balancers.

AMKO also needs permissisons to [get, list, watch, update] on:

GSLBConfig object
GlobalDeploymentPolicy object The extra update permission is to update the GSLBConfig and GlobalDeploymentPolicy objects’ status fields to reflect the current state of the object, whether its accepted or rejected.
A kubeconfig file has to be created with the permissions to read the service and ingress objects for all the member clusters.

Note: The names provided in the cluster-contexts match the OpenShift/Kubernetes cluster names.

Installing AMKO

To install AMKO via Helm,

Create the avi-system namespace:
```
 kubectl create ns avi-system
 
```

Add this repository to your helm client:


 helm repo add amko https://avinetworks.github.io/avi-helm-charts/charts/incubator/amko

Configure the parameters using values.yaml.

Search the available charts for AMKO:


 helm search repo
    
  NAME                 	CHART VERSION	         APP VERSION	       DESCRIPTION
  amko/amko              1.2.1-5032-beta        	1.2.1-5032-beta    A helm chart for Avi Multicluster Kubernetes Operator

Install AMKO:


helm install  amko/amko  --generate-name --version 1.2.1-5032-beta -f values.yaml  --set configs.controllerIP=*controller_ip* --namespace=avi-system
    
NAME: amko-1593523840
LAST DEPLOYED: Tue Jun 30 19:00:44 2020
NAMESPACE: avi-system
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace avi-system -l "app.kubernetes.io/name=amko,app.kubernetes.io/instance=amko-1593523840" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace avi-system port-forward $POD_NAME 8080:80

Verify the installation:


helm list -n avi-system
    
NAME          	NAMESPACE 	REVISION	UPDATED                             	STATUS  	CHART    	                APP VERSION
amko-1593523840	avi-system	1       	2020-06-30 19:00:44.134075 +0530 IST	deployed	amko-1.2.1-5032-beta

Configuring AMKO

AMKO is configured via two CRDs - GSLBConfig and GlobalDeploymentPolicy. The Helm based installation procedure will automatically create these two in the specified namespaces.

During AMKO installation, set the required parameters via values.yaml. These parameters are translated to the GSLBConfig and the GlobalDeploymentPolicy objects.

Parameter	Description	Default
`configs.controllerVersion`	Leader site’s controller version	18.2.9
`configs.gslbLeaderController`	GSLB leader site URL	Not Applicable
`gslbLeaderCredentials.username`	GSLB leader controller username	admin
`gslbLeaderCredentials.password`	GSLB leader controller password
`configs.memberClusters`	OpenShift/Kubernetes cluster contexts as named in the kubeconfig file	Not Applicable
`configs.refreshInterval`	Interval in seconds after which the in-memory caches are synced with the existing GS objects	1800
`configs.domainNames`	Domain names supported in the GSLB configuration	foo.com
`gdpConfig.appSelector.label{.key,.value}`	Selection criteria for applications, label key and value are provided	Not Applicable
`gdpConfig.namespaceSelector.label{.key,.value}{.key,.value}`	Selection criteria for namespaces, label key and value are provided	Not Applicable
`gdpConfig.matchClusters`	List of clusters (names must match the names in configs.memberClusters) from where the objects will be selected	Not Applicable
`gdpConfig.trafficSplit`	List of weights for clusters (names must match the names in configs.memberClusters), each weight must range from 1 to 20	Not Applicable

With the AMKO installation based on the parameters configured above, Helm creates a GSLBConfig and GlobalDeploymentPolicy object.
If appSelector and namespaceSelector were not specified via the Helm installation (values.yaml), the respective fields will be empty in the resulting GlobalDeploymentPolicy object.
This means that by default, no objects are selected.
You can edit this GlobalDeploymentPolicy object and add the required criteria for app or cluster selection.

Any changes to the GSLBConfig object would require a restart of AMKO. However, any change to the GlobalDeploymentPolicy object is handled during runtime and it is not required to restart AMKO.

To know more about the GSLBConfig object, refer to the article GlobalDeploymentPolicy CRD for AMKO.

For example, the gdpConfig.appSelector.label, gdpConfig.namespaceSelector.label, gdpConfig.matchClusters, and gdpConfig.trafficSplit can be changed during runtime without requiring an AMKO restart.

Following is an example of a Global Deployment Policy instance:


apiVersion: "amko.k8s.io/v1alpha1"
kind: "GlobalDeploymentPolicy"
metadata:
  name: "global-gdp"
  namespace: "avi-system"   // a cluster-wide GlobalDeploymentPolicy
spec:
  matchRules:
    appSelector:
      label:
        app: gslb
    namespaceSelector:
      label:
        app: gslb
 
  matchClusters:
    - cluster: cluster1-admin    // cluster names are kubernetes cluster contexts
    - cluster: cluster2-admin
 
  trafficSplit:
    - cluster: cluster1
      weight: 8
    - cluster: cluster2
      weight: 2

Here,

Namespace: namespace of this object must be avi-system.

MatchRules: MatchRules define the application selection criteria and namespace selection criteria.


  matchRules:
  appSelector:                       // application selection criteria
    label:
      app: gslb                       // kubernetes/openshift label key-value
  namespaceSelector:                 // namespace selection criteria
    label:
      ns: gslb                        // kubernetes/openshift label key-value

A combination of appSelector and namespaceSelector decide which objects will be selected for GSLB service consideration.
- appSelector: The selection criteria only for application
  label: will be used to match the ingress/service type load balancer labels (key:value pair).
- namespaceSelector: Selection criteria only for namespaces
  label: will be used to match the namespace labels (key:value pair).

Currently, GlobalDeploymentPolicy supports the following combinations:

appSelector	namespaceSelector	Result
Yes	Yes	GDP selects all the objects satisfying appSelector from namespaces which satisfy the namespaceSelector
No	Yes	GDP selects all objects from the selected namespaces (satisfying namespaceSelector)
Yes	No	GDP selects all objects satisfying the appSelector criteria from all the namespaces
No	No	No objects selected (Default)

All ingresses and services of type load balancer which satisfy the matchRules will be selected. If a specific application needs to be excluded from the GlobalDeploymentPolicy, the label needs to be removed from the application.

For example,
Select objects with label app:gslb from all the namespaces:
```
 
  matchRules:
      appSelector:
        label:
          app: gslb
  
```
Select objects with app:gslb and from namespaces labelled ns:prod:
```
  matchRules:
      appSelector:
        label:
          app: gslb
      namespaceSelector:
        label:
          ns: prod
  
```
Select all objects from namespaces labelled: ns:prod
```
  matchRules:
      namespaceSelector:
        label:
          ns: prod
  
```
matchClusters: List of clusters on which the above matchRules will be applied on. The member objects of this list are cluster contexts of the individual OpenShift/Kubernetes clusters.
trafficSplit : The GlobalDeploymentPolicy object provides a cluster wide traffic split configuration between the clusters via trafficSplit. In the GlobalDeploymentPolicy object, if you want to split the traffic in the ratio of 80:20 between cluster1-admin and cluster2-admin. Configure the traffic split as shown below:
```
    
  trafficSplit:
   - cluster: cluster1-admin
     weight: 8
   - cluster: cluster2-admin
     weight: 2
 
```
*Note: Overriding of the traffic weight for individual applications is currently not supported.
GlobalDeploymentPolicy objects are editable. Changes made to the GlobalDeploymentPolicy object reflects on the Avi objects, if applicable.
Deleting or Updating of the GlobalDeploymentPolicy matchRules trigger all the objects to be re-validated against the remaining set of rules.
Deleting a cluster member from matchClusters triggers the deletion of objects synced from that cluster in Avi.

Uninstall AMKO via Helm

To uninstall AMKO using Helm,



helm uninstall -n avi-system <amko-release-name>

To remove the GSLB services created, remove the GDP object first.
This will remove all the GSLB services selected via the GDP object.


kubectl delete gdp -n avi-system global-gdp

Delete the avi-system namespace:


kubectl delete ns avi-system

Document Revision History

Date	Change Summary
July 21, 2020	Published the AMKO Installation Guide (Tech Preview)