Install and Configure Avi Multi-Cluster Kubernetes Operator

Overview

The Avi Multi-Cluster Kubernetes Operator (AMKO) facilitates application deployments across multiple OpenShift/Kubernetes clusters.

Note: Currently, AMKO is under Tech Preview.

AKO

AMKO is aware of the following object types:

  • Kubernetes:
    • Ingress
    • Service type load balancer
  • OpenShift:
    • Routes
    • Service type load balancer

AMKO is a Kubernetes pod that will run in a separate namespace called avi-system. The Avi Kubernetes Operator (AKO) also runs in the same namespace.

The installation of AMKO is done via Helm which applies the following permissions via a Kubernetes serviceaccount:

  • READ GSLBConfig and GlobalDeploymentPolicy instances
  • UPDATE GSLBConfig and GlobalDeploymentPolicy status fields

AMKO also requires permissions to read the ingress and service objects for all the member clusters. For this, a separate kubeconfig file is created with all the required permissions from all the clusters and is passed to AMKO via an OpenShift/Kubernetes secret object.

To know more, refer to the following articles:

Requirements

This section explains the minimum version requirements to use AMKO in Kubernetes clusters and OpenShift clusters:

For Kubernetes clusters:

Components Version
Kubernetes 1.16+
AKO 0.9.1+
Avi Controller 18.2.9-4p1

For OpenShift Clusters:

Components Version
OpenShift 4.4+
AKO 1.2.1-beta
Avi Controller 118.2.9-4p1

Pre-Installation Checklist

Ensure that the following tasks are completed before commencing AMKO installation:

  • At least one OpenShift/ Kubernetes cluster

  • At least one Avi Controller

  • AMKO assumes that it has connectivity to all the member clusters’ OpenShift/Kubernetes API servers. Without this, AMKO will not be able to watch over the Kubernetes and OpenShift resources in the member clusters.

  • Choose one of the Kubernetes/OpenShift clusters where AMKO should be deployed. All the configuration for AMKO will be added to this cluster. For example, cluster-amko.

  • Create a namespace avi-system in cluster-amko:


kubectl create ns avi-system
  • Create a kubeconfig file with the permissions to read the service and the ingress/route objects for all the member clusters.

    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: <base64 encoded ca.crt data>
        server: https://10.10.10.10:6443
      name: cluster1
    - cluster:
        certificate-authority-data: <base64 encoded ca.crt data>
        server: https://10.10.10.11:6443
      name: cluster2
    
    contexts:
    - context:
        cluster: cluster1
        namespace: default
        user: admin1
      name: cluster1-admin
    
    - context:
        cluster: cluster2
        namespace: default
        user: admin2
      name: cluster2-admin
    
    kind: Config
    preferences: {}
    users:
    - name: admin1
      user:
        client-certificate-data: <client.crt>
        client-key-data: <client.key>
    - name: admin2
      user:
        client-certificate-data: <client.crt>
        client-key-data: <client.key>
 

To know more refer to the article Creating a kubeconfig file for multi cluster access.

Name this file gslb-members and generate a secret with the kubeconfig file in cluster-amko as shown below:


kubectl create secret generic gslb-config-secret --from-file gslb-members -n avi-system

The permissions provided in the kubeconfig file for all the clusters must have at least the permissions to [get, list, watch] on:

  • Kubernetes ingress and service type load balancers.
  • OpenShift routes and service type load balancers.

AMKO also needs permissisons to [get, list, watch, update] on:

  • GSLBConfig object
  • GlobalDeploymentPolicy object The extra update permission is to update the GSLBConfig and GlobalDeploymentPolicy objects’ status fields to reflect the current state of the object, whether its accepted or rejected.

  • A kubeconfig file has to be created with the permissions to read the service and ingress objects for all the member clusters.

Note: The names provided in the cluster-contexts match the OpenShift/Kubernetes cluster names.

Installing AMKO

To install AMKO via Helm,

  1. Create the avi-system namespace:

    
     kubectl create ns avi-system
     
  2. Add this repository to your helm client:
    
     helm repo add amko https://avinetworks.github.io/avi-helm-charts/charts/incubator/amko
     
  3. Configure the parameters using values.yaml.

  4. Search the available charts for AMKO:

    
     helm search repo
        
      NAME                 	CHART VERSION	         APP VERSION	       DESCRIPTION
      amko/amko              1.2.1-5032-beta        	1.2.1-5032-beta    A helm chart for Avi Multicluster Kubernetes Operator
      
  5. Install AMKO:
    
    helm install  amko/amko  --generate-name --version 1.2.1-5032-beta -f values.yaml  --set configs.controllerIP=*controller_ip* --namespace=avi-system
        
    NAME: amko-1593523840
    LAST DEPLOYED: Tue Jun 30 19:00:44 2020
    NAMESPACE: avi-system
    STATUS: deployed
    REVISION: 1
    NOTES:
    1. Get the application URL by running these commands:
      export POD_NAME=$(kubectl get pods --namespace avi-system -l "app.kubernetes.io/name=amko,app.kubernetes.io/instance=amko-1593523840" -o jsonpath="{.items[0].metadata.name}")
      echo "Visit http://127.0.0.1:8080 to use your application"
      kubectl --namespace avi-system port-forward $POD_NAME 8080:80
     
  6. Verify the installation:

    
    helm list -n avi-system
        
    NAME          	NAMESPACE 	REVISION	UPDATED                             	STATUS  	CHART    	                APP VERSION
    amko-1593523840	avi-system	1       	2020-06-30 19:00:44.134075 +0530 IST	deployed	amko-1.2.1-5032-beta
    

Configuring AMKO

AMKO is configured via two CRDs - GSLBConfig and GlobalDeploymentPolicy. The Helm based installation procedure will automatically create these two in the specified namespaces.

During AMKO installation, set the required parameters via values.yaml. These parameters are translated to the GSLBConfig and the GlobalDeploymentPolicy objects.

Parameter Description Default
configs.controllerVersion Leader site’s controller version 18.2.9
configs.gslbLeaderController GSLB leader site URL Not Applicable
gslbLeaderCredentials.username GSLB leader controller username admin
gslbLeaderCredentials.password GSLB leader controller password
configs.memberClusters OpenShift/Kubernetes cluster contexts as named in the kubeconfig file Not Applicable
configs.refreshInterval Interval in seconds after which the in-memory caches are synced with the existing GS objects 1800
configs.domainNames Domain names supported in the GSLB configuration foo.com
gdpConfig.appSelector.label{.key,.value} Selection criteria for applications, label key and value are provided Not Applicable
gdpConfig.namespaceSelector.label{.key,.value}{.key,.value} Selection criteria for namespaces, label key and value are provided Not Applicable
gdpConfig.matchClusters List of clusters (names must match the names in configs.memberClusters) from where the objects will be selected Not Applicable
gdpConfig.trafficSplit List of weights for clusters (names must match the names in configs.memberClusters), each weight must range from 1 to 20 Not Applicable

With the AMKO installation based on the parameters configured above, Helm creates a GSLBConfig and GlobalDeploymentPolicy object.
If appSelector and namespaceSelector were not specified via the Helm installation (values.yaml), the respective fields will be empty in the resulting GlobalDeploymentPolicy object.
This means that by default, no objects are selected.
You can edit this GlobalDeploymentPolicy object and add the required criteria for app or cluster selection.

Any changes to the GSLBConfig object would require a restart of AMKO. However, any change to the GlobalDeploymentPolicy object is handled during runtime and it is not required to restart AMKO.

To know more about the GSLBConfig object, refer to the article GlobalDeploymentPolicy CRD for AMKO.

For example, the gdpConfig.appSelector.label, gdpConfig.namespaceSelector.label, gdpConfig.matchClusters, and gdpConfig.trafficSplit can be changed during runtime without requiring an AMKO restart.

Following is an example of a Global Deployment Policy instance:


apiVersion: "amko.k8s.io/v1alpha1"
kind: "GlobalDeploymentPolicy"
metadata:
  name: "global-gdp"
  namespace: "avi-system"   // a cluster-wide GlobalDeploymentPolicy
spec:
  matchRules:
    appSelector:
      label:
        app: gslb
    namespaceSelector:
      label:
        app: gslb
 
  matchClusters:
    - cluster: cluster1-admin    // cluster names are kubernetes cluster contexts
    - cluster: cluster2-admin
 
  trafficSplit:
    - cluster: cluster1
      weight: 8
    - cluster: cluster2
      weight: 2

Here,

  • Namespace: namespace of this object must be avi-system.

  • MatchRules: MatchRules define the application selection criteria and namespace selection criteria.
    
      matchRules:
      appSelector:                       // application selection criteria
        label:
          app: gslb                       // kubernetes/openshift label key-value
      namespaceSelector:                 // namespace selection criteria
        label:
          ns: gslb                        // kubernetes/openshift label key-value
      
  • A combination of appSelector and namespaceSelector decide which objects will be selected for GSLB service consideration.

    • appSelector: The selection criteria only for application
      label: will be used to match the ingress/service type load balancer labels (key:value pair).

    • namespaceSelector: Selection criteria only for namespaces
      label: will be used to match the namespace labels (key:value pair).

  • Currently, GlobalDeploymentPolicy supports the following combinations:

    appSelector namespaceSelector Result
    Yes Yes GDP selects all the objects satisfying appSelector from namespaces which satisfy the namespaceSelector
    No Yes GDP selects all objects from the selected namespaces (satisfying namespaceSelector)
    Yes No GDP selects all objects satisfying the appSelector criteria from all the namespaces
    No No No objects selected (Default)
  • All ingresses and services of type load balancer which satisfy the matchRules will be selected. If a specific application needs to be excluded from the GlobalDeploymentPolicy, the label needs to be removed from the application.

    For example,
    Select objects with label app:gslb from all the namespaces:

     
      matchRules:
          appSelector:
            label:
              app: gslb
      

    Select objects with app:gslb and from namespaces labelled ns:prod:

    
      matchRules:
          appSelector:
            label:
              app: gslb
          namespaceSelector:
            label:
              ns: prod
      

    Select all objects from namespaces labelled: ns:prod

    
      matchRules:
          namespaceSelector:
            label:
              ns: prod
      
  • matchClusters: List of clusters on which the above matchRules will be applied on. The member objects of this list are cluster contexts of the individual OpenShift/Kubernetes clusters.

  • trafficSplit : The GlobalDeploymentPolicy object provides a cluster wide traffic split configuration between the clusters via trafficSplit. In the GlobalDeploymentPolicy object, if you want to split the traffic in the ratio of 80:20 between cluster1-admin and cluster2-admin. Configure the traffic split as shown below:
    
        
      trafficSplit:
       - cluster: cluster1-admin
         weight: 8
       - cluster: cluster2-admin
         weight: 2
     

    *Note: Overriding of the traffic weight for individual applications is currently not supported.

  • GlobalDeploymentPolicy objects are editable. Changes made to the GlobalDeploymentPolicy object reflects on the Avi objects, if applicable.

  • Deleting or Updating of the GlobalDeploymentPolicy matchRules trigger all the objects to be re-validated against the remaining set of rules.

  • Deleting a cluster member from matchClusters triggers the deletion of objects synced from that cluster in Avi.

Uninstall AMKO via Helm

To uninstall AMKO using Helm,



helm uninstall -n avi-system <amko-release-name>

To remove the GSLB services created, remove the GDP object first.
This will remove all the GSLB services selected via the GDP object.


kubectl delete gdp -n avi-system global-gdp

Delete the avi-system namespace:


kubectl delete ns avi-system

Document Revision History

Date Change Summary
July 21, 2020 Published the AMKO Installation Guide (Tech Preview)