Customizing Avi Controller’s Initial Configuration

Often there is a need to deploy multiple Avi Controller cluster instances starting from some common initial single Controller configuration that differs from the factory-default settings. This can be achieved by creating a JSON file with the required object definitions and then using it during deployment of subsequent Controllers, which, as leaders, can then add followers to become clusters. At the end of the process, the result is a set of identically initialized Controller clusters, ready to be individualized as needed.

Creating setup.json

In most cases, these objects can be created by referring to the Avi REST API Guide.

The example below updates the system configuration by adding 8.8.8.8 to the DNS configuration: 

{
    "SystemConfiguration": [
        {
            "dns_configuration": {
                "search_domain": "",
                "server_list": [
                    {
                        "type": "V4",
                        "addr": "8.8.8.8"
                    }
                ]
            }
        }
    ]
}

 

In the case of complex objects such as SSLKeyAndCertificate object, the JSON file can be created by running a diff command against two configuration files. In a typical deployment, it is recommended to generate setup.json on a test controller environment. This generated file can then by used a template for actual deployments. An Avi Controller configuration snapshot can be taken using the export CLI command: 

> export configuration file before.cfg
Please enter the passphrase to encrypt configuration:
Downloaded the attachment to before.cfg
Completed writing the export configuration to before.cfg

 

At this point, configuration objects can be created using the UI or CLI. After making necessary changes using the UI/CLI configuration, a snapshot is taken once again: 

> export configuration file after.cfg
Please enter the passphrase to encrypt configuration:
Downloaded the attachment to after.cfg
Completed writing the export configuration to after.cfg

Beyond this, configuration diff can be taking using a Python script Avi Networks has written expressly for the purpose of customizing another Controller’s initial configuration: 

/opt/avi/scripts/diff_config.py -f before.cfg -t after.cfg > setup.json

 

In case there is a need to encrypt the user passwords while creating setup.json with “User” object, user passwords can be encrypted using the following: 

/opt/avi/scripts/avi_passwd_tool.py --password admin --salt fF6ngAb3pvPgpbkdf2_sha256$100000$fF6ngAb3pvPg$ijkEue1M9fR/qsLVgzvPe7N0VvOxIjDiJVmK9NIx+0Q=$6$fF6ngAb3pvPg$CqAKtNRZtgXtJchrPmoxUgdLFM7rFGmta1tWb7sobQI4iSZAY2QuAOBNtboVGrmDYPMCvqXXH6lARr9RedCJT.

Deployment Using setup.json

As a best practice, it is recommended to take a configuration backup before deploying the Controller using setup.json file created by the Python script. The following command will take an encrypted backup of the existing configuration. 

/opt/avi/python/bin/portal/manage.py export_configuration --file ~/setup-old.json --passphrase secret

For a Mesos/Bare-Metal Deployment:

setup.json needs to be copied to the persistent directory in the host that is mounted as /vol in the controller container. If you are using avi_baremetal script, the default location is /opt/avi/controller/data on the host. When deploying the Controller as container, setup.json can be passed as an additional argument to avi_baremetal_setup.py script. For example: 

./avi_baremetal_setup.py -c -cc 4 -cm 12 -i 10.10.22.108 -m 10.10.22.108 --setup-json /root/configs/avi-setup.json

For a Controller Deployment as a VM:

Wait until Controller comes up. Place the config file on the Controller as /var/lib/avi/etc/setup.json (note the filename). Upon reboot/fresh-start, the Avi Controller will self-configure using the provided setup.json file.

reboot

For an OpenStack Deployment:

UserData config size is limited to 48 Kb. If size of setup.json is within allowable limits: 

># cfgdrv userdata
>nova boot --config-drive true --image avicontroller --key-name mykey --flavor 4 --user-data /root/my-avi-config.json --nic net-id=7402bf4f-240f-4172-99c1-90000ea45f86 avicontrollers

># metasvc userdata
>nova boot --config-drive false --image avicontroller --key-name mykey --flavor 4 --user-data /root/my-avi-config.json --nic net-id=7402bf4f-240f-4172-99c1-90000ea45f86 avicontrollers

If setup.json size is bigger then than the allowable limit, setup.json can be uploaded and referred in the deployment phase.

UserData can refer to the file either using “url” or “file” tag. Example of my-avi-config-url.json with URL: 

{
 "META": {
     "init_config": {
         "url": "https://s3-us-west-2.amazonaws.com/avi-controller-configs/linuxserver-awsipam-setup.json"
     }
   } 
}

Example of my-avi-config-url.json with filepath: 

{
  "META": {
    "init_config": {
      "file": "/vol/linuxserver-awsipam-setup.json"     
    }
  } 
}

For deployment:

># cfgdrv userdata indirection
>nova boot --config-drive true --image avicontroller --key-name mykey --flavor 4 --user-data /root/my-avi-config-url.json --nic net-id=7402bf4f-240f-4172-99c1-90000ea45f86 --min-count=3 --max-count=3 avicontrollers

># metasvc userdata indirection
nova boot --config-drive false --image avicontroller --key-name mykey --flavor 4 --user-data /root/my-avi-config-url.json --nic net-id=7402bf4f-240f-4172-99c1-90000ea45f86 --min-count=3 --max-count=3 avicontrollers

For an AWS Deployment:

UserData config size is limited to 16Kb. If size of setup.json is within allowable limits, cut-paste the my-avi-config.json into the user-data section during launch from AWS Web Console 

# metasvc userdata
 ec2-run-instances ami-b7ea27d7 -f /root/my-avi-config.json -t c4.2xlarge -s subnet-62f1b707 -g sg-642d8d02

 

If setup.json size is bigger then than the allowable limit, cut-paste the my-avi-config-url.json into the user-data section during launch from AWS Web Console. 

# metasvc userdata indirection
ec2-run-instances ami-b7ea27d7 -f /root/my-avi-config-url.json -t c4.2xlarge -s subnet-62f1b707 -g sg-642d8d02

 

my-avi-config-url.json has follow similar formats as discussed in the OpenStack section. Along with that, here is a sample my-avi-config-url.json file for S3 bucket: 

{
  "META": {
    "init_config": {
      "s3": "avi-controller-configs/linuxserver-awsipam-setup.json"
    }
  } 
}

 

For uploading setup.json on S3 bucket:

  • Public : use the ‘url’ style or ‘s3’ style
  • Private via RBAC on VM: use the ‘s3’ style. The VM role should have s3:GetObject action allowed for it to be able to s3-get the object using IAM.
  • Private via RBAC on S3-bucket: use the ‘s3’ style. The VM role should have AWS access. The S3 bucket should have permissions for the account or user or VM role to download the object.

Example bucket policy: 

{
   "Version": "2012-10-17",
   "Statement": [
     {
       "Sid": "AddPerm",
       "Effect": "Allow",
       "Principal": {
         "AWS": [
           "arn:aws:iam::139284885014:role/BM-AviController-Role",
           "arn:aws:iam::139284885014:root"
          ]
        },
       "Action": "s3:*",
       "Resource": "arn:aws:s3:::avi-controller-configs/*"
     }
   ]
 }

For an Azure Deployment

There are two ways to provide initial configuration in an Azure environment:

  • Using Azure CLI
  • Using Azure ATM template

Using Azure CLI

If the Avi Controller is deployed using Azure CLI, the JSON file can be provided during the deployment.


az vm create --resource-group rahulr-jenkins-resource-group --location centralus --image avi-networks:avi-vantage-adc:avi-vantage-adc-byol:17.2.7 --name Avi-Test-Controller --size Standard_F8s --subnet /subscriptions/<subscription_id>/resourceGroups/<resource_group_name>/providers/Microsoft.Network/virtualNetworks/<vnet_name>/subnets/rahulr-subnet --public-ip-address "" --nsg "" --custom-data ./initial_config.json

Using Azure ARM template

If the Avi Controller is deployed using ARM template,the JSON data is provided as the Custom Data on the Custom deployment page of the Azure portal. Navigate to Home>Templates>avi-cluster-managed-disks-market place>Custom deployment.

template

Use the required JSON template for the Custom Data field. For reference, the below JSON template is for adding 8.8.8.8 to DNS configuration.

{
    "SystemConfiguration": [
        {
            "dns_configuration": {
                "search_domain": "",
                "server_list": [
                    {
                        "type": "V4",
                        "addr": "8.8.8.8"
                    }
                ]
            }
        }
    ]
}

Copy the JSON configuration mentioned above, and add it to the Custom Data field in ARM template mentioned in the previous screenshot.