Avi Vantage Kubernetes/OpenShift Annotation Guide

Overview

When the Kubernetes/OpenShift Cloud is configured in Avi Vantage, a cloud-connector process continuously monitors Service, Ingress, and Route objects in the Kubernetes/OpenShift cluster. When new configuration or changes to these objects are detected, the Avi Controller creates or updates the Virtual Service, Pool, Pool Group and other Avi objects. This process automates the creation of Avi virtual services corresponding to the services created by the user in the Kubernetes/OpenShift cluster.

The automatic creation of virtual services in Avi Vantage requires the determination of fields like tenant, VIP placement network, pool, application profile, etc. The Avi Controller derives many of these fields from Kubernetes/OpenShift cluster objects. Some cluster objects are directly mapped to Avi objects (for example, Namespace is mapped to Tenant) while other objects are determined by a selection logic using multiple cluster objects (for example, application profile selection). This is shown in the figure below:

deployment

While some of the specific fields are derived from the cluster objects as illustrated above, other fields are used with default values. In many cases, users may require the configuration of non-default values for these fields, or may need to override the fields automatically derived from the Cluster objects by the controller. This can be achieved by configuring Avi specific ‘annotations’ in the Kubernetes/OpenShift service file for the Services/Ingress/Routes. While Kubernetes/Openshift validates of the syntax of the annotation, it does not interpret the annotations. The Avi Controller parses through these annotations, and configures the specified Avi Objects as defined in the annotation.

The configuration of syntactically incorrect JSON annotations will result in an error in Kubernetes/OpenShift while creating services. In contrast, the configuration of incorrect Avi objects or values in the annotation will only result in an error in Avi Vantage, but not in Kubernetes/OpenShift as long as the JSON syntax is correct. These errors can be reviewed in Events under Operations in the Avi UI.

This guide explains the annotation examples for the following configuration requirements:

In addition, the Building Annotations from Avi Objects section provides the steps to build custom annotations for any Avi Object that may not be covered in the examples in this document.

For more details on the Service Objects being synced between Avi Vantage and Kubernetes/OpenShift, refer to OpenShift/Kubernetes Service Configuration in Avi Vantage.

Prerequisites

Cloud installation should be performed as required using the normal procedure as mentioned in the installation guide.

Annotation Format

The values of annotation are in JSON format. While specifying it in service, ingress or route specs, Kubernetes/OpenShift treats it as strings (within single quotes). i.e. the annotations are in this format:

avi_proxy: 'string specifying avi configuration'

The annotation can be specified in a single line (without indentations) if required. But this may become difficult to read and edit as seen below:

kind: Service
apiVersion: v1
metadata:
  name: avisvc
  labels:
   svc: avisvc
  annotations:
   avi_proxy: '{"virtualservice": {"services": [{"port": 443, "enable_ssl": true}], "east_west_placement": false, "ssl_key_and_certificate_refs": ["/api/sslkeyandcertificate?name=Custom-EC-Cert"], "ssl_profile_ref": "/api/sslprofile/?name=System-Standard"}}'
spec:
  ports:
    - name: https
      port: 443
      targetPort: http
  selector:
    name: avitest

Also if the service, ingress or route specification is in JSON format, escape the double quotes inside the avi_proxy value as shown below, so that the value is read as a string.

{
    "kind":"Service",
    "apiVersion":"v1",
    "metadata":{
        "name":"avisvc",
        "labels":{
            "svc":"avisvc"
        },
        "annotations":{
            "avi_proxy":"{\\"virtualservice\\": {\\"services\\": [{\\"port\\": 443, \\"enable_ssl\\": true}], \\"east_west_placement\\": false, \\"ssl_key_and_certificate_refs\\": [\\"avisvccert\\"], \\"ssl_profile_ref\\": \\"/api/sslprofile/?name=System-Standard\\"}}"
        }
    },
    "spec":{
        "ports":[
            {
                "name":"https",
                "port":443,
                "targetPort":"http"
            }
        ],
        "selector":{
            "name":"avitest"
        }
    }
}

Using the API Version field in Annotation

As features get added and enhanced in newer versions of Avi Vantage software, the Avi Object model is liable to change between versions. However, for backward compatibility and ease-of-use, the Avi Controller automatically translates the Object model used in the annotation to the Object model of the current version on the Avi Controller. The Avi Controller uses the version field to interpret the Object model referenced in the annotation. Hence the annotations configured must conform to the Object model as specified by the version field. When a new feature is introduced, the use of its new objects will require the version field on the annotation to be at least the minimum required Avi Vantage version for the feature. To view most commonly used Avi objects, refer to the Building Annotations from Avi Objects section

Updating Apps and Annotations

Avi Vantage annotations are updated declaratively. Hence, when updating an app in Kubernetes/OpenShift, the annotations used previously need to be retained. Without this, the existing fields that are not called out in the update will be overwritten with default values. The best practice prior to updating an app or annotation is to read the existing object configuration using get in Kubernetes or OpenShift prior to making the required changes. Another option is to save or version configuration files prior to every create or update operation and use the previous configuration file as the baseline prior to making changes.

Annotations for Virtual Service Configuration

Annotations can be used for customizing the following configuration requirements for a virtual service:

Setting Static VIP

A static VIP can be manually assigned to a virtual service using the below annotation. The IP address used must be outside the range of IPs allocated for IPAM. The syntax used is "addr":"Desired IP Address" In the below example, the static IP address alloted to a virtual service is 10.130.150.203.

avi_proxy: '{
    "virtualservice":{
        "ip_address":{
            "addr":"10.130.150.203",
            "type":"V4"
        }
    }
}'

Setting VS Placement Type to North-South

The following annotation configures a VIP from North-South IPAM by setting the value of flag "east_west_placement" to false.

avi_proxy: '{"virtualservice": {"east_west_placement": false}}'

The default value of “east_west_placement” is given by the “Default Service as East-West Service” flag in cloud configuration Kubernetes/OpenShift install guide. Use this flag only in Kubernetes or OpenShift Service annotation. Ingress and route objects always create north-south virtual service.

Configuring Floating VIP

The following example creates a North-South virtual service with an auto-allocated VIP (private IP), and a static VIP (Public IP). The “auto_allocate_ip” flag is set explicitly to true when a floating IP is configured. The syntax for setting floating IP address is "floating_ip": {"addr": "Desired IP Address", "type": "V4". In the below example, the floating IP address assigned to a virtual service is 35.225.58.194.

avi_proxy: '{
    "virtualservice": {
         "vip": [{
                   "auto_allocate_ip": true,
                   "floating_ip": {"addr": "35.225.58.194", "type": "V4"},
                   "east_west_placement": false
                }]
                       },
    "version": "17.2.9"
}'

Configuring Floating VIP in AWS

The following example configures a floating virtual IP address to a virtual service with Kubernetes running on AWS using AWS IPAM.


avi_proxy: "{\
"virtualservice\": {\
"vip\": [
{\"subnet_uuid\": \"subnet-d506939c\", \"auto_allocate_ip\": true, \"vip_id\": \"0\", \"auto_allocate_floating_ip\": true}
]}}"

Setting VIP IPAM Network

If IPAM has multiple networks configured, the “network_ref” needs to be specified in the annotation to select the network from which the VIP should be allocated. The following annotation allocates the VIP to the Network1 network.

avi_proxy: '{
    "virtualservice":{
        "network_ref":"/api/network?name=Network1"
    }
}'

This is not required if IPAM has only one network or if any of the listed networks can be used.

Use the following annotation to specify a network for auto-allocation that is NOT listed in the IPAM object: Note: This annotation requires a minimum API version (17.2.4).

avi_proxy: '{
    "version": "17.2.4",
    "virtualservice":{
        "vip": [{
            "auto_allocate_ip": "true",
            "ipam_network_subnet": {
                 "network_ref":"/api/network?name=Network1"
            }
        }]
    }
}'

Advertising VIP using BGP

The following annotation enable VIP advertisements using BGP. Set the value of the enable_rhi flag to true. If the flag value is not specified, the default value is set to false.

avi_proxy: '{
    "virtualservice":{
        "enable_rhi":true
    }
}'

Enabling ECMP

The following annotation enables the network or routing to perform ECMP over the Service Engines on which the virtual service is placed (e.g. in GCP). Set the value of “scaleout_ecmp” flag to true. If not specified, the default value is set to false.

avi_proxy: '{
    "virtualservice":{
        "scaleout_ecmp":true
    }
}'

Note: “scaleout_ecmp”must not be used when “enable_rhi” is set to true.

Overriding Ingress/Route for a Virtual Service

The Use Shared Virtual Service flag in the OpenShift cloud configuration maps all HTTP/HTTPS ingress/route to the parent virtual service. The mapping is done either by using SNI (for HTTPS) or content switching based on Host header (HTTP). The following annotation overrides the existing configuration to create a dedicated virtual service for a particular ingress/route. Set the value of flag dedicate_route to true.

avi_proxy: '{
    "dedicated_route": true
}'

If the value of Use Shared Virtual Service is not set in the cloud configuration, dedicated virtual services are created for all ingresses/routes. The following annotation overrides the configuration to map the ingress/route to the parent virtual service.

avi_proxy: '{
     "dedicated_route": false
 }'

Overriding Shared Ingress/Route for a Namespace

The Use Shared Virtual Service flag in the cloud configuration maps all HTTP/HTTPS ingress/route to the parent virtual service. It mapping is done either using SNI (for HTTPS) or content switching based on host header (HTTP). The following annotation overrides the configuration to create dedicated virtual services for all ingresses/routes in the namespace.

avi_virtualservice: dedicated

If the Use Shared Virtual Service is not set in the cloud configuration, dedicated virtual services are created for all ingress/routes. The following annotation overrides this configuration to map the ingress/route in a namespace to the parent virtual service.

avi_virtualservice: shared

Note: This annotation can be used with Namespace spec. Refer to the links below for more details:

Pool Specific Annotations

Annotations can be used to customize the following configurations for a pool or pool group:

Configuring Load Balancing Algorithm

The following annotation configures the desired load balancing method for the load balancing pool. Supported values are:

  • LB_ALGORITHM_LEAST_CONNECTIONS
  • LB_ALGORITHM_ROUND_ROBIN
  • LB_ALGORITHM_FASTEST_RESPONSE
  • LB_ALGORITHM_CONSISTENT_HASH
  • LB_ALGORITHM_LEAST_LOAD
  • LB_ALGORITHM_FEWEST_SERVERS

The annotation below configures the load balancing method to round robin.

avi_proxy: '{
    "pool":{
        "lb_algorithm":"LB_ALGORITHM_ROUND_ROBIN"
    }
}'

If not specified, the default value for load balancing method is set as LB_ALGORITHM_LEAST_CONNECTIONS.

For the Consistent Hash algorithm, also specify the field to be used for hashing. Supported values for the field are:

  • LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS
  • LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS_AND_PORT
  • LB_ALGORITHM_CONSISTENT_HASH_URI
  • LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER
  • LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_STRING
  • LB_ALGORITHM_CONSISTENT_HASH_CALLID

The example below sets the load balancing method to consistent hash and field to the source IP address.

avi_proxy: '{
    "pool":{
        "lb_algorithm":"LB_ALGORITHM_CONSISTENT_HASH",
        "lb_algorithm_hash":"LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS"
    }
}'

Note: Also specify lb_algorithm_consistent_hash_hdr in annotation for consistent hash using custom header.

Enabling SSL to Back-end Connections

If application pods listen on HTTPS, backend traffic can be re-encrypted. Specifying SSL profile in the pool annotations enabled re-encryption. Use the following annotation to re-encrypt back-end connections.

avi_proxy: '{
    "pool":{
        "ssl_profile_ref":"/api/sslprofile?name=System-Standard"
    }
}'

Setting Traffic Split Ratio

Pool group annotation is used to set a ratio, to send a part of traffic to a different pool. This can be used to perform the graceful blue-green upgrade of the application. The following annotation sets traffic ratio for the svcA pool and the svcB pool as 10:1.

avi_proxy: '{
    "poolgroup":{
        "members":[
            {
                "ratio":10,
                "pool_ref":"/api/pool/?name=svcA-pool-http-tcp"
            },
            {
                "ratio":1,
                "pool_ref":"/api/pool/?name=svcB-pool-http-tcp"
            }
        ]
    }
}'

Security Specific Annotations

The following WAF-related configurations can be customized using annotations:

Overriding Service Port or Enabling SSL

The port specified under “services” will override any service port specified in the Kubernetes/OpenShift configuration. Multiple ports can be configured for a virtual service if required. SSL can be enabled for each port by setting enable_ssl to true. This is set to false by default, if not specified. The following annotation enables SSL on port 443 for the virtual service.

avi_proxy: '{
    "virtualservice":{
        "services":[
            {
                "enable_ssl":true,
                "port":443
            }
        ]
    }
}'

If services annotation is not provided, the virtual service listens on port specified in Kubernetes/OpenShift service spec.

Setting SSL Profile

The following annotation sets the SSL profile of the virtual service to Custom-SSL-Profile. SSL profile is configured only if SSL is enabled on one of the ports associated with the virtual service.

avi_proxy: '{
    "virtualservice":{
        "ssl_profile_ref":"/api/sslprofile/?name=Custom-SSL-Profile"
    }
}'

Note: If SSL profile is not specified, it defaults to System-Standard.

Configuring Certificates

The following annotation configures the SSL certificates for a virtual service. The value is a list which can contain at most one EC cert and one RSA cert only. Any other combination will result in an error and the virtual service will not be created. The annotation below sets the EC certificate as System-Default-Cert-EC, and the RSA certificate as System-Default-Cert.

avi_proxy: '{
    "virtualservice":{
        "ssl_key_and_certificate_refs":[
            "/api/sslkeyandcertificate?name=System-Default-Cert-EC",
            "/api/sslkeyandcertificate?name=System-Default-Cert"
        ]
    }
}'

Use above annotation to use new features, introduced in certain version of Avi API by setting the version to correct value.

Creating WAF Policy

The annotation below creates a new WAF policy for the virtual service. The WAF policy will be named after the virtual service name. Starting with 17.2.4, Avi Vantage supports annotation for WAF.

avi_proxy: '{"waf_policy": true, "version": "17.2.4"}'

Creating Virtual Service with existing WAF Policy

The annotation below adds an existing WAF policy to a virtual service. This can also be used if an existing WAF policy is to be shared among multiple virtual services. Altering the WAF rules affects all virtual services using the policy. The annotation below adds an existing WAF policy – waf-dem-policy to the virtual service .

avi_proxy: '{
    "virtualservice":{
        "waf_policy_ref":"/api/wafpolicy?name=waf-demo-policy"
    },
    "version":"17.2.9"
}'

Miscellaneous Annotations

Adding HTTP Policy

The annotation shown below exhibits an example of an HTTP policy which checks if the SETUP header is present and modifies value of the header from TEST to SANDBOX.

The HTTP policy is valid only if application profile is of HTTP type.

Note: Writing annotation for HTTP policies can be complicated. An easy way is to create the policy using UI and copy the JSON from the REST output of GET call on the virtual service.

avi_proxy: '{
    "virtualservice":{
        "application_profile_ref": "/api/applicationprofile?name=System-HTTP",
        "http_policies": [
            {
                "index": 11,
                "http_policy_set_ref": "/api/httppolicyset?name=Demo-httppolicy"
            }
        ]
    },
    "httppolicyset":{  
        "name":"photo-Default-Cloud-HTTP-Policy-Set",
        "is_internal_policy":false,
        "index":11,
        "http_request_policy":{  
            "rules":[  
                {  
                    "enable":true,
                    "name":"Rule 1",
                    "_toPosition":{  
                        "index":0,
                        "position":"below"
                    },
                    "match":{  
                        "hdrs":[  
                            {  
                                "match_criteria":"HDR_EXISTS",
                                "hdr":"SETUP",
                                "value":[  
                                    "TEST"
                                ]
                            }
                        ]
                    },
                    "hdr_action":[  
                        {  
                            "action":"HTTP_REPLACE_HDR",
                            "hdr":{  
                                "value":{  
                                    "val":"SANDBOX"
                                },
                                "name":"SETUP"
                            }
                        }
                    ],
                    "index":1
                }
            ]
        }
    }
}'

Setting API Version

Use th following annotation to use new features, introduced in certain version of Avi API by setting the version to correct value.

avi_proxy: '{"version": "17.2.11"}'

Adding a Virtual Service DataScript <WIP>

For attaching a Datascript to a virtual service, the DataScript is first configured on the Avi Controller UI. The UUID of the corresponding virtual service DataScript is queried from the CLI. The index refers to the order of listing of DataScripts configured on the virtual service. The following annotation attaches a single DataScript that is configured on the Avi Controller to the virtual service:


avi_proxy: '{
"Virtualservice":
{“Vs_datascripts”: [
{“index”: “0”, 
“Vs_datascript_set_ref”:“vsdatascriptset-1f7d1cb6-1b4b-4c29-bf80-028ee1c513f4”
}]
},
  "version": "17.2.11"}'

Sample Configuration Having Combination of Multiple Annotations

Above individual annotations can be used in combination to achieve desired virtual service configuration as shown in below examples:

The following annotation creates a virtual service with the following attributes:

  • Placement set as North-South
  • Listening on port 443 with SSL enabled
  • Using certificates as Custom-EC-Cert and Custom-RSA-Cert certificates
  • Metrics or logs collection turned on
  • Load balancing algorithm set to Round Robin
avi_proxy: '{
    "virtualservice":{
        "east_west_placement":false,
        "application_profile_ref":"/api/applicationprofile?name=System-Secure-HTTP",
        "ssl_profile_ref":"/api/sslprofile/?name=System-Standard",
        "analytics_policy":{
            "client_insights":"NO_INSIGHTS",
            "metrics_realtime_update":{
                "duration":0,
                "enabled":true
            },
            "full_client_logs":{
                "duration":0,
                "enabled":true
            }
        },
        "services":[
            {
                "enable_ssl":true,
                "port":443
            }
        ],
        "ssl_key_and_certificate_refs":[
            "/api/sslkeyandcertificate?name=Custom-EC-Cert",
            "/api/sslkeyandcertificate?name=Custom-RSA-Cert"
        ]
    },
    "pool":{
        "lb_algorithm":"LB_ALGORITHM_ROUND_ROBIN"
    }
}'

Building Annotations from Avi Objects

Most commonly used Avi Objects can be viewed through the OpenAPI UI (Swagger) exposed by the Avi Controller at the page below. The Objects viewed here will conform to the Object model used in the software version running on the Controller. This page also lists the Object Model in the last section: https:///swagger/</code>

The Objects, Object Model and the APIs for each version are also listed in the following guides:

Note: Use the documents for the version as referenced by the version object in the annotation.

Steps to build an annotation

  1. Use the above documents to navigate to the definition of the main object.
  2. Navigate further to list all the required sub-objects.
  3. Refer to the Object types and categories to get the required syntax for the annotation.
  4. Build the annotation and assign the custom values to the Objects.

Notes

  • Sub-objects are referenced inside nested curly braces “{}“ and comma-separated from other sub-objects of the same parent object.
  • For an Object category listed as “repeated”, additional square braces are required since this refers to an Object that can have multiple values (an array or a list of values). For example, the vip Object or discovered_networks object under virtual service require square braces, even if only one VIP or discovered network is configured as shown below:

    The below screenshot exhibits the Category field set as repeated.

repeated

Annotation snippet showing usage of vip with “[ ]”:

vip

  • Refer to specific examples found in the pages documenting each high level object like virtual service, pool, etc: