Avi Vantage Annotation Guide

Overview

Avi vantage synchronizes the Kubernetes or OpenShift service, ingress and route configurations, and auto-provision the virtual service. Avi Annotations can be used to customize various configuration requirements. This guide explains the annotation examples for the following configuration requirements:

Prerequisites

Cloud installation should be performed as required using the normal procedure as mentioned in the installation guide.

Annotation Format

The values of annotation are in JSON format. While specifying it in service, ingress or route specs, Kubernetes/OpenShift treats it as strings (within single quotes). i.e. the annotations are in this format:

avi_proxy: 'string specifying avi configuration'

The annotation can be specified in a single line (without indentations) if required. But this may become difficult to read and edit as seen below:

kind: Service
apiVersion: v1
metadata:
  name: avisvc
  labels:
  svc: avisvc
  annotations:
  avi_proxy: '{"virtualservice": {"services": [{"port": 443, "enable_ssl": true}], "east_west_placement": false, "ssl_key_and_certificate_refs": ["/api/sslkeyandcertificate?name=Custom-EC-Cert"], "ssl_profile_ref": "/api/sslprofile/?name=System-Standard"}}'
spec:
  ports:
    - name: https
      port: 443
      targetPort: http
  selector:
    name: avitest

Also if the service, ingress or route specification is in JSON format, escape the double quotes inside the avi_proxy value as shown below, so that the value is read as a string.

{
    "kind":"Service",
    "apiVersion":"v1",
    "metadata":{
        "name":"avisvc",
        "labels":{
            "svc":"avisvc"
        },
        "annotations":{
            "avi_proxy":"{\\"virtualservice\\": {\\"services\\": [{\\"port\\": 443, \\"enable_ssl\\": true}], \\"east_west_placement\\": false, \\"ssl_key_and_certificate_refs\\": [\\"avisvccert\\"], \\"ssl_profile_ref\\": \\"/api/sslprofile/?name=System-Standard\\"}}"
        }
    },
    "spec":{
        "ports":[
            {
                "name":"https",
                "port":443,
                "targetPort":"http"
            }
        ],
        "selector":{
            "name":"avitest"
        }
    }
}

Annotations for Virtual Service Configuration

Annotations can be used for customizing the following configuration requirements for a virtual service:

Enabling Real Time Metrics and Log Collection

The annotation used for real time metrics and log collection enables, and sets the duration to collect real time metrics and logs (in minutes). The following are the parameters used in the analytics policy:

  • client_insights: This value can be set to NO_INSIGHTS, PASSIVE, or ACTIVE.
  • duration: Setting the duration to 0 enables the collection indefinitely. For any other integer value, the enabled flag is set to false after the set duration.
  • enabled: This value can se set either true or false.

If analytics_policy is not specified, and the value of client_insights is set to NO_INSIGHTS, then real time metrics and logs collection is disabled. The following example exhibits the configuration when the value of client_insights is set to NO_INSIGHTS and the duration is set to O.

avi_proxy: '{
    "virtualservice":{
        "analytics_policy":{
            "client_insights":"NO_INSIGHTS",
            "metrics_realtime_update":{
                "duration":0,
                "enabled":true
            },
            "full_client_logs":{
                "duration":0,
                "enabled":true
            }
        }
    }
}'

Setting Application Profile

The following annotation sets the application profile of the virtual service to Custom-HTTP-Profile. The Custom-HTTP-Profile is the profile which was previously created with required HTTP configurations. Use the following syntax to set the application profile to the desired one. application_profile_ref": "/api/applicationprofile?name=Custom-HTTP-Profile. Replace Custom-HTTP-Prole with the profile name that you want to use.

If the profile name is not specified, then the following type of default application profile is set:

  • System-L4-Application (if SSL is not enabled)
  • System-SSL-Application (if SSL is enabled)
avi_proxy: '{
    "virtualservice": {
        "application_profile_ref": "/api/applicationprofile?name=Custom-HTTP-Profile"
    }
}'

Note: The application profile is set to System-HTTP if the service port or container port matches the port number set in the port matching configuration in the associated cloud.

Overwriting Domain Name

The following annotation configures the domain name for a virtual service. The FQDN should belong to the sub domain configured in the DNS profile. If not specified, the default FQDN is set to ... The following is the syntax for entering sub domain name: "fqdn": "domain name"

In the following example, svc1.avi.local is the domain in use.

avi_proxy: '{
    "virtualservice":{
        "fqdn": "svc1.avi.local"
    }
}'

Disabling Virtual Service Creation

The following annotation will disable creation of a virtual service for the Kubernetes/OpenShift service. A pool object containing the application pods will still be created but will not be associated with any of the virtual services. This can then be used for blue/green upgrades, canary testing, etc.

"proxy_listener":"no"

Note: Do not use this with “avi_proxy” annotation.

Annotations for Routing or Networking

The following networking or routing related configurations can be customized using annotations:

Setting Static VIP

A static VIP can be manually assigned to a virtual service using the below annotation. The IP address used must be outside the range of IPs allocated for IPAM. The syntax used is "addr":"Desired IP Address" In the below example, the static IP address alloted to a virtual service is 10.130.150.203.

avi_proxy: '{
    "virtualservice":{
        "ip_address":{
            "addr":"10.130.150.203",
            "type":"V4"
        }
    }
}'

Setting VS Placement Type to North-South

The following annotation configures a VIP from North-South IPAM by setting the value of flag "east_west_placement" to false.

avi_proxy: '{"virtualservice": {"east_west_placement": false}}'

The default value of “east_west_placement” is given by the “Default Service as East-West Service” flag in cloud configuration Kubernetes/OpenShift install guide. Use this flag only in Kubernetes or OpenShift Service annotation. Ingress and route objects always create north-south virtual service.

Configuring Floating VIP

The following example creates a North-South virtual service with an auto-allocated VIP (private IP), and a static VIP (Public IP). The “auto_allocate_ip” flag is set explicitly to true when a floating IP is configured. The syntax for setting floating IP address is "floating_ip": {"addr": "Desired IP Address", "type": "V4". In the below example, the floating IP address assigned to a virtual service is 35.225.58.194.

avi_proxy: '{
    "virtualservice": {
         "vip": [{
                   "auto_allocate_ip": true,
                   "floating_ip": {"addr": "35.225.58.194", "type": "V4"},
                   "east_west_placement": false
                }]
                       },
    "version": "17.2.9"
}'

Setting VIP IPAM Network

If IPAM has multiple networks configured, the “network_ref” needs to be specified in the annotation to select the network from which the VIP should be allocated. The following annotation allocates the VIP to the Network1 network.

avi_proxy: '{
    "virtualservice":{
        "network_ref":"/api/network?name=Network1"
    }
}'

This is not required if IPAM has only one network or if any of the listed networks can be used.

Use the following annotation to specify a network for auto-allocation that is NOT listed in the IPAM object: Note: This annotation requires a minimum API version (17.2.4).

avi_proxy: '{
    "version": "17.2.4",
    "virtualservice":{
        "vip": [{
            "auto_allocate_ip": "true",
            "ipam_network_subnet": {
                 "network_ref":"/api/network?name=Network1"
            }
        }]
    }
}'

Advertising VIP using BGP

The following annotation enable VIP advertisements using BGP. Set the value of the enable_rhi flag to true. If the flag value is not specified, the default value is set to false.

avi_proxy: '{
    "virtualservice":{
        "enable_rhi":true
    }
}'

Enabling ECMP

The following annotation enables the network or routing to perform ECMP over the Service Engines on which the virtual service is placed (e.g. in GCP). Set the value of “scaleout_ecmp” flag to true. If not specified, the default value is set to false.

avi_proxy: '{
    "virtualservice":{
        "scaleout_ecmp":true
    }
}'

Note: “scaleout_ecmp”must not be used when “enable_rhi” is set to true.

Overriding Ingress/Route for a Virtual Service

The Use Shared Virtual Service flag in the OpenShift cloud configuration maps all HTTP/HTTPS ingress/route to the parent virtual service. The mapping is done either by using SNI (for HTTPS) or content switching based on Host header (HTTP). The following annotation overrides the existing configuration to create a dedicated virtual service for a particular ingress/route. Set the value of flag dedicate_route to true.

avi_proxy: '{
    "dedicated_route": true
}'

If the value of Use Shared Virtual Service is not set in the cloud configuration, dedicated virtual services are created for all ingresses/routes. The following annotation overrides the configuration to map the ingress/route to the parent virtual service.

avi_proxy: '{
     "dedicated_route": false
 }'

Overriding Shared Ingress/Route for a Namespace

The Use Shared Virtual Service flag in the cloud configuration maps all HTTP/HTTPS ingress/route to the parent virtual service. It mapping is done either using SNI (for HTTPS) or content switching based on host header (HTTP). The following annotation overrides the configuration to create dedicated virtual services for all ingresses/routes in the namespace.

avi_virtualservice: dedicated

If the Use Shared Virtual Service is not set in the cloud configuration, dedicated virtual services are created for all ingress/routes. The following annotation overrides this configuration to map the ingress/route in a namespace to the parent virtual service.

avi_virtualservice: shared

Note: This annotation can be used with Namespace spec. Refer to the links below for more details:

Pool Specific Annotations

Annotations can be used to customize the following configurations for a pool or pool group:

Configuring Load Balancing Algorithm

The following annotation configures the desired load balancing method for the load balancing pool. Supported values are:

  • LB_ALGORITHM_LEAST_CONNECTIONS
  • LB_ALGORITHM_ROUND_ROBIN
  • LB_ALGORITHM_FASTEST_RESPONSE
  • LB_ALGORITHM_CONSISTENT_HASH
  • LB_ALGORITHM_LEAST_LOAD
  • LB_ALGORITHM_FEWEST_SERVERS

The annotation below configures the load balancing method to round robin.

avi_proxy: '{
    "pool":{
        "lb_algorithm":"LB_ALGORITHM_ROUND_ROBIN"
    }
}'

If not specified, the default value for load balancing method is set as LB_ALGORITHM_LEAST_CONNECTIONS.

For the Consistent Hash algorithm, also specify the field to be used for hashing. Supported values for the field are:

  • LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS
  • LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS_AND_PORT
  • LB_ALGORITHM_CONSISTENT_HASH_URI
  • LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER
  • LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_STRING
  • LB_ALGORITHM_CONSISTENT_HASH_CALLID

The example below sets the load balancing method to consistent hash and field to the source IP address.

avi_proxy: '{
    "pool":{
        "lb_algorithm":"LB_ALGORITHM_CONSISTENT_HASH",
        "lb_algorithm_hash":"LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS"
    }
}'

Note: Also specify lb_algorithm_consistent_hash_hdr in annotation for consistent hash using custom header.

Enabling SSL to Back-end Connections

If application pods listen on HTTPS, backend traffic can be re-encrypted. Specifying SSL profile in the pool annotations enabled re-encryption. Use the following annotation to re-encrypt back-end connections.

avi_proxy: '{
    "pool":{
        "ssl_profile_ref":"/api/sslprofile?name=System-Standard"
    }
}'

Setting Traffic Split Ratio

Pool group annotation is used to set a ratio, to send a part of traffic to a different pool. This can be used to perform the graceful blue-green upgrade of the application. The following annotation sets traffic ratio for the svcA pool and the svcB pool as 10:1.

avi_proxy: '{
    "poolgroup":{
        "members":[
            {
                "ratio":10,
                "pool_ref":"/api/pool/?name=svcA-pool-http-tcp"
            },
            {
                "ratio":1,
                "pool_ref":"/api/pool/?name=svcB-pool-http-tcp"
            }
        ]
    }
}'

Security Specific Annotations

The following WAF-related configurations can be customized using annotations:

Overriding Service Port or Enabling SSL

The port specified under “services” will override any service port specified in the Kubernetes/OpenShift configuration. Multiple ports can be configured for a virtual service if required. SSL can be enabled for each port by setting enable_ssl to true. This is set to false by default, if not specified. The following annotation enables SSL on port 443 for the virtual service.

avi_proxy: '{
    "virtualservice":{
        "services":[
            {
                "enable_ssl":true,
                "port":443
            }
        ]
    }
}'

If services annotation is not provided, the virtual service listens on port specified in Kubernetes/OpenShift service spec.

Setting SSL Profile

The following annotation sets the SSL profile of the virtual service to Custom-SSL-Profile. SSL profile is configured only if SSL is enabled on one of the ports associated with the virtual service.

avi_proxy: '{
    "virtualservice":{
        "ssl_profile_ref":"/api/sslprofile/?name=Custom-SSL-Profile"
    }
}'

Note: If SSL profile is not specified, it defaults to System-Standard.

Configuring Certificates

The following annotation configures the SSL certificates for a virtual service. The value is a list which can contain at most one EC cert and one RSA cert only. Any other combination will result in an error and the virtual service will not be created. The annotation below sets the EC certificate as System-Default-Cert-EC, and the RSA certificate as System-Default-Cert.

avi_proxy: '{
    "virtualservice":{
        "ssl_key_and_certificate_refs":[
            "/api/sslkeyandcertificate?name=System-Default-Cert-EC",
            "/api/sslkeyandcertificate?name=System-Default-Cert"
        ]
    }
}'

Use above annotation to use new features, introduced in certain version of Avi API by setting the version to correct value.

Creating WAF Policy

The annotation below creates a new WAF policy for the virtual service. The WAF policy will be named after the virtual service name. Starting with 17.2.4, Avi Vantage supports annotation for WAF.

avi_proxy: '{"waf_policy": true, "version": "17.2.4"}'

Creating Virtual Service with existing WAF Policy

The annotation below adds an existing WAF policy to a virtual service. This can also be used if an existing WAF policy is to be shared among multiple virtual services. Altering the WAF rules affects all virtual services using the policy. The annotation below adds an existing WAF policy – waf-dem-policy to the virtual service .

avi_proxy: '{
    "virtualservice":{
        "waf_policy_ref":"/api/wafpolicy?name=waf-demo-policy"
    },
    "version":"17.2.9"
}'

Miscellaneous Annotations

Adding HTTP Policy

The annotation shown below exhibits an example of an HTTP policy which checks if the SETUP header is present and modifies value of the header from TEST to SANDBOX.

The HTTP policy is valid only if application profile is of HTTP type.

Note: Writing annotation for HTTP policies can be complicated. An easy way is to create the policy using UI and copy the JSON from the REST output of GET call on the virtual service.

avi_proxy: '{
    "virtualservice":{
        "application_profile_ref": "/api/applicationprofile?name=System-HTTP",
        "http_policies": [
            {
                "index": 11,
                "http_policy_set_ref": "/api/httppolicyset?name=Demo-httppolicy"
            }
        ]
    },
    "httppolicyset":{  
        "name":"photo-Default-Cloud-HTTP-Policy-Set",
        "is_internal_policy":false,
        "index":11,
        "http_request_policy":{  
            "rules":[  
                {  
                    "enable":true,
                    "name":"Rule 1",
                    "_toPosition":{  
                        "index":0,
                        "position":"below"
                    },
                    "match":{  
                        "hdrs":[  
                            {  
                                "match_criteria":"HDR_EXISTS",
                                "hdr":"SETUP",
                                "value":[  
                                    "TEST"
                                ]
                            }
                        ]
                    },
                    "hdr_action":[  
                        {  
                            "action":"HTTP_REPLACE_HDR",
                            "hdr":{  
                                "value":{  
                                    "val":"SANDBOX"
                                },
                                "name":"SETUP"
                            }
                        }
                    ],
                    "index":1
                }
            ]
        }
    }
}'

Setting API Version

Use th following annotation to use new features, introduced in certain version of Avi API by setting the version to correct value.

avi_proxy: '{"version": "17.2.11"}'

Sample Configuration Having Combination of Multiple Annotations

Above individual annotations can be used in combination to achieve desired virtual service configuration as shown in below examples:

The following annotation creates a virtual service with the following attributes:

  • Placement set as North-South
  • Listening on port 443 with SSL enabled
  • Using certificates as Custom-EC-Cert and Custom-RSA-Cert certificates
  • Metrics or logs collection turned on
  • Load balancing algorithm set to Round Robin
avi_proxy: '{
    "virtualservice":{
        "east_west_placement":false,
        "application_profile_ref":"/api/applicationprofile?name=System-Secure-HTTP",
        "ssl_profile_ref":"/api/sslprofile/?name=System-Standard",
        "analytics_policy":{
            "client_insights":"NO_INSIGHTS",
            "metrics_realtime_update":{
                "duration":0,
                "enabled":true
            },
            "full_client_logs":{
                "duration":0,
                "enabled":true
            }
        },
        "services":[
            {
                "enable_ssl":true,
                "port":443
            }
        ],
        "ssl_key_and_certificate_refs":[
            "/api/sslkeyandcertificate?name=Custom-EC-Cert",
            "/api/sslkeyandcertificate?name=Custom-RSA-Cert"
        ]
    },
    "pool":{
        "lb_algorithm":"LB_ALGORITHM_ROUND_ROBIN"
    }
}'