Setting up Routing Rules using CRDs
Custom Resource Definitions (CRDs) are used to extend the Kubernetes APIs server with additional schemas. To know more, click here.
The Avi Kubernetes Operator (AKO) supports some CRD objects (installed through helm). The CRDs are relevant to:
Users of this category:
- Are aware of the Avi-related semantics
- Have access to the Avi controller
- Manage the lifecycle of AKO
Users of this category:
- Are owners of microservices deployed in Kubernetes
- Are assumed to know basic routing principles but may not know the specifics of Avi attributes
Advantages of CRDs
Some load balancers allow configuration options via annotations.
The following are the advantages of using CRDs:
Versioning: CRDs allow AKO to version fields appropriately due to the dependency on the Avi Controller Versions. In general, this helps preserve unique states across various deployment versions.
Syntactical Validations: CRDs can be used to verify syntax at the time of creation of the CRD object. This saves API cost and facilitates quicker feedback using a combination of field constraints and effective status messages.
Role segregation: CRDs can benefit from the Role-based access control (RBAC) policies of Kubernetes and allow stricter access to a group of users.
Types of CRDs Supported in AKO
AKO categorizes the CRDs into:
Layer 7: These CRD objects are used to express layer 7 traffic routing rules.
Infrastructure: These CRD objects are used to control Avi’s infrastructure components like Ingress class, SE group properties etc.
Layer 7 CRDs
The following are the Layer 7 CRDs currently available in AKO:
The Operator is the primary users of the HostRule CRD. The CRD is used to express additional virtual host properties. The virtual host FQDN is matched from either the Kubernetes ingress or OpenShift route-based objects.
A sample HostRule CRD looks as shown below:
apiVersion: ako.vmware.com/v1alpha1 kind: HostRule metadata: name: my-host-rule namespace: red spec: virtualhost: fqdn: foo.region1.com # mandatory fqdnType: Exact enableVirtualHost: true tls: # optional sslKeyCertificate: name: avi-ssl-key-cert type: ref alternateCertificate: name: avi-ssl-key-cert2 type: ref sslProfile: avi-ssl-profile termination: edge gslb: fqdn: foo.com includeAliases: false httpPolicy: policySets: - avi-secure-policy-ref overwrite: false datascripts: - avi-datascript-redirect-app1 wafPolicy: avi-waf-policy applicationProfile: avi-app-ref analyticsProfile: avi-analytics-ref errorPageProfile: avi-errorpage-ref analyticsPolicy: # optional fullClientLogs: enabled: true throttle: HIGH logAllHeaders: true tcpSettings: listeners: - port: 8081 - port: 6443 enableSSL: true loadBalancerIP: 10.10.10.1 aliases: # optional - bar.com - baz.com
Usage of HostRule
HostRule CRD can be created in a given namespace where the operator requires better control. This section explains the details and associated rules of using each field of the HostRule CRD.
HostRule to Virtual Service Matching Using FQDN/FQDNType
A given HostRule is applied to a virtual service if the virtual service hosts the FQDN mentioned in the HostRule CRD. This FQDN must exactly match the one the virtual service is hosting. However, to simplify the user experience and provide an easy way to apply a HostRule to an individual or multiple virtual services, the FQDNType field can be employed. It provides great flexibility when it comes to specifying the string in the FQDN field.
With the FQDN Type, one of the three matching criteria listed below can be specified:
- Exact: Matches the string character by character to the virtual service FQDNs in an exact match fashion.
- Wildcard: Matches the string to multiple virtual service FQDNs and matches the FQDNs with the provided string as the suffix. The string must start with a * to qualify for wildcard matching.
fqdn: *.alb.vmware.com fqdnType: Wildcard
- Contains: Matches the string to multiple virtual service FQDNs and matches the FQDNs with the provided string as a substring of any possible FQDNs programmed by AKO.
fqdn: Shared-VS-L7-1 fqdnType: Contains
fqdnType field defaults to
Enable/Disable Virtual Host
HostRule CRD can be used to enable/disable corresponding virtual services created by AKO on Avi. This removes any virtual host related configuration from the data plane (Avi Service Engines) in addition to disabling traffic on the virtual host/FQDN.
This property can be applied only for secure FQDNs and cannot be applied for insecure routes. The default value is true.
Express httppolicyset Object Refs
HostRule CRD can be used to express
Note: The httppolicyset objects should be pre-created in the Avi Controller.
httpPolicy: policySets: - "avi-secure-policy-ref" overwrite: false
httppolicyset currently is only applicable for secure FQDNs and cannot be applied for insecure routes. The order of evaluation of the
httppolicyset rules is in the same order they appear in the CRD definition. The list of
httppolicyset rules is always intepreted as an
AKO currently uses
httppolicyset objects on the SNI virtual services to route traffic based on host/path matches. These rules are always at a lower index than the
httppolicyset objects specified in the CRD object. In case a user would want to overwrite all
httppolicyset objects on a SNI virtual service with the ones specified in the HostRule CRD, the overwrite flag can be set to true. The default value for overwrite is
Express WAF Policy Object Refs
HostRule CRD can be used to express WAF policy references.
Create the WAF policy object in the Avi Controller prior to the CRD creation as follows:
Note: This property can be applied only for secure FQDNs and cannot be applied for insecure routes. WAF policies are useful when deep layer 7 packet filtering is required.
Express Custom Application Profiles
HostRule CRD can be used to express application profile references.
Create the application profile reference in the Avi Controller prior to the CRD creation. The application profile should be of Type
Note: This property can be applied only for secure FQDNs and cannot be applied for insecure routes. The application profiles can be used for various HTTP/HTTP2 protocol settings.
Express Custom Analytics Profiles
HostRule CRD is used to express analytics profile references.
Create the analytics profile reference in the Avi Controller prior to this CRD creation.
Note: This property can be applied only for secure FQDNs and cannot be applied for insecure routes. The analytics profiles can be used for various Network/HTTP/Health score analytics settings, log processing, etc.
Express Custom Error Page Profiles
HostRule CRD can be used to express error page profile references.
Create the error page profile reference in the Avi Controller prior to this CRD creation.
Note: This property can be applied only for secure FQDNs and cannot be applied for insecure routes. The error page profiles can be used to send a custom error page to the client generated by the proxy.
HostRule CRD can be used to express error DataScript references.
Create the DataScript references in the Avi Controller prior to this CRD creation.
datascripts: - avi-datascript-redirect-app1
Note: This property can be applied only for secure FQDNs and cannot be applied for insecure routes. The DataScripts can be used to apply custom scripts to data traffic. The order of evaluation of the DataScripts is in the same order they appear in the CRD definition.
Express TLS Configuration
For the Avi Kubernetes operator to control the TLS termination from a privileged namespace, the HostRule CRD can be created in such a namespace.
tls: sslKeyCertificate: name: avi-ssl-key-cert type: ref alternateCertificate: name: avi-ssl-key-cert2 type: ref sslProfile: avi-ssl-profile termination: edge
Here, name refers to an Avi object with the Type as ref. Alternatively, we also support a kubernetes Secret to be specified where the
sslkeyandcertificate object is created by AKO using the Secret.
tls: sslKeyCertificate: name: k8s-app-secret type: secret termination: edge
Note: Currently, only Edge is supported as the type of termination.
alternateCertificate option is provided in case the application needs to be configured to provide multiple server certificates, typically when trying to configure both RSA and ECC signed certificates. The NSX Advanced Load Balancer Controller allows a virtual service to be configured with two certificates at a time, one each of RSA and ECC. This enables Avi Controller to negotiate the optimal algorithm or cipher with the client. If the client supports ECC, in that case the ECC algorithm is preferred, and RSA is used as a fallback in cases where the clients do not support ECC.
sslProfile, additionally, can be used to determine the set of SSL versions and ciphers to accept for SSL/TLS terminated connections. If the
sslProfile is not defined, AKO defaults to the
System-Standard-PFS defined in NSX Advanced Load Balancer.
Configure GSLB FQDN
gslb: fqdn: foo.com
This additional FQDN inherits all the properties of the root FQDN specified under the the virtualHost section. Use this flag if you would want traffic with a GSLB FQDN to get routed to a site local FQDN. For example, in the above CRD, the client request from a GSLB DNS will arrive with the host header as foo.com to the VIP hosting foo.region1.com in region1. This CRD property would ensure that the request is routed appropriately to the backend service of foo.region1.com
This knob is currently supported only with the SNI model and not with Enhanced Virtual Hosting model.
includeAliases is used by AMKO. Whenever a GSLB FQDN is provided, and the
useCustomGlobalFqdn is set to true in AMKO, a GSLB Service is created for the GSLB FQDN instead of the local FQDN(hostname). For more information, see Deriving GslbService FQDNs.
When this flag is set to false, the Domain Name of the GSLB Service is set to the GSLB FQDN.
When this flag is set to true in addition to the GSLB FQDN, AMKO adds the FQDNs mentioned under aliases to domain names of the GSLB Service.
Configure Aliases for FQDN
aliases: - bar.com - baz.com
This list of FQDNs inherits all the properties of the root FQDN specified under the
virtualHost section. Traffic arrives with the host header as
bar.com to the VIP hosting
foo.region1.com and this CRD property would ensure that the request is routed appropriately to the backend service of
Aliases field must contain unique FQDNs and must not contain GSLB FQDN or the root FQDN. Users must ensure that the
fqdnType is set as Exact before setting this field.
Configure Analytics Policy
The HostRule CRD can be used to configure analytics policies such as enable/disable non-significant logs, throttle the number of non-significant logs per second on each SE, enable/disable logging of all headers, etc.:
analyticsPolicy: fullClientLogs: enabled: true throttle: HIGH logAllHeaders: true
The throttle will be in effect only when enabled is set to true. The possible values of throttle are DISABLED (0), LOW (50), MEDIUM (30) and HIGH (10).
AKO sets the duration of logging the non-significant logs to infinity by default.
It is recommended to disable the non-significant logs when it is no longer required.
Configure TCP Settings
The TCP Settings section is responsible for configuring Parent virtual service specific parameters using the HostRule CRD. The
tcpSettings block, in addition to any other parameters provided in the HostRule, is only applied to Parent VSes and dedicated VSes. The
tcpSettings block does not have any effect on child VSes.
In order to consume TCP setting configurations for parent VSes, the HostRule must be matched to a Shared/Dedicated VS FQDN, using the existing fqdn field in HostRule. Where dedicated virtual services are created corresponding to a single application, Shared virtual services host multiple application FQDNs. Therefore, in order to apply a
HostRule to a dedicated VS, users can simply provide the application FQDN in the
HostRule fqdn field. For Shared virtual services however, users can either provide the AKO programmed shared virtual service FQDN, or utilize the
fqdnType: Contains parameter with the Shared virtual service name itself.
fqdn: foo.com # dedicated VS fqdnType: Exact tcpSetting: listeners: - port: 6443 enableSSL: true fqdn: Shared-VS-L7-1.admin.avi.com # AKO configured Shared VS fqdn fqdnType: Exact tcpSetting: loadBalancerIP: 10.10.10.1 fqdn: Shared-VS-L7-1 # bound for clusterName--Shared-VS-L7-1 fqdnType: Contains tcpSetting: loadBalancerIP: 10.10.10.1
In order to overwrite the ports opened for the virtual services created by AKO, users can provide the port details under the listeners setting. The ports mentioned under this section overwrites the default open ports, 80 and 443 (SSL enabled). This is applicable only for Shared or Dedicated virtual services.
tcpSettings: listeners: - port: 80 - port: 8081 - port: 6443 enableSSL: true
Note: At least one of the ports that are mentioned in the setting must have the
enableSSL field set to true.
L7 Static IP
loadBalancerIP field can be used to provide a valid, preferred IPv4 address for L7 virtual services created for the Shared or Dedicated virtual service. The preferred IP must be part of the IPAM configured for the Cloud, and must not overlap with any other IP addresses already in use. In case of misconfigurations, AKO fails to configure the virtual service appropriately throwing and ERROR log for the same.
tcpSettings: loadBalancerIP: 10.10.10.199
Note: The HostRule CRD is not aware of the misconfigurations while it is being created, therefore the HostRule will be Accepted nonetheless.
The status messages are used to give instant feedback about the reference objects specified in the HostRule CRD.
Following are some of the sample status messages:
Accepted HostRule Object
$ kubectl get hr NAME HOST STATUS AGE secure-waf-policy foo.avi.internal Accepted 3d3h
A HostRule is accepted only when all the reference objects specified inside it exist in the Avi Controller.
A Rejected HostRule Object
$ kubectl get hr NAME HOST STATUS AGE secure-waf-policy-alt foo.avi.internal Rejected 2d23h
The reason for rejection can be obtained from the status:
status: error: duplicate fqdn foo.avi.internal found in default/secure-waf-policy-alt status: Rejected
Converting Insecure FQDNs to Secure
The HostRule CRD can be used to convert an insecure host FQDN to a secure one. This is done by specifying a TLS section in the CRD object. The
sslKeyCertificate is provided for the FQDN, will override all
sslkeyandcertificates generated for the FQDN. This is useful if:
The operator wants to convert an insecure ingress FQDN to secure.
The operator wants to override any existing secrets for a given host FQDN and define TLS termination semantics.
If the ingress object specifies a Secret for SNI termination and the HostRule CRD also specifies a sslKeyCertificate for the same
virtualhost, then the
sslkeycertificate in the HostRule CRD will take precedence over the Secret object associated with the Ingress.
If a HostRule is deleted, all the settings for the FQDNs are withdrawn from the Avi controller.
A HostRule CRD is only admitted if all the objects referenced in it, exist in the Avi Controller. If after admission the object references are deleted out-of-band, then AKO does not re-validate the associated HostRule CRD objects. The user needs to manually edit or delete the object for new changes to take effect.
Duplicate FQDN rules
Two HostRule CRDs cannot be used for the same FQDN information across namespaces. If AKO finds a duplicate FQDN in more than one HostRules, AKO honors the first HostRule that gets created and rejects the others. In case of AKO reboots, the CRD that gets honored might not be the same as the one honored earlier.
The developers are the primary users of the HTTPRule CRD. The path matching rules in the ingress or route objects define traffic routing rules to the microservices. The HTTPRule CRD can be used as a complimentary object to control additional layer 7 properties like algorithm, hash, and tls re-encrypt use cases.
A sample HTTPRule object is as shown below:
apiVersion: ako.vmware.com/v1alpha1 kind: HTTPRule metadata: name: my-http-rule namespace: purple-l7 spec: fqdn: foo.avi.internal paths: - target: /foo healthMonitors: - my-health-monitor-1 - my-health-monitor-2 loadBalancerPolicy: algorithm: LB_ALGORITHM_CONSISTENT_HASH hash: LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS tls: ## This is a re-encrypt to pool type: reencrypt # Mandatory [re-encrypt] sslProfile: avi-ssl-profile destinationCA: |- -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE-----
Note: The HTTPRule only applies to paths in the Ingress/Route objects which are specified in the same namespace as the HTTPRule CRD.
Usage of the HTTPRule CRD
The HTTPRule CRD does not have any Avi specific semantics. Hence you are free to express your preferences using this CRD without any knowledge of the Avi objects. Each HTTPRule CRD must be bound to a FQDN (both secure or insecure) to subscribe to rules for a specific hostpath combinations.
Express Load Balancer Algorithm
The load balancer policies are a predefined set of values to choose from.
Currently, the following values are supported for load balancer policy:
To configure the load balancer policy for a given ingress path,
- target: /foo loadBalancerPolicy: algorithm: LB_ALGORITHM_FEWEST_SERVERS
This rule is applied all paths matching
/foo and subsets of
To know more, refer to the Load Balancing Algorithm article.
hash field is used when the algorithm is selected as
LB_ALGORITHM_CONSISTENT_HASH. Otherwise, it is not applicable. Similarly, a
hostHeader field is used only when the hash is selected as
A sample setting with these fields is shown below:
- target: /foo loadBalancerPolicy: algorithm: LB_ALGORITHM_CONSISTENT_HASH hash: LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER hostHeader: foo
hostHeader is disregarded if it is specified in any other case. The hash field is disregarded if the algorithm is not
Express Application Persistence Profile
HTTPRule CRD can be used to express application persistence profile references. Create the application persistence profile reference in the Avi Controller prior to this CRD creation.
The application persistence profile can be used to maintain stickiness to a server instance based on cookie values, headers, and so on for a desired duration of time.
Express Health Monitors
The HTTPRule CRD can be used to express health monitor references.
Create the health monitor reference in the Avi Controller prior to this CRD creation.
To express health monitor references, use:
healthMonitors: - my-health-monitor-1 - my-health-monitor-2
The health monitors can be used to verify server health. A server (Kubernetes pods in this case) is marked as UP only when all the health monitors return successful responses. Health monitors provided here overwrite the default health monitor configuration set by AKO, that is, System-TCP for HTTP/TCP traffic and System-UDP for UDP traffic based on the ingress/service configuration.
Reencrypt Traffic to the Services
While AKO can terminate TLS traffic, it also provides an option where the users can choose to re-encrypt the traffic between the Avi SE and the backend application server. The following option is provided for reencrypt one is by providing a raw certificate using
destinationCA or by providing an Avi PKI Profile reference using the pkiProfile field:
tls: ## This is a re-encrypt to pool type: reencrypt # Mandatory [re-encrypt] sslProfile: avi-ssl-profile destinationCA: |- -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE----- tls: ## This is a re-encrypt to pool type: reencrypt # Mandatory [re-encrypt] sslProfile: avi-ssl-profile pkiProfile: avi-pki-profile
sslProfile, additionally, can be used to determine the set of SSL versions and ciphers to accept for SSL/TLS terminated connections. If the
sslProfile is not defined, AKO defaults to sslProfile
System-Standard defined in Avi.
In case of
reencrypt, if the
destinationCA is specified in the HTTP Rule CRD, as shown in the example, a corresponding PKI profile is created for that pool (host path combination). Also Note that only one of PKI profile or destination CA can be provided to configure
reencrypt for a pool corresponding to the host path backend Service.
The status messages are used to give instant feedback on whether a HTTPRule CRD was accepted or rejected.
Example of a HTTP Rule
$ kubectl get httprule NAME HOSTRULE STATUS AGE my-http-rules default/secure-waf-policy Accepted 5h34m
Avi Infra Setting
Avi Infra Setting provides a way to segregate Layer-4/Layer-7 virtual services to have properties based on different underlying infrastructure components, like Service Engine Group, intended VIP Network etc.
A sample Avi Infra Setting is as shown below:
apiVersion: ako.vmware.com/v1alpha1 kind: AviInfraSetting metadata: name: my-infra-setting spec: seGroup: name: compact-se-group network: vipNetworks: - networkName: vip-network-10-10-10-0-24 cidr: 10.10.10.0/24 v6cidr: 2002::1234:abcd:ffff:c0a8:101/64 nodeNetworks: - networkName: node-network-10-10-20-0-24 cidrs: - 10.10.20.0/24 enableRhi: true bgpPeerLabels: - peer1 - peer2 l7Settings: shardSize: MEDIUM
Avi Infra Setting with Services, Ingress or Routes
Avi Infra Setting is a cluster scoped CRD and can be attached to the intended Services, Kubernetes Ingresses and OpenShift Routes as shown below.
Attaching Avi Infra Setting to Services
Avi Infra setting resources can be attached to Services using Gateway APIs or simply by using annotations.
Using Gateway API
Gateway APIs provide interfaces to structure Kubernetes service networking.
AKO supports Gateway APIs via the
servicesAPI flag in the
values.yaml. For more information on how AKO integrates with Gateway API, see Gateway and Gateway Class.
The Avi Infra Setting resource can be attached to a Gateway Class object, via the
.spec.parametersRef as shown below:
apiVersion: networking.x-k8s.io/v1alpha1 kind: GatewayClass metadata: name: avi-gateway-class spec: controller: ako.vmware.com/avi-lb parametersRef: group: ako.vmware.com kind: AviInfraSetting name: my-infrasetting
In case the
servicesAPI flag is not set to
True, and AKO is not watching over the Gateway APIs, Services of Type
LoadBalancer can specify the Avi Infra Setting using an annotation as shown below:
annotations: aviinfrasetting.ako.vmware.com/name: "my-infrasetting"
Attaching Avi Infra Setting to Ingress
Avi Infra Settings can be applied to Ingress resources, using the
IngressClass provides a way to configure Controller-specific load balancing parameters and applies these configurations to a set of Ingress objects. AKO supports listening to
IngressClass resources in Kubernetes version 1.19+. The Avi Infra Setting reference can be provided in the
Ingress Class as shown below:
apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: avi-ingress-class spec: controller: ako.vmware.com/avi-lb parameters: apiGroup: ako.vmware.com kind: AviInfraSetting name: my-infrasetting
Attaching Avi Infra Setting to Routes
Avi Infra Settings with OpenShift Routes, via the annotation shown below:
annotations: aviinfrasetting.ako.vmware.com/name: "my-infrasetting"
Using Avi Infra Setting
Configure Service Engine Group
AviInfraSetting CRD can be used to configure Service Engine Groups for virtual services created corresponding to Services/Ingresses/ Routes.
Ensure the Service Engine Group is created and configured in the Avi Controller prior to this CRD creation.
seGroup: name: compact-se-group
AKO tries to configure labels on the SEG specified in the AviInfraSetting resources, which enables static route syncing on the member SEs. AKO configures the labels only when the SEGs do not have any other labels configured already. In case AKO finds the SEG configured with some different labels, the AviInfraSetting resource would be Rejected.
Note: Once deployed, the member SEs will not reflect any changes related to label additions or updates on the SEGs.
Therefore, label based static route syncing will not work on already deployed SEs. Ensure that the SEGs have no member SEs deployed before specifying the SEG in the AviInfraSetting resource to properly configure static route syncing.
Configure VIP Networks
AviInfraSetting CRD can be used to configure VIP networks for virtual services created corresponding to Services/Ingresses/Openshift Routes.
Ensure the network is present in the Avi Controller prior to this CRD creation.
network: vipNetworks: - networkName: vip-network-10-10-10-0-24 cidr: 10.10.10.0/24
Note: Multiple network names can be added to the CRD (only in case of AWS cloud). The Avi
virtualservices flag will acquire a VIP from each of these specified networks. If Avi fails to allocate a VIP due to IP exhaustion, this results in complete failure of entire request. This is same as vip allocation failures in single VIP.
AKO 1.5.1 updates the schema to provide VIP network information in AviInfraSetting CRD. For further details, refer to the Upgrade Notes.
Configure Pool Placement Networks
network: nodeNetworks: - networkName: node-network-10-10-20-0-24 cidrs: - 10.10.20.0/24
If two Kubernetes clusters have overlapping Pod CIDRs, the service engine needs to identify the right gateway for each of the overlapping CIDR groups. This is achieved by specifying the right placement network for the pools that helps the Service Engine place the pools appropriately.
Enable/Disable Route Health Injection
Avi Infra Setting CRD is used to enable/disable Route Health Injection (RHI) on the virtual services created by AKO.
network: enableRhi: true
This overrides the global
enableRHI flag for the virtual services corresponding to the Avi Infra Setting.
Enable/Disable Public IP
The AviInfraSetting CRD can be used to enable/disable Public IP on the virtual services created by AKO.
network: enablePublicIP: true
Note: Enabling Public IP is only supported for public clouds.
Configure BGP Peer Labels for BGP Virtual Services
AviInfraSetting CRD can be used to configure BGP Peer labels for BGP virtual services. AKO configures the virtual sevices with the appropriate peer labels, only when
enableRHI is enabled, either during AKO installation via helm chart’s values.yaml or via the AviInfraSetting CRD itself. If not set to true, the AviInfraSetting resource will be marked Rejected.
bgpPeerLabels: - peer1 - peer2
Use Dedicated VIP for Ingress
AviInfraSetting CRD can be used to allocate a dedicated vip per Ingress FQDN.
l7Settings: shardSize: DEDICATED
For the subset of ingresses, that refer to an ingress class which in turn refers to an AviInfraSetting CRD setting that has shardSize as DEDICATED, will get VIP per Ingress FQDN.
Note: The value DEDICATED is not supported when AviInfra CRD is applied to the passthrough route/ingress.
Avi Infra Setting CRD can be used to enable IPv6, IPv4 or both IPv4 and IPv6 VIPs on the virtual services created by AKO.
Note: This feature is currently under Tech-Preview.
network: vipNetworks: - networkName: vip-network-10-10-10-0-24 cidr: 10.10.10.0/24 v6cidr: 2002::1234:abcd:ffff:c0a8:101/64
Document Revision History
|September 26, 2022||Updated Aviinfrasetting support in passthrough virtual service|
|April 31, 2021||Added the sections for Configure VIP Networks, Enable/Disable Public IP, and Configure BGP Peer Labels for BGP Virtual Services supported in AKO version 1.5.1|
|April 28, 2021||Added the section for Avi Infra Setting CRD, supported in AKO version 1.4.1|
|December 18, 2020||Updated the CRDs supported in AKO version 1.3.1|
|July 22, 2020||Published the article for Custom Resource Definitions|