Avi Vantage 18.2.X Release Notes

Issues Resolved in 18.2.3 Patch Releases

Issues Resolved in 18.2.3-4p1

  • AV-62198: Avi Controller will send both avi_session_id and session_id again in the REST API response
  • AV-62702: Virtual service creation or update fails in public clouds if enable_rhi flag is set to False

Issues Resolved in 18.2.3-3p1

  • AV-61720: vCenter discovery not proceeding when a VM’s vNIC was attached to a portgroup which did not have read permission for the user
  • AV-61769: Infoblox issued duplicate IPs for VIPs with the same name/port
  • AV-61875: Some of the Service Engines remain in partitioned state if both the leader and follower Controller nodes are rebooted at the same time
  • AV-62309: Allow SSL key and certificate object to be shared from the admin tenant

Issues Resolved in 18.2.3-2p1

  • AV-61875: Some of the Service Engines can remain in partitioned state if both the leader and a follower Controller node are rebooted at the same time
  • AV-62163: Health status sync between GSLB Sites fails after upgrading to 18.2.3 as the upgrade site is unable to parse the response because of deprecated fields
  • AV-62309: Allow SSL key and certificate object to be shared from admin tenant

Issues Resolved in 18.2.3-1p1

  • AV-61787: DataScript API avi.http.saml_session_decrypt() to decrypt SAML session cookie
  • AV-61819: Service Engine failure when request with Cookie Header size greater than 4K is sent, in a SAML authenticated session
  • AV-61875: Some of the Service Engines can remain in partitioned state if both the leader and a follower Controller node are rebooted at the same time
  • AV-62053: Configuring SSL profile selectors is not possible for SNI child virtual services when the child virtual service does not have a default SSL profile
  • AV-62163: Health status sync between GSLB Sites fails after upgrading to 18.2.3 as the upgrade site is unable to parse the response because of deprecated fields
  • AV-62256: Limit request and connection memory pool usage

What’s New in 18.2.3

Release date: 2May2019

ADC

Analytics

DataScript

GSLB

  • Support for a different default LB algorithm, in case geolocation fails
  • Support for topology-based load balancing (primary/fallback sites) as a GSLB algorithm, instead of a DNS policy

Security

Containers

Public Cloud

  • AWS: Support for c5n instances
  • AWS: Support Amazon S3 for Controller configuration backups
  • AWS: Support in pool for an autoscaling group which has been created with Launch Template
  • Azure: Support for multiple VIPs in a single virtual service
  • Azure: Ability to override the Service Engine management network specified in cloud configuration, on a per-SE-group basis
  • Azure: Option to select ALB type at SE group level
  • Azure: Optimizations to VM scale set polling mechanism, to reduce API calls to Microsoft Azure

OpenStack

  • Support for multiple networks with same CIDR
  • Support for using port-security option for Neutron OpFlex plugin

Other Ecosystems

System

  • Enhancement to limit frequency of License Expiry emails
  • Support for rotating log files in the /var/log/ directory on the Controller

Issues Resolved in 18.2.3

  • AV-46453: Kubernetes: External IP is not updated when K8s service type is set to LoadBalancer
  • AV-47046: End-to-End timing graphs not displayed
  • AV-47080: Linux server cloud: Service Engine may fail on using multiple bond interfaces to advertise VIP via BGP
  • AV-47181: On logging in as an administrator, default tenant is not set to admin
  • AV-51499: Avi Vantage not caching javascript query URI when */javascript is in string group
  • AV-51582: VIP connectivity is lost when host key-value pair is configured in SE group settings
  • AV-51693: In case of a failure, GSLB health checks are not performed on newly spawned Service Engines
  • AV-52075: Reduction in Service Engine health score due to increased SE disk usage
  • AV-52588: Server inventory response pages not paginated
  • AV-52716: Service Engine failure on pool server reselect if the server is marked down at the same time
  • AV-52722: NSX security groups are not populated in the UI
  • AV-53119: Azure: Controller cluster goes down when the Controller VMs do not get scheduled for some time
  • AV-53365: Incorrect handling of Nagios health monitor requests
  • AV-53395: Azure: Service Engine CPU utilization reported by Avi Vantage is incorrect
  • AV-53448: OpenStack: Neutron APIs timeout in a large deployment
  • AV-53552: Unable to add an exclude_list to the rules for a crs_group in WAF Policy
  • AV-53563: Intermittent requests to AWS pool members fail with “connection closed abnormally: conn deleted due to config update”
  • AV-53816: Incorrect RBAC dependency causes error in Roles edited via the UI
  • AV-53899: SE OVA download failure from the Controller if the Controller is running as a docker container
  • AV-53914: SE failure when Response event DataScript runs in the context of HTTP Response generated by a request event DataScript
  • AV-54003: Autorebalance configuration does not take effect for some service engine groups
  • AV-54008: While using HTTP/2 with caching enabled, application page does not load properly
  • AV-54081: Access to the Controller fails even after ACL preventing the access is removed
  • AV-54109: Unable to update systemconfig with CLI scripting mode
  • AV-54186: Service Engine failure when certificate expires
  • AV-54752: Avi Vantage not acknowledging FIN packets, causing delays
  • AV-54922: Linux server cloud: Failure when IPv6 is configured on the VIP and IPv4 on the pool
  • AV-54931: Service Engine may fail when caching and WAF are enabled on a virtual service
  • AV-55185: Kubernetes in AWS: Virtual service failed to start due to private IP address limit on the SE
  • AV-55343: SE failure when a pool group is configured with redirect fail action with no destination
  • AV-55410: Unexpected BGP flap due to BFD timing out
  • AV-55454: SE Failure for VS with App Type System-SSL-Application when Network Profile type is set to TCP Fast
  • AV-55686: SE_HM_EVENT_SHM_UP events in the logs not preceded by any corresponding DOWN events
  • AV-55775: OpenShift: Multiple SE include/exclude attributes do not work
  • AV-56113: OpenShift on Azure: One SE stuck in OPER_DISABLED mode even though Kubernetes node is Ready state
  • AV-56197: Zone transfer through Avi DNS VS fails after a certain number of records are present
  • AV-56236: Metrics: End-to-end timing graph in Virtual Service Analytics overlay not displayed
  • AV-56495: Modifying the application’s domain name is not propagated to Infoblox DNS/IPAM
  • AV-56528: Avi Vantage UI not showing all the pages ‘select servers from network’ view
  • AV-56625: Fix for high Service Engine Persistence Table Usage
  • AV-56660: Service Engine restarts when applying an Avi Controller patch
  • AV-56674: AWS: Adding more than 200 servers to a pool fails
  • AV-56697: SNMP trap for CONTROLLER_NODE_LEFT is generated as aviSystemAlert rather than aviControllerStatusChanged
  • AV-56734: GSLB: Round robin behavior fails when num_dns_ip is set to 0 and multiple pools have the same priority
  • AV-57344: VIP traffic from an external client fails when OpenShift/Kubernetes clusters have more than 1 NIC and the VIP NIC is not the default gateway interface
  • AV-57616: Failure in metrics APIs for user-defined/custom metrics
  • AV-58101: Service Engine failure due to BGP peer monitoring blocking data path for more than 60 seconds
  • AV-58121: Kubernetes: Any non-error egress pod log also gets dumped to the screen
  • AV-58181: Handle application of IPv6 routes with /48 mask properly
  • AV-58426: Service Engines can fail to connect to the Controller due to a race condition that triggers the cluster services watcher process on the leader node to go into an inconsistent state
  • AV-58446: When the link of physical function flaps, the virtual functions need to send a reset to recover network connectivity
  • AV-58483: HTTP Response Policy is not displayed correctly in Avi Vantage UI
  • AV-58530: External Health Monitor using ldapsearch fails
  • AV-58537: Service Engine fails on GSLB follower site when the leader site pushes an incompatible TCP health monitor
  • AV-58660: Polling for Azure VM scalesets stops if a scaleset is deleted from Azure, without removing it from the Avi Pool
  • AV-58831: SNAT sharing between VSes does not work for legacy HA
  • AV-58886: Service Engine thread gets stuck when momentary access fails in the check for a specific SE pod, causing the SE’s IP resolution to fail and potentially the extra SE object not getting cleaned up
  • AV-58900: AZURE_ACCESS_FAILURE event is not generated if access to Azure APIs fails after the cloud is up
  • AV-58901: Auth Profile cannot be configured using FQDN in System configuration
  • AV-58954: DataScript transform fails when the name of a stringgroup object referred by the DataScript is changed after creation
  • AV-58986: After a Service Engine failure due to a kernel panic, the SE fails to reconnect to the Controller
  • AV-59039: Replication issues between GSLB sites
  • AV-59049: Using underscore in Service Engine group name causes daemonset creation failure in K8s/OC cloud
  • AV-59053: GCP: Malformed URL error when adding route
  • AV-59159: OpenShift: Attribute list in K8s/OC cloud configuration with additional SE groups causes excessive SEs to be spawned
  • AV-59202: Unable to set maintenance code to HTTP health monitor
  • AV-59255: All nodes in Controller markes as “initializing” with service temporarily unavailable
  • AV-59279: Existing Routes/Ingresses can get deleted if there are K8s API server connectivity issues in rare scenarios
  • AV-59388: avi_proxy gslb annotation to update content switch httppolicyset rule under child virtual service with created GSLB FQDN
  • AV-59497: After upgrade to 18.2.2 OpenShift Routes with no Host/Path will not work without explicitly sending a Host Header in the HTTP request as Avi programs a default 404 rule
  • AV-59502: Service Engines stuck in disabled state upon changing SE group CPU/Memory/Disk Size
  • AV-59530: Stale PCI ID-to-name mapping in Linux prevents release of NIC to kernel
  • AV-59542: SE may fail with UDP per pkt virtual service preserving client IP and client port if client reuses the port
  • AV-59639: AWS deployment fails if userdata is not provided
  • AV-59642: VS Placement fails to follow legacy HA tags for VS with shared VIPs sometimes, when all such VSes were disabled and are enabled in any order
  • AV-59647: AWS: When servers are moved to standby in autoscale groups and then terminated, it can cause polling of ASGs to stop
  • AV-59658: While integrating with OpenStack Queens or higher releases, image upload might fail if interoperable image-import feature is enabled in glance service
  • AV-59699: Cisco ACI: Secondary SE may directly send a RST packet instead of tunneling it to the primary causing wrong MAC learning for the VIP
  • AV-59736: Process se_dp on Service Engine crashes when a Virtual Service referencing a shared pool is deleted
  • AV-59922: Updating an ingress annotation with invalid JSON causes the Virtual Service to be deleted
  • AV-60068: Service Engine failure when a parent VS is disabled while there is an existing connection to the child VS and connection multiplexing is disabled
  • AV-60201: Kubernetes ingress annotation does not respect specified version field
  • AV-60256: SE data NIC does not inherit configured security groups on AWS
  • AV-60304: On config restore to new Controller, Service Engines unable to connect back to Controller
  • AV-60460: When connection multiplexing is turned off, the requests coming on the client connection are sent on the backend connection
  • AV-60527: Controller with ipset rules configured does not bring up the eth0 as /etc/network/pre-up.d script is failing
  • AV-60591: Egress pod replication Controller requires additional rights and initContainers in 18.2.2
  • AV-61073: Azure: Update of the pool fails when same IP is being used by another server in different scale set

Known Issues and Workarounds in 18.2.3

  • AV-61294: Uploads to HTTP/2 VIPs can fail in some cases, especially with a combination of a fast client and slow server. It is recommended to disable HTTP/2 on VIPs. This does not affect any file uploads to HTTP/1 VIPs.
  • AV-61380: When Avi Vantage is upgraded from 17.2.x to 18.2.3 on GCP in DPDK mode, the Service Engine loses its management interface when it comes up after the upgrade. The SE can be recovered by rebooting the SE VM after the upgrade.
  • AV-61787: Unable to decrypt SAML session cookie due to the error in the avi.crypto.decrypt API
  • AV-61819: Service Engine fails when a request with cookie header size > 4KB is sent in a SAML-authenticated session
  • AV-61875: Some of Service Engines can remain in partitioned state if both the leader and a follower Controller node are rebooted at the same time
  • AV-62053: Configuring SSL profile selectors is not possible for SNI child virtual services when the child VS does not have a default SSL profile
  • AV-62163: Health status syncing between GSLB sites fails as the upgrade site is unable to parse the response because of deprecated fields
  • AV-62256: Disabled check for the request and connection memory pool usage causes SE crash
  • AV-62702: Virtual service creation or update fails in public clouds if enable_rhi flag is set to False
  • AV-62262: Traffic loss on virtual service caused due to an unsupported user-defined metric in the DataScript
  • AV-62821: For geo load-balancing at GSLB service level, when the distance between the members is smaller compared to the number of members in the pool, then some of the pools are considered to be “equi-distant” from the client, and a different pool than the desired one could be picked

Issues Resolved in 18.2.2 Patch Releases

Issues Resolved in 18.2.2-9p1

  • AV-61345: Add GRATARP support for BGP virtual service

Issues Resolved in 18.2.2-8p2

  • AV-61355: SAML: Service Engine fails when request on an old connection comes in after SSO has been disabled
  • AV-61787: DataScript API avi.http.saml_session_decrypt() to decrypt SAML session cookie
  • AV-61819: Service Engine failure when request with Cookie Header size greater than 4K is sent, in a SAML authenticated session

Issues Resolved in 18.2.2-8p1

  • AV-60068: Service Engine failure when a parent virtual service is disabled while there is an existing connection to the child virtual service and the connection multiplexing is disabled

Issues Resolved in 18.2.2-7p1

  • AV-55775: OpenShift: Multiple SE include/exclude attributes do not work
  • AV-57344: VIP traffic from an external client fails when OpenShift/K8S clusters have more than 1 NIC and the VIP NIC is not the default gateway interface
  • AV-58121: Any non error egress pod log also gets dumped to the screen
  • AV-58886: SE thread gets stuck when momentary access fails in the check for a specific SE pod, causing the SE’s IP resolution to fail and potentially the extra SE object not getting cleaned up
  • AV-59279: Existing routes/ingresses can get deleted if there are K8S API server connectivity issues in rare scenarios
  • AV-59378: Default drop rule for host matching results in 404 for traffic for a route with no host defined
  • AV-59497: After upgrade to 18.2.2 OpenShift routes with no host/path will not work without explicitly sending a host header in the HTTP request as Avi programs a default 404 rule
  • AV-59502: SEs can be stuck in disabled state upon changing SE group CPU/memory/disksize

Issues Resolved in 18.2.2-6p1

  • AV-58900: AZURE_ACCESS_FAILURE event is not generated if access to Azure APIs fails after the cloud is up

Issues Resolved in 18.2.2-5p1

  • AV-58426: Service Engine fails to connect to the Controller triggering issues with cluster service watcher process

Issues Resolved in 18.2.2-4p1

  • AV-59394: Reset connection when client certification validation fails

Issues Resolved in 18.2.2-3p2

  • AV-61073: Azure: Update of the pool fails when same IP is used by another server in different scale set

Issues Resolved in 18.2.2-3p1

  • AV-58660: Polling for Azure VM scalesets stops if a scaleset is deleted from Azure, without being removed from Avi pool

Issues Resolved in 18.2.2-2p1

  • AV-57344: VIP traffic from an external client fails when OpenShift/K8S clusters have more than 1 NIC and the VIP NIC is not the default gateway interface
  • AV-58886: SE thread stuck when momentary access fails for a specific SE pod check causing the SE’s IP resolution to fail and potentially the extra SE object is not cleaned up

Issues Resolved in 18.2.2-1p3

  • AV-61051: Disable PCAP look-ahead logic to bring down CPU utilisation in dispatcher
  • AV-58426: Service Engines can fail to connect to the Controller due to a race condition that triggers the cluster services watcher process on the leader node to go into an inconsistent state that responds to the Service Engine with no active members in the cluster

Issues Resolved in 18.2.2-1p1

  • AV-56674: Adding more than 200 servers to a pool fails on AWS

What’s New in 18.2.2

Release date: 6Mar2019

ADC

Containers

  • OpenShift: Configuration knob to assign FQDNs automatically to a virtual service in OpenShift clouds
  • Kubernetes: Support for egress taints and tolerances in egress pod scheduling

OpenStack

Public Cloud

  • Azure: Support for user-configured polling interval for Azure virtual machine scale sets

Security

System

UI

Key Changes in 18.2.2

Issues Resolved in 18.2.2

  • AV-46453: Kubernetes: External IP is not updated when k8s service type is set to LoadBalancer
  • AV-51499: Avi Vantage not caching javascript query URI when ‘*/javascript’ is in the string group
  • AV-52075: Post-upgrade Service Engine health score reduced due to increased disk usage
  • AV-52588: Server inventory response pages not paginated
  • AV-53119: Controller cluster HA: Fixes for better reconvergence
  • AV-53301: Virtual Service -> Security overlay graphs missing data
  • AV-53365: Incorrect handling of Nagios health monitor requests
  • AV-53395: Azure: Rectify Service Engine CPU utilization values reported by Avi Vantage
  • AV-53448: OpenStack: Fix timeout issues with cloud connector RPC requests
  • AV-53547: Reduction of max SE per virtual service in the SE group does not take effect even after virtual service is disabled/enabled
  • AV-53552: Allow addition of an exclude_list to the rules for a crs_group in WAF policy
  • AV-53899: Service Engine OVA download failure from the Controller
  • AV-53902: Configuring proxy protocol in UI does not work
  • AV-53914: Service Engine failure when response event DataScript runs in the context of HTTP response generated by a request event DataScript
  • AV-53966: Controller services may restart on Controller instances that have a large number of CPUs
  • AV-53972: Metrics database usage increases on using client insights
  • AV-54003: Autorebalance configuration did not take effect for some Service Engine groups
  • AV-54008: On using HTTP/2 with caching enabled, application page does not load properly
  • AV-54081: Access to the Controller fails even after ACL preventing the access is removed
  • AV-54109: Unable to update system configuration with CLI scripting mode
  • AV-54186: Virtual service goes into fault state when certificate expiry warning is generated
  • AV-54302: Avi with Infoblox DNS profile: DNS PTR record created in forward lookup zone instead of reverse lookup zone
  • AV-54379: Service Engine crash after bond VLAN interface was deleted on bonded VLAN interface
  • AV-54752: Increase in latency with Avi not acknowledging TCP FIN packets for few flows
  • AV-54922: Linux server cloud: IPv6 on the VIP and IPv4 on the pool fails
  • AV-54931: Intermittent Service Engine failure when caching and WAF are enabled on a virtual service
  • AV-54964: SQL injection possible while using some APIs
  • AV-55142: Unable to configure a pool with autoscaling configuration if autoscale group is created with Launch Template
  • AV-55185: K8s in AWS: Virtual service failed to start due to private IP address limit on the Service Engine
  • AV-55343: Service Engine failure when a pool group is configured with redirect fail action with no destination
  • AV-55454: Service Engine failure for virtual service with application type System-SSL-Application when network profile type is set to TCP Fast
  • AV-55686: SE_HM_EVENT_SHM_UP events in the logs not preceded by any corresponding DOWN events
  • AV-55850: License: Fix in workflow for creating a new cloud with Bandwidth license
  • AV-55941: Azure: Pool members not deleted despite deleting servers from the corresponding Azure virtual machine scale set
  • AV-56113: OpenShift on Azure: One Service Engine keeps entering OPER_DISABLED mode even though K8S node is in Ready state
  • AV-56128: Support rotation of log files in /var/log/
  • AV-56197: Zone transfer through Avi DNS virtual service fails after a certain number of records are present
  • AV-56495: Modifying the application’s domain name is not propogated to Infoblox DNS/IPAM
  • AV-56625: Over a period of few days SE Persistence table usage increased to 99%
  • AV-56660: Service Engine restarts on applying Controller patch that requires a Controller reboot
  • AV-56745: Enhancement to reduce frequency of license expiry emails
  • AV-57619: User-defined metrics are incrementing even after the DataScript referencing the metrics is deleted
  • AV-58867: Fix for cloud configuration failure when Keystone V2 is used. Restrict the OpenStack flavor listing to public flavors in the UI SE group settings

Known Issues in 18.2.2

  • AV-59656: Log screen for few virtual services may never load and spin indefinitely
  • AV-56674: Adding more than 200 servers to a pool fails on AWS
  • AV-58537: Service Engine fails on GSLB follower site when the leader site pushes an incompatible TCP health monitor
  • AV-58867: Keystone V2 endpoint configured for OpenStack is not supported
  • AV-62821: For geo load-balancing at GSLB service level, when the distance between the members is smaller compared to the number of members in the pool, then some of the pools are considered to be “equi-distant” from the client, and a different pool than the desired one could be picked

What’s New in 18.2.1

Release date: 21Dec2018

ADC

GSLB

  • Ability to disable a GSLB pool

Logging

  • Support for large trap payload in aviSystemAlert trap

Networking

  • Visibility for status of bond interfaces
  • Ability to use HTTP server reselect to select an available back-end server when connection to another has failed

Private Cloud

  • Avi supports VMware hardware versions 10 and above. Support for hardware versions 8/9, corresponding to ESX5.0/5.1, has been deprecated.

Issues Resolved in 18.2.1

  • AV-32521: traceroute within the namespace does not show the hops
  • AV-33959: URL invalid encoding for redirect action
  • AV-41861: Memory leak during RSS scaleout
  • AV-42759: Azure: Latency increases after some time
  • AV-43980: Secure channel flapping between the Controller and SE when GRO is enabled
  • AV-44473: Import configuration fails if string contains Unicode character
  • AV-44659: Error message on saving HTTP security policy with rate-limit and local response HTML file
  • AV-45040: Unable to update the virtual service name to have () parentheses from UI, but can change from REST API and CLI
  • AV-45221: Virtual service placement stuck at “AWAITING_VNIC_IP” for SNI parent
  • AV-45496: Service Engine may fail if TLS persistence is used for a non-SSL pool
  • AV-45852: OpenShift: Delay in creating Avi routes
  • AV-45943: Health monitor fails if there is a \r\n\r\n before the HTTP/x.x in the send string
  • AV-46045: Linux server cloud: Service Engine may fail when DPDK is enabled on Mellanox NICs in a port channel
  • AV-46061: Third-party GSLB sites are not shown in the list of DNS policy primary and fallback sites
  • AV-46169: Syslog message with invalid PRI 324
  • AV-46742: SE stuck at OPER_DISABLING while the cluster and SEs are having intermittent network partitioning issues
  • AV-46899: OpenShift: Stale Avi bridge ports are not being cleaned up
  • AV-47080: Linux server cloud: Service Engine may fail on using multiple bond interfaces to advertise VIP via BGP
  • AV-47140: SMTP error while running email test
  • AV-47333: Upgrade hung on remote task when the time is not synced between Service Engine and the Controller
  • AV-47437: Linux server cloud: Default route may not take effect on using Mellanox NICs in in-band mode
  • AV-47568: Service Engine failure due to a corrupted persistence cookie
  • AV-47574: vCenter API version 6.7U1 is not supported by Avi Controller
  • AV-47600: Service Engine may stop processing packets if it has been up for more than 392 days
  • AV-47650: Service Engine advertising routes to BGP for virtual service that are not placed
  • AV-47797: When RSS is enabled, connections to pool servers delayed due to dropped SYN+ACK packets causing retransmits
  • AV-47800: When VIP to SNAT is enabled, changing non-critical fields (e.g., name) causes virtual service to detach and reattach to Service Engines
  • AV-50783: Virtual service cannot be enabled due to IP address exhaustion
  • AV-50784: Microsoft Azure: HTTP health monitor fails for VMs added to a pool from a scale set because of underscore (“_”) in the hostname

Performing the Upgrade

Upgrade prerequisite: The current version of the Avi Controller must be 17.2 or later.

Upgrade Instructions

Protocol Ports Used by Avi Vantage for Management Communication

Supported Platforms

Refer to System Requirements: Ecosystem

Product Documentation

For more information, please see the following documents, also available within this Knowledge Base.

Installation Guides

Open Source Package Information

Avi Networks software, Copyright © 2013-2019 by Avi Networks, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php