Avi Vantage 18.2.X Release Notes

Issues Resolved in 18.2.7 Patch Releases

Issues Resolved in 18.2.7-2p1

  • AV-75610: Connection closed when a reset frame is received on a half-closed HTTP/2 stream
  • AV-75832: Collecting Tech support via API might stall
  • AV-75965: Upgrading the system with se_patch does not work
  • AV-76037: HTTP cookie persistence does not work when connection multiplexing is disabled
  • AV-76301: In non DPDK environments supporting hot plug, adding and removing multiple interfaces owing to incorrect interface memory accounting causes connection memory depletion. This leads to errors for new connection request processing

What’s New in 18.2.7

Issues Resolved in 18.2.7

  • AV-66745: Packet processing may be delayed due to periodic host monitoring
  • AV-67995: Scheduled backups stop running if maximum number of backups are already present and the oldest backup cannot be deleted
  • AV-68168: CSP: vNIC addition may fail after a reboot of the SE due to a change in the mapping between interface name and MAC address
  • AV-70131: Avi Controller not able to stream logs to Kafka
  • AV-70181: Underscore is not allowed in GSLB service application name
  • AV-71143: UI: HTTP/2 header names in the application log overlay do not match the HTTP/2 specification
  • AV-71214: OpenStack: “Use single role for all tenants” missing in UI for OpenStack cloud
  • AV-71231: Large tx packets are not segmented to clone servers may cause delays in packet processing logic
  • AV-71349: Service Engine process can get into infinite loop when corrupted SSL data is received from backend
  • AV-71557: Virtual service in a non-admin tenant cannot be deleted if it refers to a health monitor from “admin” tenant
  • AV-71880: Restrict ElasticSearch memory allocation on Avi Controller to 32 GB, in line with ElasticSearch guidelines
  • AV-71935: Service Engine may fail due to a race condition when a virtual service with connection multiplex disabled has client IP persistence enabled
  • AV-71988: AWS: Virtual service sharing the same VIP are placed on different vNICs on the Service Engine
  • AV-72113: Health monitor does not use the correct hostname if a pool member with same IP:port has different hostname
  • AV-72194: NSX-V: Incorrect distributed firewall rule populated and incorrect port for health monitor used, on disabling and re-enabling a virtual service
  • AV-72196: GeoDb files are not processed in the right manner when the DNS virtual service has a lot of GSLB configuration downloaded to the SE at the same time and the GeoFiles are huge. In some of these scenarios, the SE will not be able to sequence to Geo configuration to the SE, thus causing the Geo discrepancy in the DNS flow AV-72565: FQDN resolution at very low dns_refresh_interval starves GSLB leader from issuing health status queries to follower
  • AV-72594: VCenter: UI does not allow configuring IP subnet and IP pool for discovered networks
  • AV-72685: CLI : Command timeouts and CLI session disconnects due to shell.py running at high CPU intermittently
  • AV-72886: Virtual services created via Contrail LBaaS driver may become unreachable because of incorrect security group attachment on Service Engine
  • AV-72888: Service Engine runs out of memory due to large number of external health monitors being scheduled
  • AV-72951: ‘all-tenant’ queries for non-admin user fail if the user does not have access to ‘admin’ tenant
  • AV-73051: GSLB: Suppress alerts for event “GSLB Site Exception Status”
  • AV-73189: Service Engine fails when a HTTP policy redirect action has tokens specified in path or host, which are not found in the actual request
  • AV-73191: Service Engine failure when logging of all headers is enabled, along with HTTP 1.0 responses
  • AV-73209: SSL clients that do not specify the elliptical curve in the handshake do not work with any PFS ciphers, resulting in ‘no shared ciphers’ error
  • AV-73211: SSL handshake may fail if the client does not send the curve list in client hello while negotiating PFS ciphers, as Avi assumes “Secp256r1” curve
  • AV-73323: OpenShift: Tenant deletion retries keep the WebApp busy making the portal inaccessible
  • AV-73509: When “Use VIP as SNAT” is enabled for a virtual service, if a pool goes down, it does not come back up as the IP address is withdrawn from BGP
  • AV-73724: Service Engine failure when “server reselect” is enabled for a pool
  • AV-73764: Service Engine failure when “server reselect” is configured on a pool, which is used by a Virtual Service that is configured to process HTTP/2 requests
  • AV-74217: enable_route_ingress_hardening flag will disable the following things in Avi OpenShift cloud:
    • No HTTP drop rules will be added for paths that do not match the host/path combination specified in the ingress/route object
    • No HTTP headers will be added for any host/path combination. Only the path will be added as a HTTP policy set object
    • A default pool group will be added that would mimic the behaviour seen in Avi Vantage version 18.2.5 or earlier

Known Issues and Workarounds in 18.2.7

  • AV-74599: Objects having non-ASCII characters in names fail during full_system export or upgrade. Run the script export_unicode_issue_script.py at /opt/avi/scripts on the Controller to identify such objects.
  • AV-75832: Collecting Tech support via API might stall. Follow the steps below for the workaround:
    • Login to the bash shell of the Controller with sudo su
    • Execute the command ps -eaf | grep shell.py
    • You will notice multiple entries for shell.py
    • Kill the PID that does not have the --server as shown below:
        
          root@admin# ps -aef | grep shell.py
          root 1690 1 0 2019 ? 00:00:43 avi-cliserver /opt/avi/python/bin/cli/bin/shell.py --server
          root 8191 8187 0 00:55 ? 00:00:59 python /opt/avi/python/bin/cli/bin/shell.py --file /var/lib/avi/tech_support/serviceengine_kh-se-rvhte.20200109-005550/serviceengine.txt_2wVJXS_temp.cli --user admin --token ff0117a1fe3d170d88483b1b06ddea60859b53bc
          root 22573 12725 0 22:03 pts/0 00:00:00 grep --color=auto shell.py
          root@admin# kill -9 8191

  • AV-75965: Upgrading the system with se_patch does not work. Follow the steps below for the workaround:
    For system on 18.2.6:
    1. Upload controller_patch.pkg_in_18.2.6
    2. Apply the controller_patch via patch controller controller_patch.pkg_in_18.2.6
    3. For an upgrade to 18.2.8, do a normal upgrade with se_patch in 18.2.8
    4. For an upgrade to to 18.2.7, upgrade the system using <controller_patch_in_18.2.7> se_patch <se_patch>
    For system on 18.2.7:
    1. Upload controller_patch.pkg_in_18.2.7
    2. Apply the controller_patch via patch controller controller_patch.pkg_in_18.2.7
    3. For an upgrade to 18.2.8, do a normal upgrade with se_patch in 18.2.8

  • AV-76037: HTTP cookie persistence does not work when connection multiplexing is disabled

Issues Resolved in 18.2.6 Patch Releases

Issues Resolved in 18.2.6-7p1

  • AV-73191: Service Engine failure when logging of all headers is enabled, along with HTTP 1.0 responses
  • AV-73209: SSL clients that do not specify the elliptical curve in the handshake do not work with any PFS ciphers, resulting in ‘no shared ciphers’ error
  • AV-75603: Destination persistence support using client mask

Issues Resolved in 18.2.6-6p1

  • AV-72886: Virtual services created via Contrail LBaas driver may become unreachable because of incorrect security group attachment on Service Engine

Issues Resolved in 18.2.6-4p3

  • AV-71582: Status field for Service type load balancer can flip between valid and null values periodically
  • AV-74439: In Azure environment, certain scenarios lead to egress source IPs not being free resulting in reuse of these IPs for other egress services
  • AV-76003: On using multiple destinations for egress service, egress pod may not get created due to the size of the destinations’ information
  • AV-76038: If multiple networks are associated with north-south Avi IPAM profile, egress service creation can lead to IPs from multiple networks getting allocated, thus depleting the static pool of IPs faster than needed

Issues Resolved in 18.2.6-4p2

  • AV-73323: OpenShift: Tenant deletion retries keep the WebApp busy making the portal inaccessible
  • AV-74315: In certain OpenShift deployments, upgrade of Avi SE can fail on using hostname for the OpenShift master API server in Avi OpenShift cloud configuration

Issues Resolved in 18.2.6-4p1

  • AV-66745: Packet processing may be delayed due to periodic host monitoring
  • AV-73191: se_dp crashes when all_headers is enabled with HTTP 1.0 responses
  • AV-73209: SSL clients that do not specify the elliptical curve in the handshake do not work with any PFS ciphers, resulting in “no shared ciphers” error
  • AV-73323: OpenShift: Tenant deletion retries keep the WebApp busy making the portal inaccessible
  • AV-74217: enable_route_ingress_hardening flag will disable the following things in Avi OpenShift cloud:
    • No HTTP drop rules will be added for paths that do not match the host/path combination specified in the ingress/route object
    • No HTTP header will be added for any host/path combination. Only the path will be added as a HTTP policy set object
    • A default pool group will be added that would mimic the behaviour seen in Avi Vantage version 18.2.5 or earlier

Issues Resolved in 18.2.6-3p1

  • AV-67846: Support for disabling Avi created security groups on Service Engines in OpenStack cloud

Issues Resolved in 18.2.6-2p3

  • AV-73209: SSL clients that do not specify the elliptical curve in the handshake do not work with any PFS ciphers, resulting in ‘no shared ciphers’ error
  • AV-73580: WAF whitelisting might not match for mix of IP ranges and individual IP addresses
  • AV-73874: WAF PSM rule might not match for case insensitive locations

Issues Resolved in 18.2.6-2p2

  • AV-72888: Service Engine runs out of memory due to many scheduled external health monitors

Issues Resolved in 18.2.6-2p1

  • AV-73191: se_dp crash when all_headers is enabled for HTTP 1.0 responses
  • AV-73209: SSL clients that do not specify the elliptical curve in the handshake do not work with any PFS ciphers, resulting in ‘no shared ciphers’ error

What’s New in 18.2.6

ADC

Networking

Security

Avi Metrics

Issues Resolved in 18.2.6

  • AV-53043: The Controller iptables are not updated when ipaddrgroup was modified
  • AV-53097: Infoblox IPAM/DNS profile features downgraded in 17.2.14
  • AV-59662: After upgrade, the older metrics are not visible
  • AV-60084: If multiple FQDNs are added to a virtual service, only the first one gets registered to AWS Route 53
  • AV-63972: The changes in ipaddrgroup are not reflected in the ipset list for specific ranges
  • AV-65713: GSLB: Re-ordering the fallback site list in the DNS policy or topology policy rule may have no effect
  • AV-65826: Automatic certificate renewal script is timing out in a specific tenant and then renewing the certificate in the admin tenant
  • AV-65920: OpenShift: IP allocation from OpenStack IPAM fails in an OpenShift environment, if the network for IPAM and virtual machine for the OpenShift node are in different tenants in OpenStack
  • AV-66302: Azure: Listing of Azure virtual machine scale sets fails with RPC timed out error during pool creation, if there are many virtual machine scale sets present in the resource group
  • AV-66905: Handled north-south traffic originating from within the node when default gateway for outgoing traffic of the virtual service is configured, and handled the container or pod traffic by adding the routes in the container or pod
  • AV-66909: Connectivity issues with the API server can cause API calls to take significant amount of time, stalling syncing of Ingresses/Apps
  • AV-67000: UI: Infoblox IPAM: Creating virtual services with placement_networks selected clears subnet field in ipam_network_subnet API request
  • AV-67064: Azure: In a combination of virtual services with and without public IP addresses placed on the same SE, a virtual service scale-in causes downtime
  • AV-67113: BGP route advertisement fails if an SE BGP peer is a part of /31 network
  • AV-67143: Log manager is not ready when messages from the SE are received
  • AV-67316: OpenShift: On Controller upgrade from Avi version below 17.2.14 (or upgraded from < 17.2.14 to a newer release) to 18.2.5, some old, inactive routes may not be updated
  • AV-67377: AWS cloud configured with non-existent management network can result in reachability issues for virtual services in all clouds
  • AV-67550: WAF: Intermittent corruption in response data when WAF response rules are enabled
  • AV-67644: SE failure due to memory exhaustion in se_log_agent process
  • AV-67647: Child SNI virtual services does not get placed in VMware / ACI cloud
  • AV-67660: Upgrade might fail from 18.2.3 to 18.2.5 during configuration import
  • AV-67724: BGP profile level keepalive or hold timer fails to take effect due to per peer default timers
  • AV-67895: Malformed packet causes policy engine to misbehave, causing SE failure
  • AV-68183: The Controller based events are not getting generated as alerts and not sent as trap/syslog
  • AV-68190: The SNI hostname is not sent to the back end when HTTPS monitor is bound to the pool and the SSL attributes are not enabled in the HTTPS health monitor
  • AV-68191: With certain OpenShift 3.11 versions, securitycontextconstraints API is not backwards compatible causing route sync to fail
  • AV-68191: OpenShift: With certain OpenShift 3.11 versions, securitycontextconstraints API is not backward compatible, causing route sync to fail
  • AV-68319: Back-end services hosted on the Kubernetes nodes can become unreachable from the SEs hosted on the same node(s) when using RancherOS with Calico CNI
  • AV-68385: Azure: VM goes into inconsistent state with the error NIC not found when the NIC is deleted during VM creation
  • AV-68512: OpenShift: Service Engine running on OpenShift on RHEL 7.7 stops processing packets a few minutes after initialization
  • AV-68519: Added option to close connection if plain-text HTTP request received on SSL service port
  • AV-68565: Error in downloading configuration backup from Avi Controller
  • AV-68971: OpenShift: Unable to create a virtual service because the application profile was referenced from the wrong tenant
  • AV-68995: SE may crash with PingID policy when a user identity is set
  • AV-69183: gRPC auth keys copied to wrong directory on follower nodes
  • AV-69186: Application learning is not working when PSM groups are created in a different tenant
  • AV-69223: No logs are displayed in the UI when the search service is down
  • AV-69265: Traffic capture does not get terminated even after reaching the configured duration
  • AV-69266: Azure: Creating se_dp processors based on number of cores
  • AV-69301: When a clone server is deleted, there is a possibility of an SE crash
  • AV-69317: GSLB FQDN uniqueness check fails, leading to sites being out of sync
  • AV-69318: A vCenter password with non-ASCII characters is not accepted due to encoding issues
  • AV-69351: With connection multiplexing feature enabled for Layer 7 virtual service, traffic cloning with preserve_client_ip does not work as expected
  • AV-69577: In AWS configuration dialog, the cross account roles may not be listed when Use cross account assume role option is selected
  • AV-69630: Azure VIP handling in Avi can cause IP address pool to be shared by both regular virtual services and egress source IPs, resulting in conflicts
  • AV-69715: High memory usage reported on Service Engines after upgrade to 18.2.5
  • AV-70130: If the system has shared VIP virtual services, the Service Engines of these virtual services can get stuck in admin_down_requested state resulting in a cascading effect of errors in the upgrade process and scaling in / migration operations on the virtual service
  • AV-70164: Creating a GSLB service for a TLS enabled ingress object fails in a Kubernetes environment
  • AV-70442: GSLB Health Monitor not functioning as expected due to incorrect namespace
  • AV-70447: When Keystone token is used for authentication, tenant check validation was not performed for that user resulting in allowed access for resources in other tenants
  • AV-70456: When a client sends a DNS request to an Avi DNS virtual service, and the client request gets directed to a site based on a DNS topology policy, the client location in the client logs is reported incorrectly as the IP address group used in the DNS policy
  • AV-71043: Virtual services go to fault state due to SSLCert update
  • AV-71117: While editing an LDAP profile, the SE crashes if the information in the field Required User Group Membership (require_user_groups) is removed
  • AV-71303: If virtual service IP addresses get deleted from the Oracle cloud, virtual service placement fails
  • AV-71331: When System-DNS application profile is used for the DNS virtual service, DNS resolution via TCP leaves TCP client connections open
  • AV-71471: Inbound rules are missing for the VIPs created after configuring vip_default_gateway, and when the OpenShift or Kubernetes cloud is updated multiple times before this configuration
  • AV-71490: Infoblox IPAM-only configuration fails if DNS view default is renamed or non-default network view is used
  • AV-71672: Backup of large configuration fails if the total size of objects of a given type exceeds a specific size limit
  • AV-71743: GSLB: When a GSLB group name is longer than 75 characters, it may result in an SE fatal error
  • AV-72190: GSLB: Updates to GSLB objects do not percolate to the follower sites if the original GSLB object had errors in the past

Key Changes in 18.2.6

  • Upgrade process starting with Avi Vantage release 18.2.6, is a two-step process. It includes the following:
    • Uploading an image or a patch using the image REST API.
    • Initiating upgrade operations using the new REST API or Avi CLI.
  • APIs for upgrade and upgrade status for the Avi Vantage release 18.2.6 are different from the APIs used before 18.2.6 release.
  • Avi Controller: Default disk size of the Avi Controller has been increased from 64 GB to 128 GB
  • Licensing: License enforcement enabled: Service Engine capacity is restricted to the licenses available on the Controller
  • UI: Tenant switching moved to a drop-down for easier operation
  • UI: Application dashboard displayed automatically on switching tenants
  • UI: New interface for monitoring upgrades and triggering emergency rollback
  • (Tech Preview) ProjectX : Controller - Avi customer portal communication for automated case creation and tech-support upload

Known Issues and Workarounds in 18.2.6

  • AV-72774: OpenStack: Virtual service stops working intermittently after upgrading to 18.2.6. To avoid this ensure that the TX ring size is modified to 128 and reboot the Service Engine to apply the configuration.
  • AV-74599: Objects having non-ASCII characters in names fail during full_system export or upgrade. Run the script at https://github.com/avinetworks/devops/blob/master/python/export_unicode_issue_script.py on the Controller to identify such objects.

Issues Resolved in 18.2.5 Patch Releases

Issues Resolved in 18.2.5-4p2

  • AV-71043: Virtual services go to Fault state due to SSLCert update

Issues Resolved in 18.2.5-4p1

  • AV-67064: Azure: With a combination of virtual services with and without public IP addresses placed on the same Service Engine, a virtual service scale in causes downtime
  • AV-67644: SE failure due to memory exhaustion in the Service Engine logging event process

Issues Resolved in 18.2.5-3p5

  • AV-70131: Controller not able to stream logs
  • AV-72190: Updates to GSLB objects do not percolate to the follower sites if the original GSLB object had errored in the past
  • AV-72196: GeoDB is not processed correctly if the SE is under configuration pressure
  • AV-72384: SE process can get to infinite loop when corrupted SSL data is received from backend
  • AV-72565: FQDN resolution at very low dns_refresh_interval starves GSLB Leader from issuing health status queries to follower
  • AV-72951: ‘all-tenant’ queries for non-admin user fails if the user does not have access to ‘admin’ tenant

Issues Resolved in 18.2.5-3p4

  • AV-70456: When a client sends a DNS request to an Avi DNS virtual service, and the client request gets directed to a site based on a DNS topology policy, the client location in the client logs is reported incorrectly as the IP address group used in the DNS policy
  • AV-71331: When System-DNS application profile is used for the DNS virtual service, DNS resolution via TCP leaves TCP client connections open
  • AV-71606: A GSLB group name longer than 75 character may result in an SE fatal error
  • AV-71672: Backup of large configuration fails if the total size of objects of a given type exceeds a specific size limit
  • AV-72113: Health monitor does not use the correct hostname, if a pool member with same IP:port has a different hostname

Issues Resolved in 18.2.5-3p3

  • AV-65216: When DNS resolution is used for pool the port number resets to inherit the default port in the pool
  • AV-68565: Not able to download backup file from the Controller
  • AV-70130: If the system has shared VIP virtual services, the Service Engines of these virtual services can get stuck in admin_down_requested state resulting in a cascading effect of errors in the upgrade process and scaling in / migration operations on the virtual service

Issues Resolved in 18.2.5-3p2

  • AV-69317: GSLB FQDN uniqueness check fails leading to SITE_OUT_OF_SYNC

Issues Resolved in 18.2.5-3p1

  • AV-59662: After upgrade, older metrics are not visible
  • AV-67414: Time to Live (TTL) value is zero for DNS responses for static DNS records and GSLB service. Avi Vantage does not use TTL configured in the DNS application profile. Workaround is to configure TTL in the GSLB service and for the static records
  • AV-67644: SE failure due to memory exhaustion in se_log_agent process
  • AV-67798: Support more than 16 fallback sites for DNS policy
  • AV-67981: Connection Multiplexing is not allowed on a virtual service referencing pool groups

Issues Resolved in 18.2.5-2p21

  • AV-76140: AWS: Virtual service placement on Service Engines may fail due to cloud connector timeout if there are more than 500 virtual services

Issues Resolved in 18.2.5-2p20

  • AV-71043: Virtual services go to fault state due to SSLCert update

Issues Resolved in 18.2.5-2p19

  • AV-73983: BGP based virtual services fail to get placed in vCenter write access clouds when multiple BGP peers are configured
  • AV-74134: Virtual service manager returns SYSERR_RM_NO_SE_IN_SE_GRP_VIP_ACC error for BGP enabled virtual service
  • AV-76037: HTTP cookie persistence does not work when connection multiplexing is disabled
  • AV-76301: In non DPDK environments supporting hot plug, adding and removing multiple interfaces owing to incorrect interface memory accounting causes connection memory depletion. This leads to errors for new connection request processing

Issues Resolved in 18.2.5-2p18

  • AV-73846: With NSX-V integration when a new virtual service is created, DFW section is created but no firewall rules are added

Issues Resolved in 18.2.5-2p17

  • AV-71935: Service Engine may fail due to a race condition when a virtual service with connection multiplex disabled has client IP persistence enabled

Issues Resolved in 18.2.5-2p16

  • AV-74134: Virtual service manager returns SYSERR_RM_NO_SE_IN_SE_GRP_VIP_ACC error for BGP enabled virtual service

Issues Resolved in 18.2.5-2p15

  • AV-69317: GSLB FQDN uniqueness check fails leading to SITE_OUT_OF_SYNC
  • AV-72120: GSLB followers out of sync
  • AV-72325: Oracle Cloud: Virtual service placement on SE may fail for short duration after restarting Avi Controller or cloud connector
  • AV-72449: Avi Controller may fail to refresh pool servers associated with AWS autoscale group
  • AV-72667: Unable to access the Controller UI using IE/Edge browser
  • AV-73591: UI support for IE11

Issues Resolved in 18.2.5-2p14

  • AV-64159: All traffic is allowed to server security group when the virtual service is disabled
  • AV-70456: When a client sends a DNS request to an Avi DNS virtual service, and the client request gets directed to a site based on a DNS topology policy, the client location in the client logs is reported incorrectly as the IP address group used in the DNS policy
  • AV-71231: Large transmission packets are not segmented to clone servers causing delays in packet processing logic
  • AV-71672: Large configuration backup may fail if the total size of objects of a given type exceeds an internal limit
  • AV-71988: AWS: Virtual service sharing the same VIP are placed on different vNICs on the Service Engine
  • AV-72113: Health monitor does not use the correct hostname if a pool member with same IP:port has a different hostname
  • AV-72194: NSX distributed firewall DFW populated with incorrect rule allowing any to any access while creating or disabling a virtual service with incorrect port service to run health monitor
  • AV-72539: NSX-v DFW rule creation fails with NSX-v 6.4.5 and above due to API change

Issues Resolved in 18.2.5-2p13

  • AV-71059: Upgrade from 17.2.7 fails in the migrate_config step if a separate partition is used for metrics
  • AV-71349: Service Engine process can get to infinite loop when corrupted SSL data is received from the backend

Issues Resolved in 18.2.5-2p12

  • AV-67550: Intermittent corruption in response data when WAF response rules are enabled
  • AV-67600: Azure: Connectivity issues to Azure APIs can cause some operations to fail with an error message: unsupported operand type(s) for -=: 'Retry' and 'int'
  • AV-70707: WAF learning: Flagged or erroneous requests are used for learning
  • AV-71994: SE occasionally skips sending application learning data to the Controller
  • AV-72042: WAF learning does not create PSM rules automatically
  • AV-72360: WAF learning messages do not reach the correct Controller

Issues Resolved in 18.2.5-2p11

  • AV-70442: When a DNS virtual service is placed on an SE that contains multiple name spaces, and the interface on which the DNS VS is placed is a port-channel, the VRF chosen by the DNS VS for health monitoring GSLB services may not be the right one resulting in health monitors staying down
  • AV-71303: If virtual service IP addresses get deleted from the cloud, virtual service placement fails
  • AV-71331: When System-DNS application profile is used for the DNS virutal service, DNS resolution via TCP leaves TCP client connections open
  • AV-71490: Infoblox IPAM-only configuration fails if DNS view default is renamed or non-default network view is used
  • AV-71606: A GSLB group name longer than 75 character may result in an SE fatal error

Issues Resolved in 18.2.5-2p10

  • AV-67892: Upgrade taking longer than expected due to SeScaleOutReady time out
  • AV-69186: Application learning is not working when PSM groups are created in different tenant
  • AV-69211: Event verification failed with percent_remaining is not 0.0 error

Issues Resolved in 18.2.5-2p9

  • AV-68512: Service Engine running on OpenShift RHEL 7.7 stops processing packets in a few minutes after initialization
  • AV-69577: In AWS configuration dialog, the cross account roles may not be listed when use cross account assume role option is selected

Issues Resolved in 18.2.5-2p8

  • AV-65216: When DNS resolution is used for pool the port number resets to inherit the default port in the pool
  • AV-69578: Update GeoDB to latest MaxMind GeoLite2
  • AV-69715: High memory usage reported after upgrading to 18.2.5
  • AV-70130: If the system has shared VIP virtual services, the Service Engines of these virtual services can get stuck in admin_down_requested state resulting in a cascading effect of errors in the upgrade process and scaling in / migration operations on the virtual service

Issues Resolved in 18.2.5-2p7

  • AV-67918: TCP-Proxy idle timeout range needs to be enhanced
  • AV-68183: The Controller based events are not getting generated as alerts and not sent as Trap/syslog
  • AV-68512: Service engine running on OpenShift on RHEL 7.7 stops processing packets in a few minutes after initialization
  • AV-69223: No logs in the UI as search service is down
  • AV-69301: When a clone server is deleted, there is a possibility of the SE crash due to invalid clone server indexing

Issues Resolved in 18.2.5-2p6

  • AV-60084: If multiple FQDNs are added to a virtual service, only the first one gets registered to AWS Route 53
  • AV-66909: Connectivity issues with the API server can cause API calls to take significant amount of time, stalling syncing of ingresses/apps
  • AV-67000: UI: Infoblox IPAM: Virtual service create with placement_networks selected clears subnet field in ipam_network_subnet API request
  • AV-68191: With certain OpenShift 3.11 versions, securitycontextconstraints API is not backwards compatible causing route sync to fail
  • AV-68565: Not able to download backup file from the Controller
  • AV-68949: UI: Subnet for VIP allocation is removed once allocation IP type is removed and then selected again
  • AV-69360: Traffic to scaled out virtual service fails on RancherOS based K8s

Issues Resolved in 18.2.5-2p5

  • AV-67723: DataScript API to get latitude and longitude co-ordinates for an IPv4 address

Issues Resolved in 18.2.5-2p4

  • AV-67723: DataScript API to get latitude and longitude co-ordinates for an IPv4 address

Issues Resolved in 18.2.5-2p3

  • AV-67113: BGP route advertisement fails if Service Engine and BGP peer are part of the /31 network
  • AV-67644: SE failure due to memory exhaustion in the Service Engine logging event process
  • AV-67895: Service Engine failure due to malformed packet causing policy engine to misbehave

Issues Resolved in 18.2.5-2p2

  • AV-66551: Virtual service is not placed on a Service Engine in VMware write access cloud, if ID networks are configured for static IP allocation under race conditions
  • AV-67647: Child SNI virtual services do not get placed on VMware / ACI cloud
  • AV-67798: Support more than 16 fallback sites in DNS policy

Issues Resolved in 18.2.5-2p1

  • AV-59662: Post upgrade, old metrics are not visible on Avi Vantage
  • AV-67316: Upgrade from Avi OpenShift deployment versions of (<17.2.14 to 18.2.5) may cause certain old inactive routes to not get updated. This version list also includes 17.2.10 -> 17.2.x(14+) -> 18.2.5, 17.2.10 -> 18.2.x(2+) -> 18.2.5
  • AV-67414: Time to Live (TTL) value is zero for DNS responses for static DNS records and GSLB service. Avi Vantage does not use TTL configured in the DNS application profile. Workaround is to configure TTL in the GSLB service and for the static records

What’s New in 18.2.5

ADC

Analytics

Automation

DataScript

DNS

GSLB

Layer 7 Proxy

  • Support for IP to ASID mapping
  • Whitelisting support for SAML authentication

Logging

Networking

Public Cloud

Security

System

Issues Resolved in 18.2.5

  • AV-56238: Stale NIC offload flags in mbufs were stalling NIC transmit queues
  • AV-58188: DNS health monitor does not allow querying AAAA record
  • AV-59904: Support for using port-security option for Neutron OpFlex plugin
  • AV-60072: OpenShift: If a pod goes into “not_ready_addresses” state temporarily, it may be removed from the pool in Avi causing traffic disruption to the route
  • AV-60897: Update-pciids hangs when there is no internet connectivity
  • AV-61057: AWS Autoscale groups with target groups attached in the environment causes polling of autoscale groups to fail
  • AV-62259: Multiple dispatchers are not in effect even when enabled for Intel 25G NIC
  • AV-63248: OpenStack: Virtual services may become unavailable during an upgrade for upto 10 minutes in OpenStack environment with Nuage SDN integration
  • AV-63282: OpenStack: Virtual service with references to missing networks in OpenStack can cause other virtual services to go down
  • AV-63405: Listing of AWS Autoscaling groups in the pool configuration UI can fail and cause AWS_ASG_FAIILURE event
  • AV-63454: Support for Syslog over TLS
  • AV-63632: Health monitor fails even on a successful response if the response has a header size that is > 2048 bytes
  • AV-63829: OpenStack: Glance image upload fails
  • AV-64025: Service Engine may fail during metrics reporting for a DNS virtual service
  • AV-64167: OpenStack: Avi deletes OpenStack port that was created for IP reservation
  • AV-64198: When GSLB site cookie persistence is enabled , the corresponding SP pool gets created in default cloud instead of actual cloud where the virtual service (GSLB pool member) is present
  • AV-64256: Service Engine fails if a virtual service with connection multiplexing disabled in the application profile refers to a pool group
  • AV-64306: With HTTP1.0, non-KeepAlive TCP connection can linger even after the request is served causing clients to slowdown
  • AV-64643: Azure: Payload can be truncated if multiple smaller packets are coalesced to a single packet of size 64K because of GRO
  • AV-64656: avi.http.redirect() in datascript does not keep virtual service in up state
  • AV-64674: SACK related vulnerabilities identified by CVE-2019-11477, CVE-2019-11478, and CVE-2019-11479
  • AV-64858: show serviceengine <se> bgp debug in a highly scaled out system causes SE agent to stall leading to SE disconnection
  • AV-64896: Disabling debug_vrf_all flag under debugvrfcontext fails to disable the debugs
  • AV-65152: AWS: Clone server configuration causes VIPs to go down if preserve_client_ip is not used
  • AV-65212: Using IP instead of DNS Name in CSR, results in SAN being populated with DNS:x.x.x.x instead of IP:x.x.x.x

Known Issues and Workarounds in 18.2.5

  • AV-64852: Upgrade fails if object names contain URI reserved characters
  • AV-67414: Time to Live (TTL) value is zero for DNS responses for static DNS records and GSLB service. Avi Vantage does not use TTL configured in the DNS application profile. Workaround is to configure TTL in the GSLB service and for the static records.

Key Changes in 18.2.5

  • For container environment, the NTP and DNS settings need to be configured on the host. The existing system configuration on the Controller will not be applicable.

Issues Resolved in 18.2.4 Patch Releases

Issues Resolved in 18.2.4-12p1

  • AV-72685: If multiple CLI sessions are running simultaneously and a heavy object is loading in the memory, then the CLI usage increases leading to command timeouts

Issues Resolved in 18.2.4-11p1

  • AV-71043: Virtual services go to Fault state due to SSLCert update

Issues Resolved in 18.2.4-10p1

  • AV-68995: Service Engine might crash with PingID policy when user identity is set

Issues Resolved in 18.2.4-9p1

  • AV-68505: Azure: SE creation with PAYG license may fail

Issues Resolved in 18.2.4-8p2

  • AV-70447: When Keystone token is used for authentication, tenant check validation was not performed for the user that allowed access to resources in other tenants

Issues Resolved in 18.2.4-8p1

  • AV-65826: Automatic certificate renewal script is timing out in specific tenant and then renewing the certificate in admin tenant

Issues Resolved in 18.2.4-7p2

  • AV-65216: When DNS resolution is used for pool the port number resets to inherit the default port in the pool

Issues Resolved in 18.2.4-7p1

  • AV-59662: Post upgrade, old metrics are not visible on Avi Vantage.
  • AV-65483: Under some race conditions, an Avi Controller node can regenerate the ssh keys that are used by other Avi Controllers or Service Engines to connect to this Avi Controller node, leading to loss of connectivity between them.

Issues Resolved in 18.2.4-5p2

  • AV-65408: AWS cloud connector may fail to attach VIPs to Service Engines if the number of VIPs are more than 300

Issues Resolved in 18.2.4-5p1

  • AV-65026: AWS: Security group rules allowing all traffic from 0.0.0.0/0 get added to the Service Engines even if SG_INGRESS_DATA option is set to None

Issues Resolved in 18.2.4-4p3

  • AV-65216: When DNS resolution is used for pool the port number resets to inherit the default port in the pool
  • AV-65483: Under some race conditions, a Controller node can regenerate its SSH keys that are used by other Controllers/Service Engines to connect to this Controller node, leading to loss of connectivity between them
  • AV-65826: Automatic certificate renewal script is timing out in specific tenant and then renewing the certificate in admin tenant
  • AV-67892: Upgrade taking longer than expected due to SeScaleOutReady time out

Issues Resolved in 18.2.4-4p2

  • AV-64372: System patch does not get applied after a Controller reboot when the Controller is running as a docker container
  • AV-65216: When DNS resolution is used for pool the port number resets to inherit the default port in the pool

Issues Resolved in 18.2.4-4p1

  • AV-64092: Unable to bind the “Placement Network” to virtual service from the Controller UI
  • AV-64351: Upgrade fails if there is an orphaned SNI child virtualservice in the configuration
  • AV-64556: SNI child virtual service placement is not in sync after upgrade when “ignore-failure” option is used to resume the upgrade
  • AV-64988: On multi VIP based setup in AWS where virtual services are scaled out across AZs each SE upgrade can take about 11-12 minutes (or more)
  • AV-65026: AWS: Security group rules allowing all traffic from 0.0.0.0/0 get added to the Service Engines even if SG_INGRESS_DATA option is set to None
  • AV-65408: AWS cloud connector may fail to attach VIPs to SEs if the number of VIPs are more than 300

Issues Resolved in 18.2.4-3p1

  • AV-63777: Unable to list networks while creating a virtual service in UI for AWS cloud

Issues Resolved in 18.2.4-2p4

  • AV-66026: In Avi Vantage version 18.2.4, based on the selinux status, if not in privileged mode Avi egress pods may not come up

Issues Resolved in 18.2.4-2p3

  • AV-64092: Unable to bind the “Placement Network” to virtual service from the Controller UI
  • AV-66143: Support for SafeNet 7.x

Issues Resolved in 18.2.4-2p2

  • AV-65219: Automatic deletion and recovery of GSLB service

Issues Resolved in 18.2.4-2p1

  • AV-62309: Allow SSL key and certificate object to be shared from admin tenant

What’s New in 18.2.4

Issues Resolved in 18.2.4

  • AV-59538: Service Engine unable to connect back to the Controller after an upgrade from an Avi Vantage version prior to 17.2.8
  • AV-60128: GSLB not marking pool member down
  • AV-61294: Uploads to HTTP/2 VIPs can fail
  • AV-61300: HTTP/2 POST requests with no “Content-Length” header gets a “400 Bad request” response
  • AV-61769: Duplicate IPs obtained from Infoblox for VIPs with the same name/port
  • AV-61819: Service Engine fails when a request with a cookie header size > 4k is sent in a SAML-authenticated session
  • AV-61875: Few Service Engines remain in partitioned state if both the leader Controller node and a follower Controller node are rebooted at the same time
  • AV-61948: Service Engine fails during HTTP/2 upload, when connectivity to the back-end servers is down
  • AV-62053: Configuring SSL profile selectors is not possible for SNI child virtual services when the child virtual service does not have a default SSL profile
  • AV-62163: Health status syncing between GSLB sites fail after upgrading to 18.2.3 due to a deprecated field
  • AV-62198: The session_id field is missing in the Avi REST API response, causing API failures
  • AV-62203: UI: Connector lines were not rendering between the tree-view components on the virtual services dashboard
  • AV-62256: Limit request and connection memory pool usage
  • AV-62436: Service Engine fails while parsing decoded arguments in an HTTP URI, under memory pressure
  • AV-62702: Virtual service creation or update fails in public clouds if enable_rhi flag is set to False
  • AV-62744: Virtual service configured with PingAccess Agent integration does not support HTTP/2
  • AV-62830: Service Engine fails when configuring PingAccess authentication profile
  • AV-62836: Failure of HTTP/2 POST requests initiated via the Chrome browser
  • AV-62852: API call to filter event logs gets stuck at percent_remaining:78 after upgrade
  • AV-62916: GSLB health monitoring fails in AWS due to a mismatch of the VRF UUID between the Avi Controller and Service Engine
  • AV-62960: HTTP POST requests from client without the Expect header can fail with a 400 response
  • AV-62966: Licensing statistics might account for deleted Service Engines and prevent further Service Engines from getting created
  • AV-62967: AWS: Moving from access-key or secret-key-based authentication to IAM role-based authentication retained stale access key, causing permission-related failures attached to the keys and subsequent virtual service downtime
  • AV-63025: GSLB may fail to consider geolocation configuration when DNS virtual service state is toggled
  • AV-63213: Memory leak due to PingAccess-Agent-specific application logs
  • AV-63226: Certificates are not being renewed with the intended SANs through the certificate management profile
  • AV-63296: Some HTTP/2 POST requests get a 503 response
  • AV-63407: Memory leak when PingAccess Agent is configured
  • AV-63471: Failure in API calls to sslkeyandcertificate
  • AV-63472: Updating a virtual service, using PATCH method on /virtualservice endpoint results in {“error”: “Mandatory key not found: vip_id”}
  • AV-63480: Avi RUM (client insight) requests do not complete, hogging memory of data-path objects on Service Engine(s)
  • AV-63588: Updating the VIP of a virtual service in OpenStack fails with an invalid subnet error
  • AV-63802: Upgrade from 17.2.14 to 18.2.3 aborted due to error in config_migrate
  • AV-63928: After installing a Service Engine patch, newly created SEs are still instantiated without the patch

Issues Resolved in 18.2.3 Patch Releases

Issues Resolved in 18.2.3-4p1

  • AV-62198: Avi Controller will send both avi_session_id and session_id again in the REST API response
  • AV-62702: Virtual service creation or update fails in public clouds if enable_rhi flag is set to False

Issues Resolved in 18.2.3-3p1

  • AV-61720: vCenter discovery not proceeding when a VM’s vNIC was attached to a portgroup which did not have read permission for the user
  • AV-61769: Infoblox issued duplicate IPs for VIPs with the same name/port
  • AV-61875: Some of the Service Engines remain in partitioned state if both the leader and follower Controller nodes are rebooted at the same time
  • AV-62309: Allow SSL key and certificate object to be shared from the admin tenant

Issues Resolved in 18.2.3-2p1

  • AV-61875: Some of the Service Engines can remain in partitioned state if both the leader and a follower Controller node are rebooted at the same time
  • AV-62163: Health status sync between GSLB Sites fails after upgrading to 18.2.3 as the upgrade site is unable to parse the response because of deprecated fields
  • AV-62309: Allow SSL key and certificate object to be shared from admin tenant

Issues Resolved in 18.2.3-1p5

  • AV-69266: Azure: Creation of se_dp processors based on number of cores

Issues Resolved in 18.2.3-1p4

  • AV-63480: Client insight requests not completed on the Service Engine hogging data path objects memory

Issues Resolved in 18.2.3-1p3

  • AV-63226: Certificates not renewed with the intended SANs through the certificate management profile

Issues Resolved in 18.2.3-1p2

  • AV-61294: Uploads to HTTP/2 VIPs can fail
  • AV-61948: Service Engine fails during a HTTP/2 upload, when connectivity to the back-end servers is down
  • AV-62198: Avi Controller will send both avi_session_id and session_id again in the REST API response
  • AV-62203: The connector lines not rendering between the tree view components
  • AV-62436: Service Engine failure while parsing decoded arguments in an HTTP URI, under memory pressure
  • AV-62702: Virtual service creation or update fails in public clouds if enable_rhi flag is set to False
  • AV-62744: Virtual service configured with ping access auth profile does not support HTTP/2
  • AV-62830: Service Engine failure while configuring ping access profile
  • AV-62916: GSLB health monitoring fails in AWS environment due to a mismatch of the VRF UUID between the Controller and SE, causing route lookup failure while sending out health monitoring packets from incorrect VRF, leading to health monitor failing
  • AV-62960: HTTP POST requests from client without Expect Header can fail with a 400 error
  • AV-62966: Licensing statistics might account for deleted Service Engines and prevent further Service Engines from getting created
  • AV-62967: Virtual services on AWS in down state after an upgrade from version 17.2.2 to 18.2.3

Issues Resolved in 18.2.3-1p1

  • AV-61787: DataScript API avi.http.saml_session_decrypt() to decrypt SAML session cookie
  • AV-61819: Service Engine failure when request with Cookie Header size greater than 4K is sent, in a SAML authenticated session
  • AV-61875: Some of the Service Engines can remain in partitioned state if both the leader and a follower Controller node are rebooted at the same time
  • AV-62053: Configuring SSL profile selectors is not possible for SNI child virtual services when the child virtual service does not have a default SSL profile
  • AV-62163: Health status sync between GSLB Sites fails after upgrading to 18.2.3 as the upgrade site is unable to parse the response because of deprecated fields
  • AV-62256: Limit request and connection memory pool usage

What’s New in 18.2.3

Release date: 2May2019

ADC

Analytics

DataScript

GSLB

  • Support for a different default LB algorithm, in case geolocation fails
  • Support for topology-based load balancing (primary/fallback sites) as a GSLB algorithm, instead of a DNS policy

Security

Containers

Public Cloud

OpenStack

  • Support for multiple networks with same CIDR
  • Support for using port-security option for Neutron OpFlex plugin

Other Ecosystems

System

  • Enhancement to limit frequency of License Expiry emails
  • Support for rotating log files in the /var/log/ directory on the Controller

Issues Resolved in 18.2.3

  • AV-46453: Kubernetes: External IP is not updated when K8s service type is set to LoadBalancer
  • AV-47046: End-to-End timing graphs not displayed
  • AV-47080: Linux server cloud: Service Engine may fail on using multiple bond interfaces to advertise VIP via BGP
  • AV-47181: On logging in as an administrator, default tenant is not set to admin
  • AV-51499: Avi Vantage not caching javascript query URI when */javascript is in string group
  • AV-51582: VIP connectivity is lost when host key-value pair is configured in SE group settings
  • AV-51693: In case of a failure, GSLB health checks are not performed on newly spawned Service Engines
  • AV-52075: Reduction in Service Engine health score due to increased SE disk usage
  • AV-52588: Server inventory response pages not paginated
  • AV-52716: Service Engine failure on pool server reselect if the server is marked down at the same time
  • AV-52722: NSX security groups are not populated in the UI
  • AV-53119: Azure: Controller cluster goes down when the Controller VMs do not get scheduled for some time
  • AV-53365: Incorrect handling of Nagios health monitor requests
  • AV-53395: Azure: Service Engine CPU utilization reported by Avi Vantage is incorrect
  • AV-53448: OpenStack: Neutron APIs timeout in a large deployment
  • AV-53552: Unable to add an exclude_list to the rules for a crs_group in WAF Policy
  • AV-53563: Intermittent requests to AWS pool members fail with “connection closed abnormally: conn deleted due to config update”
  • AV-53816: Incorrect RBAC dependency causes error in Roles edited via the UI
  • AV-53899: SE OVA download failure from the Controller if the Controller is running as a docker container
  • AV-53914: SE failure when Response event DataScript runs in the context of HTTP Response generated by a request event DataScript
  • AV-54003: Autorebalance configuration does not take effect for some service engine groups
  • AV-54008: While using HTTP/2 with caching enabled, application page does not load properly
  • AV-54081: Access to the Controller fails even after ACL preventing the access is removed
  • AV-54109: Unable to update systemconfig with CLI scripting mode
  • AV-54186: Service Engine failure when certificate expires
  • AV-54752: Avi Vantage not acknowledging FIN packets, causing delays
  • AV-54922: Linux server cloud: Failure when IPv6 is configured on the VIP and IPv4 on the pool
  • AV-54931: Service Engine may fail when caching and WAF are enabled on a virtual service
  • AV-55185: Kubernetes in AWS: Virtual service failed to start due to private IP address limit on the SE
  • AV-55343: SE failure when a pool group is configured with redirect fail action with no destination
  • AV-55410: Unexpected BGP flap due to BFD timing out
  • AV-55454: SE Failure for VS with App Type System-SSL-Application when Network Profile type is set to TCP Fast
  • AV-55686: SE_HM_EVENT_SHM_UP events in the logs not preceded by any corresponding DOWN events
  • AV-55775: OpenShift: Multiple SE include/exclude attributes do not work
  • AV-56113: OpenShift on Azure: One SE stuck in OPER_DISABLED mode even though Kubernetes node is Ready state
  • AV-56197: Zone transfer through Avi DNS VS fails after a certain number of records are present
  • AV-56236: Metrics: End-to-end timing graph in Virtual Service Analytics overlay not displayed
  • AV-56495: Modifying the application’s domain name is not propagated to Infoblox DNS/IPAM
  • AV-56528: Avi Vantage UI not showing all the pages ‘select servers from network’ view
  • AV-56625: Fix for high Service Engine Persistence Table Usage
  • AV-56660: Service Engine restarts when applying an Avi Controller patch
  • AV-56674: AWS: Adding more than 200 servers to a pool fails
  • AV-56697: SNMP trap for CONTROLLER_NODE_LEFT is generated as aviSystemAlert rather than aviControllerStatusChanged
  • AV-56734: GSLB: Round robin behavior fails when num_dns_ip is set to 0 and multiple pools have the same priority
  • AV-57344: VIP traffic from an external client fails when OpenShift/Kubernetes clusters have more than 1 NIC and the VIP NIC is not the default gateway interface
  • AV-57616: Failure in metrics APIs for user-defined/custom metrics
  • AV-58101: Service Engine failure due to BGP peer monitoring blocking data path for more than 60 seconds
  • AV-58121: Kubernetes: Any non-error egress pod log also gets dumped to the screen
  • AV-58181: Handle application of IPv6 routes with /48 mask properly
  • AV-58426: Service Engines can fail to connect to the Controller due to a race condition that triggers the cluster services watcher process on the leader node to go into an inconsistent state
  • AV-58446: When the link of physical function flaps, the virtual functions need to send a reset to recover network connectivity
  • AV-58483: HTTP Response Policy is not displayed correctly in Avi Vantage UI
  • AV-58530: External Health Monitor using ldapsearch fails
  • AV-58537: Service Engine fails on GSLB follower site when the leader site pushes an incompatible TCP health monitor
  • AV-58660: Polling for Azure VM scalesets stops if a scaleset is deleted from Azure, without removing it from the Avi Pool
  • AV-58831: SNAT sharing between VSes does not work for legacy HA
  • AV-58886: Service Engine thread gets stuck when momentary access fails in the check for a specific SE pod, causing the SE’s IP resolution to fail and potentially the extra SE object not getting cleaned up
  • AV-58900: AZURE_ACCESS_FAILURE event is not generated if access to Azure APIs fails after the cloud is up
  • AV-58901: Auth Profile cannot be configured using FQDN in System configuration
  • AV-58954: DataScript transform fails when the name of a stringgroup object referred by the DataScript is changed after creation
  • AV-58986: After a Service Engine failure due to a kernel panic, the SE fails to reconnect to the Controller
  • AV-59039: Replication issues between GSLB sites
  • AV-59049: Using underscore in Service Engine group name causes daemonset creation failure in K8s/OC cloud
  • AV-59053: GCP: Malformed URL error when adding route
  • AV-59159: OpenShift: Attribute list in K8s/OC cloud configuration with additional SE groups causes excessive SEs to be spawned
  • AV-59202: Unable to set maintenance code to HTTP health monitor
  • AV-59255: All nodes in Controller markes as “initializing” with service temporarily unavailable
  • AV-59279: Existing Routes/Ingresses can get deleted if there are K8s API server connectivity issues in rare scenarios
  • AV-59388: avi_proxy gslb annotation to update content switch httppolicyset rule under child virtual service with created GSLB FQDN
  • AV-59497: After upgrade to 18.2.2 OpenShift Routes with no Host/Path will not work without explicitly sending a Host Header in the HTTP request as Avi programs a default 404 rule
  • AV-59502: Service Engines stuck in disabled state upon changing SE group CPU/Memory/Disk Size
  • AV-59530: Stale PCI ID-to-name mapping in Linux prevents release of NIC to kernel
  • AV-59542: SE may fail with UDP per pkt virtual service preserving client IP and client port if client reuses the port
  • AV-59639: AWS deployment fails if userdata is not provided
  • AV-59642: VS Placement fails to follow legacy HA tags for VS with shared VIPs sometimes, when all such VSes were disabled and are enabled in any order
  • AV-59647: AWS: When servers are moved to standby in autoscale groups and then terminated, it can cause polling of ASGs to stop
  • AV-59658: While integrating with OpenStack Queens or higher releases, image upload might fail if interoperable image-import feature is enabled in glance service
  • AV-59699: Cisco ACI: Secondary SE may directly send a RST packet instead of tunneling it to the primary causing wrong MAC learning for the VIP
  • AV-59736: Process se_dp on Service Engine crashes when a Virtual Service referencing a shared pool is deleted
  • AV-59922: Updating an ingress annotation with invalid JSON causes the Virtual Service to be deleted
  • AV-60068: Service Engine failure when a parent VS is disabled while there is an existing connection to the child VS and connection multiplexing is disabled
  • AV-60201: Kubernetes ingress annotation does not respect specified version field
  • AV-60256: SE data NIC does not inherit configured security groups on AWS
  • AV-60304: On config restore to new Controller, Service Engines unable to connect back to Controller
  • AV-60460: When connection multiplexing is turned off, the requests coming on the client connection are sent on the back-end connection
  • AV-60527: Controller with ipset rules configured does not bring up the eth0 as /etc/network/pre-up.d script is failing
  • AV-60591: Egress pod replication Controller requires additional rights and initContainers in 18.2.2
  • AV-61073: Azure: Update of the pool fails when same IP is being used by another server in different scale set

Known Issues and Workarounds in 18.2.3

  • AV-61294: Uploads to HTTP/2 VIPs can fail in some cases, especially with a combination of a fast client and slow server. It is recommended to disable HTTP/2 on VIPs. This does not affect any file uploads to HTTP/1 VIPs.
  • AV-61380: When Avi Vantage is upgraded from 17.2.x to 18.2.3 on GCP in DPDK mode, the Service Engine loses its management interface when it comes up after the upgrade. The SE can be recovered by rebooting the SE VM after the upgrade.
  • AV-61787: Unable to decrypt SAML session cookie due to the error in the avi.crypto.decrypt API
  • AV-61819: Service Engine fails when a request with cookie header size > 4KB is sent in a SAML-authenticated session
  • AV-61875: Some of Service Engines can remain in partitioned state if both the leader and a follower Controller node are rebooted at the same time
  • AV-62053: Configuring SSL profile selectors is not possible for SNI child virtual services when the child VS does not have a default SSL profile
  • AV-62163: Health status syncing between GSLB sites fails as the upgrade site is unable to parse the response because of deprecated fields
  • AV-62256: Disabled check for the request and connection memory pool usage causes SE crash
  • AV-62702: Virtual service creation or update fails in public clouds if enable_rhi flag is set to False
  • AV-62262: Traffic loss on virtual service caused due to an unsupported user-defined metric in the DataScript
  • AV-62821: For geo load-balancing at GSLB service level, when the distance between the members is smaller compared to the number of members in the pool, then some of the pools are considered to be equi-distant from the client, and a different pool than the desired one could be picked

Issues Resolved in 18.2.2 Patch Releases

Issues Resolved in 18.2.2-9p1

  • AV-61345: Add GRATARP support for BGP virtual service

Issues Resolved in 18.2.2-8p2

  • AV-61355: SAML: Service Engine fails when request on an old connection comes in after SSO has been disabled
  • AV-61787: DataScript API avi.http.saml_session_decrypt() to decrypt SAML session cookie
  • AV-61819: Service Engine failure when request with Cookie Header size greater than 4K is sent, in a SAML authenticated session

Issues Resolved in 18.2.2-8p1

  • AV-60068: Service Engine failure when a parent virtual service is disabled while there is an existing connection to the child virtual service and the connection multiplexing is disabled

Issues Resolved in 18.2.2-7p1

  • AV-55775: OpenShift: Multiple SE include/exclude attributes do not work
  • AV-57344: VIP traffic from an external client fails when OpenShift/K8S clusters have more than 1 NIC and the VIP NIC is not the default gateway interface
  • AV-58121: Any non error egress pod log also gets dumped to the screen
  • AV-58886: SE thread gets stuck when momentary access fails in the check for a specific SE pod, causing the SE’s IP resolution to fail and potentially the extra SE object not getting cleaned up
  • AV-59279: Existing routes/ingresses can get deleted if there are K8S API server connectivity issues in rare scenarios
  • AV-59378: Default drop rule for host matching results in 404 for traffic for a route with no host defined
  • AV-59497: After upgrade to 18.2.2 OpenShift routes with no host/path will not work without explicitly sending a host header in the HTTP request as Avi programs a default 404 rule
  • AV-59502: SEs can be stuck in disabled state upon changing SE group CPU/memory/disksize

Issues Resolved in 18.2.2-6p3

  • AV-71043: Virtual services go to Fault state due to SSLCert update

Issues Resolved in 18.2.2-6p2

  • AV-67064: Azure: With a combination of virtual services with and without public IP addresses placed on the same Service Engine, a virtual service scale-in can cause down time

Issues Resolved in 18.2.2-6p1

  • AV-58900: AZURE_ACCESS_FAILURE event is not generated if access to Azure APIs fails after the cloud is up

Issues Resolved in 18.2.2-5p1

  • AV-58426: Service Engine fails to connect to the Controller triggering issues with cluster service watcher process

Issues Resolved in 18.2.2-4p1

  • AV-59394: Reset connection when client certification validation fails

Issues Resolved in 18.2.2-3p2

  • AV-61073: Azure: Update of the pool fails when same IP is used by another server in different scale set

Issues Resolved in 18.2.2-3p1

  • AV-58660: Polling for Azure VM scalesets stops if a scaleset is deleted from Azure, without being removed from Avi pool

Issues Resolved in 18.2.2-2p1

  • AV-57344: VIP traffic from an external client fails when OpenShift/K8S clusters have more than 1 NIC and the VIP NIC is not the default gateway interface
  • AV-58886: SE thread stuck when momentary access fails for a specific SE pod check causing the SE’s IP resolution to fail and potentially the extra SE object is not cleaned up

Issues Resolved in 18.2.2-1p3

  • AV-61051: Disable PCAP look-ahead logic to bring down CPU utilisation in dispatcher
  • AV-58426: Service Engines can fail to connect to the Controller due to a race condition that triggers the cluster services watcher process on the leader node to go into an inconsistent state that responds to the Service Engine with no active members in the cluster

Issues Resolved in 18.2.2-1p1

  • AV-56674: Adding more than 200 servers to a pool fails on AWS

What’s New in 18.2.2

Release date: 6Mar2019

ADC

Containers

  • OpenShift: Configuration knob to assign FQDNs automatically to a virtual service in OpenShift clouds
  • Kubernetes: Support for egress taints and tolerances in egress pod scheduling

OpenStack

Public Cloud

  • Azure: Support for user-configured polling interval for Azure virtual machine scale sets

Security

System

UI

Key Changes in 18.2.2

Issues Resolved in 18.2.2

  • AV-46453: Kubernetes: External IP is not updated when k8s service type is set to LoadBalancer
  • AV-51499: Avi Vantage not caching javascript query URI when ‘*/javascript’ is in the string group
  • AV-52075: Post-upgrade Service Engine health score reduced due to increased disk usage
  • AV-52588: Server inventory response pages not paginated
  • AV-53119: Controller cluster HA: Fixes for better reconvergence
  • AV-53301: Virtual Service -> Security overlay graphs missing data
  • AV-53365: Incorrect handling of Nagios health monitor requests
  • AV-53395: Azure: Rectify Service Engine CPU utilization values reported by Avi Vantage
  • AV-53448: OpenStack: Fix timeout issues with cloud connector RPC requests
  • AV-53547: Reduction of max SE per virtual service in the SE group does not take effect even after virtual service is disabled/enabled
  • AV-53552: Allow addition of an exclude_list to the rules for a crs_group in WAF policy
  • AV-53899: Service Engine OVA download failure from the Controller
  • AV-53902: Configuring proxy protocol in UI does not work
  • AV-53914: Service Engine failure when response event DataScript runs in the context of HTTP response generated by a request event DataScript
  • AV-53966: Controller services may restart on Controller instances that have a large number of CPUs
  • AV-53972: Metrics database usage increases on using client insights
  • AV-54003: Autorebalance configuration did not take effect for some Service Engine groups
  • AV-54008: On using HTTP/2 with caching enabled, application page does not load properly
  • AV-54081: Access to the Controller fails even after ACL preventing the access is removed
  • AV-54109: Unable to update system configuration with CLI scripting mode
  • AV-54186: Virtual service goes into fault state when certificate expiry warning is generated
  • AV-54302: Avi with Infoblox DNS profile: DNS PTR record created in forward lookup zone instead of reverse lookup zone
  • AV-54379: Service Engine crash after bond VLAN interface was deleted on bonded VLAN interface
  • AV-54752: Increase in latency with Avi not acknowledging TCP FIN packets for few flows
  • AV-54922: Linux server cloud: IPv6 on the VIP and IPv4 on the pool fails
  • AV-54931: Intermittent Service Engine failure when caching and WAF are enabled on a virtual service
  • AV-54964: SQL injection possible while using some APIs
  • AV-55142: Unable to configure a pool with autoscaling configuration if autoscale group is created with Launch Template
  • AV-55185: K8s in AWS: Virtual service failed to start due to private IP address limit on the Service Engine
  • AV-55343: Service Engine failure when a pool group is configured with redirect fail action with no destination
  • AV-55454: Service Engine failure for virtual service with application type System-SSL-Application when network profile type is set to TCP Fast
  • AV-55686: SE_HM_EVENT_SHM_UP events in the logs not preceded by any corresponding DOWN events
  • AV-55850: License: Fix in workflow for creating a new cloud with Bandwidth license
  • AV-55941: Azure: Pool members not deleted despite deleting servers from the corresponding Azure virtual machine scale set
  • AV-56113: OpenShift on Azure: One Service Engine keeps entering OPER_DISABLED mode even though K8S node is in Ready state
  • AV-56128: Support rotation of log files in /var/log/
  • AV-56197: Zone transfer through Avi DNS virtual service fails after a certain number of records are present
  • AV-56291: OpenShift: Performance degradation for large packets when the flow is handled by the secondary SE
  • AV-56495: Modifying the application’s domain name is not propogated to Infoblox DNS/IPAM
  • AV-56625: Over a period of few days SE Persistence table usage increased to 99%
  • AV-56660: Service Engine restarts on applying Controller patch that requires a Controller reboot
  • AV-56745: Enhancement to reduce frequency of license expiry emails
  • AV-57619: User-defined metrics are incrementing even after the DataScript referencing the metrics is deleted
  • AV-58867: Fix for cloud configuration failure when Keystone V2 is used. Restrict the OpenStack flavor listing to public flavors in the UI SE group settings

Known Issues in 18.2.2

  • AV-59656: Log screen for few virtual services may never load and spin indefinitely
  • AV-56674: Adding more than 200 servers to a pool fails on AWS
  • AV-58537: Service Engine fails on GSLB follower site when the leader site pushes an incompatible TCP health monitor
  • AV-58867: Keystone V2 endpoint configured for OpenStack is not supported
  • AV-62821: For geo load-balancing at GSLB service level, when the distance between the members is smaller compared to the number of members in the pool, then some of the pools are considered to be “equi-distant” from the client, and a different pool than the desired one could be picked

What’s New in 18.2.1

Release date: 21Dec2018

ADC

GSLB

  • Ability to disable a GSLB pool

Logging

  • Support for large trap payload in aviSystemAlert trap

Networking

Private Cloud

  • Avi supports VMware hardware versions 10 and above. Support for hardware versions 8/9, corresponding to ESX5.0/5.1, has been deprecated.

Issues Resolved in 18.2.1

  • AV-32521: traceroute within the namespace does not show the hops
  • AV-33959: URL invalid encoding for redirect action
  • AV-41861: Memory leak during RSS scaleout
  • AV-42759: Azure: Latency increases after some time
  • AV-43980: Secure channel flapping between the Controller and SE when GRO is enabled
  • AV-44473: Import configuration fails if string contains Unicode character
  • AV-44659: Error message on saving HTTP security policy with rate-limit and local response HTML file
  • AV-45040: Unable to update the virtual service name to have () parentheses from UI, but can change from REST API and CLI
  • AV-45221: Virtual service placement stuck at “AWAITING_VNIC_IP” for SNI parent
  • AV-45496: Service Engine may fail if TLS persistence is used for a non-SSL pool
  • AV-45852: OpenShift: Delay in creating Avi routes
  • AV-45943: Health monitor fails if there is a \r\n\r\n before the HTTP/x.x in the send string
  • AV-46045: Linux server cloud: Service Engine may fail when DPDK is enabled on Mellanox NICs in a port channel
  • AV-46061: Third-party GSLB sites are not shown in the list of DNS policy primary and fallback sites
  • AV-46169: Syslog message with invalid PRI 324
  • AV-46742: SE stuck at OPER_DISABLING while the cluster and SEs are having intermittent network partitioning issues
  • AV-46899: OpenShift: Stale Avi bridge ports are not being cleaned up
  • AV-47080: Linux server cloud: Service Engine may fail on using multiple bond interfaces to advertise VIP via BGP
  • AV-47140: SMTP error while running email test
  • AV-47333: Upgrade hung on remote task when the time is not synced between Service Engine and the Controller
  • AV-47437: Linux server cloud: Default route may not take effect on using Mellanox NICs in in-band mode
  • AV-47568: Service Engine failure due to a corrupted persistence cookie
  • AV-47574: vCenter API version 6.7U1 is not supported by Avi Controller
  • AV-47600: Service Engine may stop processing packets if it has been up for more than 392 days
  • AV-47650: Service Engine advertising routes to BGP for virtual service that are not placed
  • AV-47797: When RSS is enabled, connections to pool servers delayed due to dropped SYN+ACK packets causing retransmits
  • AV-47800: When VIP to SNAT is enabled, changing non-critical fields (e.g., name) causes virtual service to detach and reattach to Service Engines
  • AV-50783: Virtual service cannot be enabled due to IP address exhaustion
  • AV-50784: Microsoft Azure: HTTP health monitor fails for VMs added to a pool from a scale set because of underscore (“_”) in the hostname

Performing the Upgrade

Upgrade prerequisite: The current version of the Avi Controller must be 17.2 or later.

Upgrade Instructions

Protocol Ports Used by Avi Vantage for Management Communication

Supported Platforms

Refer to System Requirements: Ecosystem

Product Documentation

For more information, please see the following documents, also available within this Knowledge Base.

Installation Guides

Open Source Package Information

Avi Networks software, Copyright © 2013-2019 by Avi Networks, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php