NSX Advanced Load Balancer 22.1.X Release Notes

Patch Release Notes for 22.1.3

Issues Resolved in 22.1.3 Patch Releases

Issues Resolved in 22.1.3-2p4

Release Date: 28 April 2023

  • AV-159552: Multiple event files created owing to frequent Auth Manager restarts.
  • AV-168862: If a pool name has the character ‘/’ in its name, the part of the name after ‘/’ will be appended to the URI and sent to server.
  • AV-177055: Bulk SE created under Patched SE Group takes longer to download the se.pkg.
  • AV-171116: Root/Intermediate EC CA cert is not loading because of errors due to missing algorithm values in specific certificates.
  • AV-172541: The toggle button to enable/ disable new log views is not showing up if the user does not have specific permissions.
  • AV-173206: SE may fail when rate-limiting requests that collect client insights.
  • AV-174338: Missing IPtable rules in cis_mode resolved by including the IPv6 ports as well on the Controller.
  • AV-174847: Virtual service events stopped displaying due to an exception raised in the generation of ADD_NW_FAIL event.
  • AV-175496: Service Engines were failing because the GSLB Service had multiple groups with the same name.
  • AV-176511: Invalid or non-UTF-8 characters displayed when parsing metrics.
  • AV-176638: Sensitive Information of snmpv3 configs exposed in portal-webapp.log on failure of snmptrap send-attempt from the Controller.
  • AV-176939: Unable to save changes in the Edit My Account screen when the user does not have Controller-write permissions.

Issues Resolved in 22.1.3-2p3

Release Date: 07 April 2023

  • AV-174523: Cloud custom tags do not get copied to AMI snapshots.

  • AV-174263: HSM initialization fails, causing the traffic to VIP using the HSM certificate also to fail.

  • AV-173693: Issue with batch server enable/disable API for all tenant’s roles without tenant-specific roles and superuser users without roles.

  • AV- 173679: In an OpenStack cloud, the Avi Controller spins up all the SEs in parallel by executing APIs to OpenStack Controller. In a large scale deployment, SE creation can fail and hence a script is provided which can set the limit for the OpenStack cloud.

  • AV-172832: SE creation fails in AWS us-east-1 region after upgrading to the 22.1.3 version.

  • AV-172752: Support for modifying the number of parallel SE group upgrades.

  • AV-172671: Virtual service failures due to insufficient memory and low packet buffer events on SE’s (causing connection drops).

  • AV-172510: Duplicate logging of warning messages during an upgrade.

  • AV-172051: When a virtual service was attached to an SSL-enabled pool group, the scheme header was sent as an HTTP header to the backend server.

  • AV-171581: Upgrade to the 20.04 Ubuntu Controllers failed if FQDNs were used instead of Controller IP addresses due to issues with updating DNS resolvers after an upgrade.

  • AV-171222: When an SNI child Virtual Service with a DataScript in the CLIENT_SSL_PRE_CONNECT event is deleted while processing traffic, it results in Service Engine failure.

  • AV-170903: Disabling of pool servers is blocked for NSX Group-based pool servers.

  • AV-170762: SE DP start process is blocked due to stale processes in the control group from the previous invocation.

  • AV-170091: IPv6 VIP traffic fails, and RST is seen in traffic capture. This is specific to PCAP deployments, where inet6 interface is present on both default (Linux) and Avi namespaces.

  • AV-166845: If a pool’s name does not have the word pool in it, it cannot be used in the avi.requests DataScript functions.

Issues Resolved in 22.1.3-2p2

Release Date: 29 March 2023

  • AV-169773: If a GSLB service has topology rules for selecting GSLB site with AVI members, In cases of problems or delays in site status sync response, may not confirm to the configured rule action.

  • AV-166709: SE vNICs are receiving multicast traffic causing performance degradation.

  • AV-168433: For anonymous SMTP connection with Google fails because we are sending ‘[127.0.0.1]’ as hostname in ehlo message. This hostname works with local SMTP servers.

  • AV-168867: CC_IP_ATTACHED events getting incorrectly generated for Virtual Services.

  • AV-169464: OpenSSL-1.1.1f stack in the Controller and Service Engine are vulnerable to CVE-2023-0286, CVE-2023-0215, and CVE-2022-4304.

  • AV-169473: OpenSSL-1.0.2g stack in the Controller and Service Engine are vulnerable to CVE-2023-0286, CVE-2023-0215, and CVE-2022-4304.

  • AV-169779: In Analytics Profile, the following fields are not retaining updated values:
    • Log Severity
    • Non-Significant Log Severity
    • Filtered Log Severity
    • Hostname
    • Facility
    • AV-171058: The error, “failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: \“python3\”: executable file not found in $PATH: unknown” displayed on ControlScript execution in FIPS mode.
  • AV-171173: Upgrade fails with error: “KeyError: ‘protocol” if config JSON has alerts with audit_compliance_event in it.

  • AV-171698: In some cases, WAF requests can become slow if client_request_max_body_size in the WAF Profile is set to high values.

  • AV-171819: Postgres replication file cleanup logic blocking leader election after leader power-off.

  • AV-171954: Memory for negative headers match cases is not getting allocated on Service Engine, which may result in SE failure while accessing the invalid memory.

  • AV-172563: Garbage Collect API “/api/cloud//gc" fails for OpenStack cloud.

  • AV-173569: There may be a benign SSHD crash on the Controllers when upgrading to 22.1.3, applying patches 22.1.3-2p2 or higher. However, there is no other impact on the system’s functionality.

  • AV-174223: Potential failure of Log Manager during Geo-IP translation.

Issues Resolved in 22.1.3-2p1

Release Date: 28 February 2023

  • AV-147689: IP addresses allocated to a VS VIP are not released when the creation or update of VS VIP fails.
  • AV-169086: Unable to log in to NSX Advanced Load Balancer using Safari.
  • AV-168432: ControlScript does not work in a Docker Controller.
  • AV-168904: When a DNS virtual service is attached to a deactivated GSLB service and later deleted, it causes stale entries in the GSLB service DNS virtual service list. When such a GSLB service is enabled, it may cause SE failure or memory corruption.
  • AV-169398: Let’s Encrypt renewal breaks in version 22.1.3 if renewal is executed by a different user other than that specified in the certificate management script parameters.
  • AV-170014: Federated Health Monitors are not displayed in the UI.
  • AV-170116: When a DNS virtual service is bound to disabled GSLB services, on disabling and re-enabling of the DNS virtual service, the virtual service may get stuck in the OPER_DOWN state.
  • AV-170759: Internal error updating and obtaining metrics for some tables seen in the UI and REST APIs.
  • AV-170926: CSR generation from the UI fails when certificate management profile associated with CSR has any sensitive fields. In case of Let’sEncrypt, CSR generation fails with the error Invalid Credentials.
  • AV-171114: Queries triggered by metrics manager fail owing to the difference in query format due to Postgres migration from version 10 to 14.

What’s New in 22.1.3

Release Date: 31 January 2023
To refer to the upgrade checklist, click here.

Cloud Connector

AWS

GCP

LSC

NSX-T

  • Support of ENS mode for VMware deployments (Tech Preview).

  • General Availability of NSX Security-only (Distributed Firewall (DFW) to a Distributed Virtual Port Group (DVPG) mode deployments in the NSX-T Cloud). This requires VMware NSX version 3.2.2 or later.

  • The scale of virtual services has been increased from 2000 to 5000 with NSX-T cloud in the NSX Advanced Load Balancer large Controller cluster.

Oracle

VMware vCenter/ vSphere ESX

Core LB Features

DNS and IPAM

GSLB

Monitoring and Observability

Networking

System

Web Application Firewall (WAF) and Application Security

User Interface

Issues Resolved in 22.1.3

  • AV-145995: Possible configuration loss after a leader transitions to a follower which was not replicating the configuration from the leader. The configuration loss is seen if either one of the configuration’s and metrics’s database replication is working fine and the other is not.

  • AV-148598: High CPU usage observed while streaming logs to external server using se_log_agent because of frequent connection resets.

  • AV-148700: SSH service configured to ban/forbid Weak Host Key Algorithm and Key Exchange Algorithm.

  • AV-150213: Frequent enabling and disabling of servers of a pool within a few seconds can cause loss of reporting of pool-server metrics on Service Engines with five or fewer virtual services placed on them.

  • AV-150320: With SSL session resumption enabled, the pool’s SSL is using a TLS ticket from an SSL session with failed PKI validation.

  • AV-150990: Unable to edit the VRF context in an NSX-T Cloud configured to use Overlay transport zones, through the UI.

  • AV-151537: When an empty DataScript is configured from the CLI, the DataScript page fails to load in the UI.

  • AV-154300: The server batch operations do not work when using markers on pools configured with granular RBAC.

  • AV- 155317: DPDK driver failure with bond interfaces configured on LSC deployments containing a mix of Mellanox and other network interfaces.

  • AV-154144: NSX-T cloud deployment fails to perform virtual service placement when the management segment name is changed on NSX-T.

  • AV-154511: Controller versions 22.1.1 and 22.1.2 do not have the Service Engine metrics vm_stats.avg_cpu_usage, vm_stats.avg_mem_usage populated to the UI.

  • AV-154738: The Avi Controller does not fetch all the services or groups from the NSX-T manager.

  • AV-155145: Compliance mode by enabling common criteria is allowed without having to remove TLS 1.0 values from all the configured SSL/TLS profiles.

  • AV-155512: Spaces in VIP Address Allocation Network between characters displays the error, Request field uuid contains bad character.

  • AV-1555471: The NSX Advanced Load Balancer GCP cloud disables BFD on GCP Cloud routers while creating new VIP.

  • AV-156737: Through the UI, the NSX-T cloud does not display all the segments available for selection for data or management segments when there are more than 1000 segments.

  • AV-156741: Prometheus-metrics API doesn’t fetch all metrics for cases when output exceeds default dimension limit of 1000.

  • AV-157154: In an NSX-T cloud in VLAN mode, the Server Network dropdown does not list placement server networks despite having configured subnets.

  • AV-157333: Intermittent error in the IP reputation Sync.

  • AV-157419: Infoblox IPAM/DNS fails to use the non-default DNS view.

  • AV-157767: Over four billion packet transactions on a TCP connection can lead to a SE crash due to a counter overflow.

  • AV-157962: Caching incomplete objects received from the backend server could lead to Service Engine failure if the connection to the server closes abnormally.

  • AV-158056: WAF ignores the ignore_incomplete_request_body_error flag from the WAF profile when running in DETECTION mode.

  • AV-158229: REST requests fail if the API version is set prior to 20.1.1.

  • AV-158310: LDAP authentication fails when multiple LDAP auth profiles are configured.

  • AV-158550: A no-access deployment of NSX Advanced Load Balancer in legacy HA mode in OpenStack environment sends a copy of the packet to standby SE, causing issues with virtual service traffic.

  • AV-158634: On upgrading to 22.1.1-2p3, the NSX Advanced Load Balancer UI fails to load.

  • AV-159182: During network downtime, packet buffers can get queued up causing packet buffer exhaustion leading to SE failure.

  • AV-159203: Memory exhaustion on Service Engine causes Service Engine failure when attempting to establish a connection to the LDAP server.

  • AV-159228: A virtual service with only an EC certificate with OCSP stapling enabled, can cause Service Engine failure.

  • AV-159311: Under memory pressure , SE may fail due to connection memory allocation failures when processing buffered requests.

  • AV-159031: Infoblox IPAM configuration cannot be completed using the UI if the IPv6 field is blank.

  • AV-159527: The operational state of the GSLB pool Member operational state may be shown as OPER_DISABLED even when is it UP on one of the Service Engines.

  • AV-159539: SNMPv3 traps and SNMPv3 GET responses use different engine ID.

  • AV-160229: In NSX Advanced Load Balancer version 22.1.1, SE creation might fail in the NSX-T cloud setup if the management network is of Type Opaque Network.

  • AV-160400: When working with a virtual service and an attached WAF policy in a non-admin tenant, log recommendations might give the error message WafCRS not found.

  • AV-160418: Certificates with wildcard domains are not getting processed when using Enhanced Virtual Hosting.

  • AV-160532: In a setup with large number of configuration objects, when flexible upgrade is initiated, there could be loss of configuration to the Service Engines running in the older version, leading to traffic loss.

  • AV-160593: When Client Insights for a virtual service is set to Active, the virtual service is vulnerable to an HTTP desync attack on the /__avirum__ endpoint.

  • AV-160771: SE fails to come up due to memory fragmentation in DPDK mode when the packet buffer’s memory exceeds 16G. With this update, the packet buffers are reduced, and the SE would be operational, but in a degraded mode. A host reboot is required for this update to take effect.

  • AV-160898: Under some conditions, in virtual services referring to a WAF policy in which WAF CRS is selected at the SE boot up, some rules in the CRS section of WAF do not run all transformations before evaluating a request causing false negatives in rules 941160, 941170, 941210, 941220, 941310, 941350 and 942190.

  • AV-160899: Switching Persistence profiles between App cookie and client IP Address can lead to SE failure.

  • AV-160929: SYSLOG_JSON format-based syslog for config update event-based alerts can result in failure in sending syslog messages.

  • AV-161155: Several failed tasks in vCenter displayed the status The request refers to an unexpected or unknown error type. However, the failed tasks do not interrupt services on NSX Advanced Load Balancer or vCenter.

  • AV-161259: SE failure when updating HTTP Policy sets to stop using IP Reputation database and when the SE handles HTTP persistent connections during the update.

  • AV-162794: If any Tier 1 gateway or segment configured in NSX-T cloud is deleted, it marks the cloud down, and may cause disruption on all the virtual services configured in the cloud.

  • AV-162724: OpenStack Cloud: During VSVIP creation or update, IP address from a different subnet may be allocated when multiple subnets are available in the GUI.

  • AV-162948: L3 encapsulation for a scaled-out UDP virtual service with the udp-per-pkt load balancing network profile may lead to SE failure.

  • AV-163134: The show service engine command can cause SE failure if the command output is too long to fit in a page.

  • AV-163620: Memory leak when the flag collect_client_fingerprints is enabled in the application profile.

  • AV-164508: SE might fail while processing multiple EVH HTTP/2 requests.

  • AV-164511: Log streaming stops working intermittently.

  • AV-165093: Rules being removed from the IP table on deleting management access control on the Controller.

  • AV-165161: Service Engine failure when a HTTP/2 server sends two RST_STREAM frames consecutively after a DATA frame with END_STREAM flag enabled in the same stream.

  • AV-166183: Filename format of the Analytics Engine’s event mapping index leads to scale and performance issues for event handling.

  • AV-165248: From NSX Advanced Load Balancer version 21.1.4 onwards, disabling one of the virtual services with shared VIP might display a warning if any of those virtual services have child virtual service.

  • AV-166279: Service Engine failure seen with NTLM requests with Unicode characters in the username.

  • AV-167068: If a network is moved to a custom VRF, the virtual services /pools attached to the network will go down in vCenter cloud.

  • AV-166777: On upgrade from version 20.1.7 onwards to patch versions 22.1.2-2px, the SE creation on Azure cloud fails because certain set of instance types are not supported by Azure anymore.

  • AV-168482: In version 22.1.3, when updating a WAF policy via API, using an X-Avi-Version lower than 22.1.3, the update may fail with Cannot add or remove elements in the list of internal field WafPolicy.required_data_files. This can happen if a pre-crs or post-crs rule is added, which is using the @pmf operator on a data file from the WAF profile which is not already used in the current ruleset.

  • AV-177074: Resume of suspended SE groups that have virtual services scaled out may result in the exception, “Se-Scale-in Operations for SE failed.Timedout in executing ResMonWorkerService.SeScaleInRpc request_pb : se_uuid: <> transaction_uuid: <>”

    Key Changes in 22.1.3

  • The length of the Linux counterpart of interfaces in DPDK mode is limited to 11 characters. The interfaces whose names exceed 11 characters will be placed in fault state.
    To recover from the fault state, remove the SE from the cloud configuration, change the Linux bond interface name to include less than 11 characters, and then add the SE back to the cloud configuration.

  • The vCenter cloud no longer updates the host name of the pool servers with the VM name. Starting with version 22.1.3, the Hostname of the pool server VMs will not be updated by the cloud connector. User-defined names can be used as hostnames. The cloud connector will maintain the ‘external_uuid’ field in sync with the vCenter VM’s name.

  • Starting with version 22.1.3, the standard Controller image (controller.ova) is signed using a SHA-256 hash. This image is compatible with the deployment and life-cycle management of Avi Controller from the NSX Manager UI from NSX version 4.1.0 onwards. For older versions of NSX, the image controller-sha1.ova should be used.
    Note: This only impacts the deployment of the Avi Controller from NSX Manager.

  • The application profile of a DNS virtual service cannot be updated if it is attached to the system configuration. To change the virtual service’s application profile type, detach the virtual service from the system configuration and then modify the application profile.

  • The names of management and data networks imported from NSX-T will be updated on the Avi Controller to match the respective name in NSX-T.

  • The procedure to restore a Controller cluster after a failure, via the restore_config script, has changed. Cluster formation is now a two-step process:
    1. Restore the configuration on one of the nodes
    2. Reform the cluster by inviting the two new nodes to the cluster In addition, the following parameters used by the restore_config script are no longer supported:
      • DO_NOT_FORM_CLUSTER
      • VIP
      • FOLLOWER_IP [FOLLOWER_IP ...] For more information, see Backup and Restore .

System Limits

  • Route limit per VRF is increased from 500 to 2000

  • The maximum number of virtual services with real time metrics enabled in the Controller size LARGE has been increased from 200 to 1500.

  • The minimum requirement of memory for the Controller in Essentials is increased from 16G to 24G.

Known Issues in 22.1.3

  • AV-159518: Image upload fails for containerized Controller deployments while upgrading to version 22.1.3 from versions 20.1.1 through 20.1.7 and 21.1.1 through 21.1.4.
    • Workaround 1: Execute the following command under root user.
      
      $> sed -i "s/client_max_body_size 5000M;/client_max_body_size 10000M;/g" /etc/nginx/sites-enabled/default && service nginx restart && service maintenanceportal restart
      
    • Workaround 2: Manually update client_max_body_size to 10000M for /api/image in /etc/nginx/sites-enabled/default and restart maintenance_portal and Nginx.
  • AV-157854: When configuring OAuth virtual services using the UI, at least one scope needs to be configured. If no scopes are required for the deployment, then configure a placeholder scope in the UI and remove it later using the CLI.

  • AV-159426: Istio on AKO is not supported for NSX Advanced Load Balancer Controller version 22.1.3. AKO cannot push workload certificates from istiod to the Controller because of a change in certificate validation that requires common name to be present in certificates. The X.509 certificates generated by istiod, do not have common name.

  • AV-163964: Possible connection failure when a Layer 4 virtual service is configured with TCP Fast Path profile, and the Service Engine MTU is lesser than both the client and server MTUs.
    Workaround: Disable the TSO feature for the relevant Service Engine groups:
    
    configure serviceenginegroup <SEGroup-Name>
    disable_tso
    save
    
  • AV-173603: SAML authentication does not work with EVH if WAF is not enabled on the parent virtual service or if request body buffering is not enabled on the parent virtual service.
    Workaround: Enable request body buffering on the parent virtual service.
    Navigate to Templates > Profiles > Application. Select the required HTTP profile. Under the DDOS tab, click Enable Request Body Buffering to enable request body buffering for the parent virtual service.

  • AV-174983: Importing a certificate without a subject Common Name fails with error { “error”: “‘common_name’” }.
    Workaround: Import the certificate with a Common Name. See the SSL Certificates article for more information on Common Name in SSL Certificates.

Checklist for Upgrade to NSX Advanced Load Balancer Version 22.1.3

Refer to this section before initiating upgrade.

For updates on Cloud Services in version 22.1.3, see NSX Advanced Load Balancer Cloud Services Version 22.1.3 is available on VMWare Docs.

Patch Release Notes for 22.1.2

Issues Resolved in 22.1.2-2p7

Release Date: 13 April 2023

  • AV-166018: SE failure during boot-up due to race condition between SE-Agent and SE-log-agent.
  • AV-157546: Connections may be dropped at the SE when GRO is enabled at the SE Group and when the TCP timestamp option is not present in TCP data.

Issues Resolved in 22.1.2-2p6

Release Date: 02 April 2023

  • AV-171698: In some cases, WAF requests can become slow if client_request_max_body_size in the WAF Profile is set to high values.
  • AV-171581: Upgrade to 20.04 Ubuntu Controllers failed if FQDNs were used instead of Controller IPs due to issues with updating DNS resolvers after upgrade.

Issues Resolved in 22.1.2-2p4

Release Date: 24 December 2022

  • AV-162794: If any Tier 1 gateway or segment configured in NSX-T cloud is deleted, it brings down the cloud, and may cause disruption on all the virtual services configured in the cloud
  • AV-161155: Several failed tasks in vCenter displayed the status ‘The request refers to an unexpected or unknown error type’. However, the failed tasks do not interrupt services on NSX Advanced Load Balancer or vCenter.
  • AV-157154: Server network dropdown does not have options listed.

What’s New in 22.1.2-2p3

Release Date: 16 November 2022

  • Support for SSL Session ID persistence using DataScripts.
    Changes made to Default-TLS DataScript template through the UI are overwritten by the latest Default-TLS with this upgrade.

Issues Resolved in 22.1.2-2p3

  • AV-152343: Virtual service placement is stuck at OPER_RESOURCES due to an internal race condition which clears the discovered networks on the virtual services.
  • AV-155512: Spaces in VIP Address Allocation Network between characters displays the error Request field uuid contains bad character.
  • AV-157962: Caching-incomplete objects received from the backend server can lead to Service Engine failure if the connection to the server closes abnormally.
  • AV-158310: LDAP auth fails when multiple LDAP auth profiles are configured.
  • AV-159228: A virtual service with only an EC certificate with OCSP stapling enabled can cause Service Engine failure.

Issue Resolved in 22.1.2-2p2

Release Date: 26 October 2022

  • AV-158634: On upgrading to 22.1.2-2p1, the NSX Advanced Load Balancer UI fails to load.\

Issues Resolved in 22.1.2-2p1

Release Date: 20 October 2022

Note: Patch version 22.1.2-2p1 has been withdrawn due to a known issue (AV-158634: On upgrading to 22.1.2-2p1, the NSX Advanced Load Balancer UI fails to load. The Controller is available and functional via CLI and APIs).
If you have already applied 22.1.2-2p1, do either one of the following:
  • AV-157159: Avi SE creation fails in NSX-T security-only mode
  • AV-156899: On the UI, the Trusted IP configuration is not saved in a WAF Policy with Learning enabled.
  • AV-155317: DPDK driver crash with bond interfaces configured on LSC deployments containing a mix of Mellanox and other network interfaces.
  • AV-153369: Create and edit APIs are slow.
  • AV-151537: When VsDataScriptSet is configured from the CLI and has no events attached, it causes failure in loading the DataScripts page via the UI.

Known Issue in 22.1.2-2p1

  • AV-158634: On upgrading to 22.1.1-2p1, the NSX Advanced Load Balancer UI fails to load. The Controller is available and functional via CLI and APIs. See advisory note for more information.

Issues Resolved in 22.1.2 Patch Releases

Issues Resolved in 22.1.2-2p5

Release Date: 01 March 2023

  • AV-152343: Virtual Service placement is stuck at OPER_RESOURCES due to an internal race condition which clears the discovered networks on the virtual services.
  • AV-159539: SNMPv3 traps and SNMPv3 GET responses are using different engine IDs.
  • AV-163620: Memory leak when flag ‘collect_client_fingerprints’ is enabled in the application profile.
  • AV-164049: vCenter cloud creation fails to discover vCenter objects, if there are any distributed virtual port group with traffic filtering and marking feature enabled.
  • AV-165161: Service Engine may fail while processing consecutive RST_STREAM frames from a HTTP/2 server that belong to the same stream.
  • AV-166777: On upgrade from version 20.1.7 onwards to patch versions 22.1.2-2px, the SE creation on Azure cloud fails because certain set of instance types are not supported by Azure anymore.
  • AV-168413: During continuous config operations, the agent memory usage might increase over a period of time.
  • AV-168432: ControlScript does not work in a Docker Controller.
  • AV-168862: If a pool name has the character ‘/’ in its name, the part of the name after ‘/’ will be appended to the URI and sent to server.
  • AV-169464: OpenSSL-1.1.1f stack in the Controller and Service Engine is vulnerable to CVE-2023-0286, CVE-2023-0215, and CVE-2022-4304.
  • AV-169473: OpenSSL-1.0.2g stack in the Controller and Service Engine is vulnerable to CVE-2023-0286, CVE-2023-0215, and CVE-2022-4304.
  • AV-171114: Queries triggered by metrics manager fail owing to the difference in query format due to Postgres migration from version 10 to 14.

What’s New in 22.1.2

Release Date: 29 September 2022

Cloud Connector

Core LB Features

  • Support for load balancing in the round-robin mode at a per-SE level instead of the default per core.

System

Issues Resolved in 22.1.2

  • AV-142908: On failure of a macro-API containing a VSVIP change or a VSVIP post failure, the IP allocated is not released on failure.
  • AV-146153: IPAM auto-allocate function does not work properly in NSX Advanced Load Balancer when connected to an Infoblox IPAM with a non-default network view and DNS view.
  • AV-148238: Oracle client-based external health monitor may cause failure of other external health monitors due to excessive logging.
  • AV-149146: Increased disk usage when application signature is enabled in the pulse connector configuration.
  • AV-149858: External logs are not received on the external server when the whole pod/container got deleted or re-imaged.
  • AV-150877: Connections are terminated if the application profile is set as System-SSL-Application, and the session is idle for 10 minutes.
  • AV-150977: Unable to set auth mapping profile in VMware NSX ALB Basic.
  • AV-150990: Unable to edit the VRF context in an NSX-T Cloud configured to use Overlay transport zones, through the UI.
  • AV-151386: Log recommendations for CRS rules 920470, 920320, 920340, and 920341, which recommend to exclude REQUEST_HEADERS: Content-Type and REQUEST_HEADERS: User-Agent from the rule results in a broken rule which will always FLAG or REJECT the request.
  • AV-151431: When connection multiplexing is disabled, persistence to a pool from prior requests can override the content-switching pool group selected by an HTTP request policy.
  • AV-151469: SSL profile with only TLS1.3 protocol and TLS1.3 ciphers can cause a fault on the Service Engine.
  • AV-151491: Virtual service creation fails when the shared option is selected for datastore scope in the SE group.
  • AV-151550: Upgrade fails for the FIPS-enabled setup for which configuration was imported after FIPS-mode was enabled.
  • AV-151763: Service Engine failure when an HTTP/2 server sends an RST_STREAM after a HEADERS frame with END_STREAM flag.
  • AV-151942: Fetching Transport nodes API fails when the transport_zone_id filter is used.
  • AV-152018: NSX Advanced Load Balancer does not display an error for duplicate VIP addresses.
  • AV-152071: Controller service (security manager) fails as postgres database connection is not concurrency safe.
  • AV-152250: When using the Certificate Management profile to auto-renew certificates, auto-renewal of certificates is triggered multiple times until the certificate is deleted from the Controller.
  • AV-152343: Virtual service gets stuck in the OPER_RESOURCES state due to an internal race condition that clears the virtual service’s discovered networks.
  • AV-152444: Portal connector service logs can reveal user-sensitive information configured in the system configuration.
  • AV-152581: Postfix package has a stale dependency on open SSL 1.1.1 in FIPS mode.
  • AV-153196: When connection multiplexing is enabled, with HTTP cookie mode of persistence, the cookie with the first request does not get sent.
  • AV-153348: In VMware cloud, unable to uncheck the Use Content Library checkbox even if the content library has not been selected in the edit mode.
  • AV-153627: The service engine might crash when disabling and enabling sharing pool.
  • AV-153725: False alert about IP reputation and App Signature sync failure when a registered Controller is disconnected from the pulse portal.
  • AV-153739: vCenter discovery may get stuck when using a static IP address for SE data vNIC allocation.
  • AV-154157:When using exclusions on a WAF policy with case-insensitive, non-regex match on the path field, the performance of WAF goes down dramatically. This is especially the case if these exclusions are on a group level.
  • AV-154173: Disabling debugging for a virtual service does not stop debug logs from being written by the Service Engine.
  • AV-155045: On an update to a virtual service with the bgp_peer_labels configured, the virtual service briefly goes down and comes up, resulting in connection drops.

Key Changes in 22.1.2

  • Connections are terminated if the application profile is set as System-SSL-Application, and the session idle time is set as 10 minutes. Now the idle connection timeout for SSL connections is increased to 60 minutes.
  • Postgres has been updated from version 10 to 14. Prior to upgrading to version 22.1.2, it is recommended to export the metrics database as in case of rollback from NSX Advanced Load Balancer 22.1.2, Avi metrics data will not be restored.
  • Network objects in NSX Advanced Load Balancer now sync with the name of the associated port group in vCenter. Previously, changing name of the port group and name of the network in NSX Advanced Load Balancer was independent of each other.

    Checklist for Upgrade to NSX Advanced Load Balancer Version 22.1.2

    Refer to this section before initiating upgrade.

  • Upgrade to NSX Advanced Load Balancer to 22.1.1 is only supported from the following versions:

  • Version 18.2.6 through 18.2.13

  • Version 20.1.1 through 20.1.9

  • Version 21.1.1 through 21.1.4

  • Version 21.1.5 through 22.1.2

  • Before upgrading to version 22.1.2, export the Avi metrics database. In case of rolling back from NSX Advanced Load Balancer 22.1.2 to an earlier version, import the metrics database to prevent loss of metrics data.

  • Starting with NSX Advanced Load Balancer version 22.1.2, the minimum memory recommended for an Essentials Controller is 16 GB. Ensure that the memory of an Essentials Controller is at least 16 GB before upgrade.

  • vCenter Read Access is no longer supported. vCenter Read Access was deprecated as announced in the 21.1.3 release notes. Ensure that any vCenter cloud in Read-Access mode is converted either to Write-Access, Full-Access or No-Access/No Orchestrator mode before upgrading to 22.1.1.

  • The ControlScripts framework has been updated. This requires the ControlScripts to be modified prior to upgrade or on upgrade. See the ControlScripts article for more information.

  • As mentioned in the Key Changes, starting with NSX Advanced Load Balancer version 22.1.1, there is an enforcement on the string length in the name field for all objects. Use the script available here to identify all the objects that exceed the name length threshold. Ensure that the object names are modified before upgrading.

Issues Resolved in 22.1.1 Patch Releases

Issues Resolved in 22.1.1-2p6

Release Date: 03 April 2023

  • AV-171698: In some cases, WAF requests can become slow if client_request_max_body_size in the WAF Profile is set to high values.
  • AV-171581: Upgrade to 20.04 Ubuntu Controllers failed if FQDNs were used instead of Controller IPs due to issues with updating DNS resolvers after upgrade.

Issues Resolved in 22.1.1-2p5

Release Date: 21 March 2023

  • AV-168413: During continuous config operations, the agent memory usage might increase over a period of time.
  • AV-166279: Service Engine failure seen with NTLM requests with unicode characters in the username.
  • AV-165161: Service Engine may fail while processing consecutive RST_STREAM frames from a HTTP/2 server that belong to the same stream.
  • AV-164049: vCenter cloud creation fails to discover vCenter objects, if there are any distributed virtual port group with traffic filtering and marking feature enabled.
  • AV-163620: Memory leak when flag ‘collect_client_fingerprints’ is enabled in the application profile.
  • AV-161259: SE failure when updating HTTP Policy sets to stop using IP Reputation database and when the SE handles HTTP persistent connections during the update.
  • AV-160898: Under some conditions, in virtual services referring to a WAF policy in which WAF CRS is selected at the SE boot up, some rules in the CRS section of WAF do not run all transformations before evaluating a request causing false negatives in rules 941160, 941170, 941210, 941220, 941310, 941350 and 942190.
  • AV-160593: When Client Insights for a virtual service is set to Active, the virtual service is vulnerable to an HTTP desync attack on the /__avirum__ endpoint.
  • AV-160229: In NSX Advanced Load Balancer version 22.1.1, SE creation might fail in the NSX-T cloud setup if the management network is of a type opaque network.
  • AV-158267: Service Engine failure seen with NTLM requests with unicode characters in the username.
  • AV-157546: Connections may be dropped at the SE when the TCP timestamp option is not present in TCP data.
  • AV-156765: Once Cloud Services get disconnected, it does not get connected without manual intervention.

Issue Resolved in 22.1.1-2p4

  • AV-158634: On upgrading to 22.1.1-2p3, the NSX Advanced Load Balancer UI fails to load.

Issues Resolved in 22.1.1-2p3

Release Date: 13 October 2022

  • AV-155512: Spaces in VIP Address Allocation Network between characters displays the error Request field uuid contains bad character.
  • AV-153369: Create and edit APIs are slow.

Issues Resolved in 22.1.1-2p2

Release Date: 16 September 2022

  • AV-154173: On enabling debugging for a virtual service, and disabling it, there are still debug logs written by the SE.
  • AV-154157: When using exclusions on a WAF policy with case-insensitive, non-regex match on the path field, the performance of WAF goes down drastically, especially if the exclusions are on a group level.
  • AV-153348: In VMware Cloud, unable to uncheck the Use Content Library checkbox even if the content library has not been selected in edit mode.

Issues Resolved in 22.1.1-2p1

Release Date: 01 September 2022

  • AV-152250: When using the Certificate Management Profile to auto-renew certificates, auto-renewal of certificates is triggered multiple times until the certificate is deleted from the Controller.
  • AV-151763: Service Engine failure when a HTTP/2 server sends an RST_STREAM after a HEADERS frame with END_STREAM flag set.
  • AV-151491: Virtual service creation fails when the shared option is selected for datastore scope in the SE group.
  • AV-150990: Unable to edit VRF in NSX-T Cloud in Overlay mode through the UI.
  • AV-150977: Unable to set auth mapping profile in the Basic edition.
  • AV-148246: Parallel execution of the SSL certificate renewal scripts may fail.

What’s New in 22.1.1

Release Date: 15 July 2022

Cloud Connector

Core LB Features

GSLB

Networking

Monitoring and Observability

  • Support to exclude or include system events in All Events and Config Audit Trail pages.

  • Support for RTM in prometheus-metrics API calls.

SDK and Integrations

  • Multi-tenancy support for VMware ALB VRO Plugin.

  • Swagger support for Basic, Essentials, and Enterprise licensing tiers.

System

User Interface

  • UI enhancements across configuration objects.

  • The SE UUID column is introduced to the Service Engine page as an optional column.

  • Controller name and site name (if the site name is available), are displayed on the browser tab.

  • Support to search NSX Advanced Load Balancer objects using markers from the NSX Advanced Load Balancer UI.

Web Application Firewall (WAF) and API Security

Issues Resolved in 22.1.1

  • AV-132402: Setting non-default argument separator in the WAF Profile takes no effect.

  • AV-137080: BFD echo mode does not work with NSX Advanced Load Balancer.

  • AV-139518: On converting a No Orchestrator cloud to NSX-T cloud, some fields specific to the NSX-T cloud are read only and can not be configured. The option to and from the NSX-T Cloud type is not supported and the option to do so has been removed from Convert Cloud Type.

  • AV-136469: When adding a GSLB pool member for a follower site through the NSX Advanced Load Balancer UI, clicking the Virtual Services drop down list displays an error VirtualService object not found!.

  • AV-140199: For the TLS client, handshake API does not work as expected when connection is terminated after log server restart.

  • AV-141435: Shell login hangs when the number of connections reaching WAIT_TIMEDOUT increase on the Shell server.

  • AV-141493: When the Controller of version 21.1.3 or higher is configured with Cloud Services, rolling back Service Engines to version earlier than 21.1.3 results in failure of the corresponding SEs.

  • AV-142030: Password reset link for admin account fails with the error message {error: “Invalid token”}.

  • AV-142116: When incoming fragmented IPv4 packets (carrying TCP payload) post-reassembly get redirected to SE Linux interface in DPDK mode of operation, exhibit issue with IP checksum.

  • AV-142174: Service Engine can fail if a virtual service is deleted while an ICAP request is being processed.

  • AV-142218: False positives in Bot Management as requests are classified as Bad Bot based on the fact that the source IP is from public cloud providers range.

  • AV-142620: Under VS VIP configuration, under Private IP, when the VIP Address Allocation Network is updated, the NSX Advanced Load Balancer UI was retaining the IP address associated with the network configured earlier.

  • AV-143099: SSL certificate generation using control scripts for flows trying to connect to external SSL certificate authority (for example, LetsEncrypt, Venafi, Sectigo) may fail.

  • AV-143121: With Infoblox IPAM, if an invalid domain is specified in the config, host record creation requests result in a timed-out error from Infoblox leading to the leader node UI and CLI becoming unresponsive.

  • AV-143198: Service Engine may fail if the L7 virtual service listening service is configured with L4 app profile using override_application_profile and is followed by the virtual service’s network profile update.

  • AV-146331: DNS section for virtual service VIPs were not loading for AWS and Azure Cloud types.

  • AV-143699: When using WAF and CRS rules, a CRS rule which is part of a default deactivated CRS group (for example, group CRS_950_Data_Leakages) is executed.

  • AV-143798:
    • Controller cluster goes down because the node appeared to run out of listening sockets
    • Intermittent 401 errors when trying to create or edit configuration via Terraform.
    • Internal goroutine API calls to the Controller display 401 errors
  • AV-143988: POST API call made to Macro API /api/macro containing GSLB objects fails with the error message ” error”: “_perf() got multiple values for keyword argument ‘defer_octavius_request’“.

  • AV-144016: SE might crash when updating a WAF policy that is referenced by a virtual service in fault state, with open connections.

  • AV-144226: In a combination of virtual services with different network profiles, when Ignore Time Wait is enabled in some network profiles and disabled in the others, Ignore Time Wait enabled in a TCP proxy profile is not honored.

  • AV-144235: Packet capture is not working on a virtual service when dedicated dispatcher is enabled on the SE.

  • AV-144262: Creating/ updating IP address groups fails with the error {“error”: “Check checks.IpAddrGroupCheck Panicked!”} when UUID is present in the system configuration (ApiAccess and SshAccess).

  • Upgrade fails in the WaitUntilClusterReadyLocally task due to timeout on waiting for the image_manager queue.

  • AV-144544: When using write-access OpenStack cloud connector in large OpenStack environments, the NSX Advanced Load Balancer API can time out during bulk virtual service VIP operations.

  • AV-144971: Updating large IpAddrGroups can fail with a service timeout.

  • AV-145264: Creating a DNS-type Health monitor without any input in the dns_monitor field (keeping the dns_mmonitor field blank) results in a failure.

  • AV-145662: NSX-T cloud creation is failing if there is no input in the Object Name Prefix, although this field is not mandatory in the UI.

  • AV-145696: When the virtual service VIP is deleted from the Controller, the corresponding AWS Route 53 records are not removed.

  • AV-145754: HTTP requests received with both Content-Length and Transfer-Encoding:Chunked headers, will be generating a significant application log with the message Client sent a request with both chunked Transfer-Encoding and Content-Length header.

  • AV-146000: When sending RST packets, longstanding flows (for more than 30 sec) during upgrade leads to longer timeouts.

  • AV-146188: Deleting an FQDN from virtual service VIP deletes all the FQDNs of a VIP on AWS Route 53.

  • AV-146644: The error NUM_VIRTUALSERVICES: limit value 200, object count 200 is displayed when creating the 200th virtual service in UI of medium and large Controller sizes.

  • AV-146648: se_agent segmentation fault when Controller cluster size changes while a user-agent cache request ( required for bot management) is ongoing from SE to Controller.

  • AV-146774: When the albservicesconfig object is updated through the CLI or the API, there is a subsequent delay in syncing IP reputation and app signature, depending upon the configured time interval for service.

  • AV-147689: IP addresses allocated to a VS VIP are not released when the creation or update of VS VIP fails.

  • AV-148117: In case of an LSC cloud type with se_dp_isolation enabled, when the system is in stress, the show serviceengine cpu command might get stuck occasionally.

Key Changes in 22.1.1

  • In version 22.1.1, the string length of the name field for all objects cannot exceed 256 characters. Creation and modification of an object fails if the name exceeds the maximum string length except for the following objects, for which the maximum string length is 280 characters:
    • DNSPolicy
    • HTTPPolicySet
    • NetworkSecurityPolicy
    • VsVip
    • Pool
    • PoolGroup
      If any name exceeds the maximum character threshold on upgrade, the upgrade will fail during the migration step and rollback. See Checklist for Upgrade for more information.
  • Only single X-forward-proto will be sent to the server. If the client request contains an X-forward-proto header, then NSX Advanced Load Balancer rewrites it.

  • Search of usable networks in IPAM is now insensitive to case.

  • Jumbo frame support for NSX Advanced Load Balancer environments revised.

  • If user-defined bot mapping is specified in the bot detection policy, no input is required in the system bot mapping reference.

  • The name of individual BotMappingRule objects in a BotMapping object is mandatory. Hence, you will not be able to create any new objects without a name.

  • ControlScripts that make API calls back to the Controller API using localhost must be updated to use the DOCKER_GATEWAY environment variable instead.

  • It is recommended for a Service Engine to have at least 4 GB of memory when GeoDB is in use.

  • Prior to NSX Advanced Load Balancer version 22.1.1, it was only possible to control the update (PUT) action on any resource field. Starting with NSX Advanced Load Balancer version 22.1.1, if the access is disallowed for any field, creation of objects is not permitted as well.
  • Network objects in NSX Advanced Load Balancer now sync with the name of the associated port group in vCenter. Previously, changing name of the port group and name of the network in NSX Advanced Load Balancer was independent of each other.

Ecosystem Changes

  • vCenter Read Access was deprecated as announced in the 21.1.3 release notes. Ensure that any vCenter cloud in Read-Access mode is converted either to Write-Access, Full-Access or No-Access/No Orchestrator mode before upgrading to 22.1.1.

Known Issue in 22.1.1

  • AV-155317: DPDK driver crash with bond interfaces configured on LSC deployments containing a mix of Mellanox and other network interfaces.

Checklist for Upgrade to NSX Advanced Load Balancer Version 22.1.1

Refer to this section before initiating upgrade.

  • Upgrade to NSX Advanced Load Balancer to 22.1.1 is only supported from the following versions:

  • Version 18.2.6 through 18.2.13

  • Version 20.1.1 through 20.1.9

  • Version 21.1.1 through 21.1.4

  • Version 21.1.5 through 22.1.1

  • Starting with NSX Advanced Load Balancer version 22.1.1, the minimum memory recommended for an Essentials Controller is 16 GB. Ensure that the memory of an Essentials Controller is at least 16 GB before upgrade.

  • vCenter Read Access is no longer supported. vCenter Read Access was deprecated as announced in the 21.1.3 release notes. Ensure that any vCenter cloud in Read-Access mode is converted either to Write-Access, Full-Access or No-Access/No Orchestrator mode before upgrading to 22.1.1.

  • The ControlScripts framework has been updated. This requires the ControlScripts to be modified prior to upgrade or on upgrade. See the ControlScripts article for more information.

  • As mentioned in the Key Changes, starting with NSX Advanced Load Balancer version 22.1.1, there is an enforcement on the string length in the name field for all objects. Use the script available here to identify all the objects that exceed the name length threshold. Ensure that the object names are modified before upgrading.

For updates on Cloud Services in version 21.1.1, see Cloud Services Release Notes.

Supported Platforms

Refer to System Requirements: Ecosystem

Product Documentation

For more information, please see the following documents, also available within this Knowledge Base.

Installation Guides

Copyrights and Open Source Package Information

For copyright information and packages used, refer to open_source_licenses.pdf.

Avi Networks software, Copyright © 2015-2022 by Avi Networks, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php

Additional Reading

VMware Ports and Protocols