NSX Advanced Load Balancer 22.1.X Release Notes

What’s New in 22.1.2

Release Date: 29 September 2022

Cloud Connector

Core LB Features

  • Support for load balancing in the round-robin mode at a per-SE level instead of the default per core.

System

Issue Resolved in 22.1.2

  • AV-146153: IPAM auto-allocate function does not work properly in NSX Advanced Load Balancer when connected to an Infoblox IPAM with a non-default network view and DNS view.
  • AV-148238: Oracle client-based external health monitor may cause failure of other external health monitors due to excessive logging.
  • AV-149146: Increased disk usage when application signature is enabled in the pulse connector configuration.
  • AV-149858: External logs are not received on the external server when the whole pod/container got deleted or re-imaged.
  • AV-150877: Connections are terminated if the application profile is set as System-SSL-Application, and the session is idle for 10 minutes.
  • AV-150977: Unable to set auth mapping profile in VMware NSX ALB Basic.
  • AV-150990: Unable to edit the VRF context in an NSX-T Cloud configured to use Overlay transport zones, through the UI.
  • AV-151386: Log recommendations for CRS rules 920470, 920320, 920340, and 920341, which recommend to exclude REQUEST_HEADERS: Content-Type and REQUEST_HEADERS: User-Agent from the rule results in a broken rule which will always FLAG or REJECT the request.
  • AV-151431: When connection multiplexing is disabled, persistence to a pool from prior requests can override the content-switching pool group selected by an HTTP request policy.
  • AV-151469: SSL profile with only TLS1.3 protocol and TLS1.3 ciphers can cause a fault on the Service Engine.
  • AV-151491: Virtual service creation fails when the shared option is selected for datastore scope in the SE group.
  • AV-151550: Upgrade fails for the FIPS-enabled setup for which configuration was imported after FIPS-mode was enabled.
  • AV-151763: Service Engine failure when an HTTP/2 server sends an RST_STREAM after a HEADERS frame with END_STREAM flag.
  • AV-151942: Fetching Transport nodes API fails when the transport_zone_id filter is used.
  • AV-152018: NSX Advanced Load Balancer does not display an error for duplicate VIP addresses.
  • AV-152071: Controller service (security manager) fails as postgres database connection is not concurrency safe.
  • AV-152250: When using the Certificate Management profile to auto-renew certificates, auto-renewal of certificates is triggered multiple times until the certificate is deleted from the Controller.
  • AV-152343: Virtual service gets stuck in the OPER_RESOURCES state due to an internal race condition that clears the virtual service’s discovered networks.
  • AV-152444: Portal connector service logs can reveal user-sensitive information configured in the system configuration.
  • AV-152581: Postfix package has a stale dependency on open SSL 1.1.1 in FIPS mode.
  • AV-153196: When connection multiplexing is enabled, with HTTP cookie mode of persistence, the cookie with the first request does not get sent.
  • AV-153348: In VMware cloud, unable to uncheck the Use Content Library checkbox even if the content library has not been selected in the edit mode.
  • AV-153627: The service engine might crash when disabling and enabling sharing pool.
  • AV-153725: False alert about IP reputation and App Signature sync failure when a registered Controller is disconnected from the pulse portal.
  • AV-153739: vCenter discovery may get stuck when using a static IP address for SE data vNIC allocation.
  • AV-154157:When using exclusions on a WAF policy with case-insensitive, non-regex match on the path field, the performance of WAF goes down dramatically. This is especially the case if these exclusions are on a group level.
  • AV-154173: Disabling debugging for a virtual service does not stop debug logs from being written by the Service Engine.
  • AV-155045: On an update to a virtual service with the bgp_peer_labels configured, the virtual service briefly goes down and comes up, resulting in connection drops.

Key Changes in 22.1.2

  • Connections are terminated if the application profile is set as System-SSL-Application, and the session idle time is set as 10 minutes. Now the idle connection timeout for SSL connections is increased to 60 minutes.
  • Postgres has been updated from version 10 to 14. Prior to upgrading to version 22.1.2, it is recommended to export the metrics database as in case of rollback from NSX Advanced Load Balancer 22.1.2, Avi metrics data will not be restored.

Checklist for Upgrade to NSX Advanced Load Balancer Version 22.1.2

Refer to this section before initiating upgrade.

Issues Resolved in 22.1.1 Patch Releases

Issues Resolved in 22.1.1-2p2

Release Date: 16 September 2022

  • AV-154173: On enabling debugging for a virtual service, and disabling it, there are still debug logs written by the SE.
  • AV-154157: When using exclusions on a WAF policy with case-insensitive, non-regex match on the path field, the performance of WAF goes down drastically, especially if the exclusions are on a group level.
  • AV-153348: In VMware Cloud, unable to uncheck the Use Content Library checkbox even if the content library has not been selected in edit mode.

Issues Resolved in 22.1.1-2p1

Release Date: 01 September 2022

  • AV-152250: When using the Certificate Management Profile to auto-renew certificates, auto-renewal of certificates is triggered multiple times until the certificate is deleted from the Controller.
  • AV-151763: Service Engine failure when a HTTP/2 server sends an RST_STREAM after a HEADERS frame with END_STREAM flag set.
  • AV-151491: Virtual service creation fails when the shared option is selected for datastore scope in the SE group.
  • AV-150990: Unable to edit VRF in NSX-T Cloud in Overlay mode through the UI.
  • AV-150977: Unable to set auth mapping profile in the Basic edition.
  • AV-148246: Parallel execution of the SSL certificate renewal scripts may fail.

What’s New in 22.1.1

Release Date: 15 July 2022

Cloud Connector

Core LB Features

GSLB

Networking

Monitoring and Observability

  • Support to exclude or include system events in All Events and Config Audit Trail pages.

  • Support for RTM in prometheus-metrics API calls.

SDK and Integrations

  • Multi-tenancy support for VMware ALB VRO Plugin.

  • Swagger support for Basic, Essentials, and Enterprise licensing tiers.

System

User Interface

  • UI enhancements across configuration objects.

  • The SE UUID column is introduced to the Service Engine page as an optional column.

  • Controller name and site name (if the site name is available), are displayed on the browser tab.

  • Support to search NSX Advanced Load Balancer objects using markers from the NSX Advanced Load Balancer UI.

Web Application Firewall (WAF) and API Security

Issues Resolved in 22.1.1

  • AV-132402: Setting non-default argument separator in the WAF Profile takes no effect.

  • AV-137080: BFD echo mode does not work with NSX Advanced Load Balancer.

  • AV-139518: On converting a No Orchestrator cloud to NSX-T cloud, some fields specific to the NSX-T cloud are read only and can not be configured. The option to and from the NSX-T Cloud type is not supported and the option to do so has been removed from Convert Cloud Type.

  • AV-136469: When adding a GSLB pool member for a follower site through the NSX Advanced Load Balancer UI, clicking the Virtual Services drop down list displays an error VirtualService object not found!.

  • AV-140199: For the TLS client, handshake API does not work as expected when connection is terminated after log server restart.

  • AV-141435: Shell login hangs when the number of connections reaching WAIT_TIMEDOUT increase on the Shell server.

  • AV-141493: When the Controller of version 21.1.3 or higher is configured with Cloud Services, rolling back Service Engines to version earlier than 21.1.3 results in failure of the corresponding SEs.

  • AV-142030: Password reset link for admin account fails with the error message {error: “Invalid token”}.

  • AV-142116: When incoming fragmented IPv4 packets (carrying TCP payload) post-reassembly get redirected to SE Linux interface in DPDK mode of operation, exhibit issue with IP checksum.

  • AV-142174: Service Engine can fail if a virtual service is deleted while an ICAP request is being processed.

  • AV-142218: False positives in Bot Management as requests are classified as Bad Bot based on the fact that the source IP is from public cloud providers range.

  • AV-142620: Under VS VIP configuration, under Private IP, when the VIP Address Allocation Network is updated, the NSX Advanced Load Balancer UI was retaining the IP address associated with the network configured earlier.

  • AV-143099: SSL certificate generation using control scripts for flows trying to connect to external SSL certificate authority (for example, LetsEncrypt, Venafi, Sectigo) may fail.

  • AV-143121: With Infoblox IPAM, if an invalid domain is specified in the config, host record creation requests result in a timed-out error from Infoblox leading to the leader node UI and CLI becoming unresponsive.

  • AV-143198: Service Engine may fail if the L7 virtual service listening service is configured with L4 app profile using override_application_profile and is followed by the virtual service’s network profile update.

  • AV-146331: DNS section for virtual service VIPs were not loading for AWS and Azure Cloud types.

  • AV-143699: When using WAF and CRS rules, a CRS rule which is part of a default deactivated CRS group (for example, group CRS_950_Data_Leakages) is executed.

  • AV-143798:
    • Controller cluster goes down because the node appeared to run out of listening sockets
    • Intermittent 401 errors when trying to create or edit configuration via Terraform.
    • Internal goroutine API calls to the Controller display 401 errors
  • AV-143988: POST API call made to Macro API /api/macro containing GSLB objects fails with the error message ” error”: “_perf() got multiple values for keyword argument ‘defer_octavius_request’“.

  • AV-144016: SE might crash when updating a WAF policy that is referenced by a virtual service in fault state, with open connections.

  • AV-144226: In a combination of virtual services with different network profiles, when Ignore Time Wait is enabled in some network profiles and disabled in the others, Ignore Time Wait enabled in a TCP proxy profile is not honored.

  • AV-144235: Packet capture is not working on a virtual service when dedicated dispatcher is enabled on the SE.

  • AV-144262: Creating/ updating IP address groups fails with the error {“error”: “Check checks.IpAddrGroupCheck Panicked!”} when UUID is present in the system configuration (ApiAccess and SshAccess).

  • Upgrade fails in the WaitUntilClusterReadyLocally task due to timeout on waiting for the image_manager queue.

  • AV-144544: When using write-access OpenStack cloud connector in large OpenStack environments, the NSX Advanced Load Balancer API can time out during bulk virtual service VIP operations.

  • AV-144971: Updating large IpAddrGroups can fail with a service timeout.

  • AV-145264: Creating a DNS-type Health monitor without any input in the dns_monitor field (keeping the dns_mmonitor field blank) results in a failure.

  • AV-145662: NSX-T cloud creation is failing if there is no input in the Object Name Prefix, although this field is not mandatory in the UI.

  • AV-145696: When the virtual service VIP is deleted from the Controller, the corresponding AWS Route 53 records are not removed.

  • AV-145754: HTTP requests received with both Content-Length and Transfer-Encoding:Chunked headers, will be generating a significant application log with the message Client sent a request with both chunked Transfer-Encoding and Content-Length header.

  • AV-146000: When sending RST packets, longstanding flows (for more than 30 sec) during upgrade leads to longer timeouts.

  • AV-146188: Deleting an FQDN from virtual service VIP deletes all the FQDNs of a VIP on AWS Route 53.

  • AV-146644: The error NUM_VIRTUALSERVICES: limit value 200, object count 200 is displayed when creating the 200th virtual service in UI of medium and large Controller sizes.

  • AV-146648: se_agent segmentation fault when Controller cluster size changes while a user-agent cache request ( required for bot management) is ongoing from SE to Controller.

  • AV-146774: When the albservicesconfig object is updated through the CLI or the API, there is a subsequent delay in syncing IP reputation and app signature, depending upon the configured time interval for service.

  • AV-148117: In case of an LSC cloud type with se_dp_isolation enabled, when the system is in stress, the show serviceengine cpu command might get stuck occasionally.

Key Changes in 22.1.1

  • In version 22.1.1, the string length of the name field for all objects cannot exceed 256 characters. Creation and modification of an object fails if the name exceeds the maximum string length except for the following objects, for which the maximum string length is 280 characters:
    • DNSPolicy
    • HTTPPolicySet
    • NetworkSecurityPolicy
    • VsVip
    • Pool
    • PoolGroup
      If any name exceeds the maximum character threshold on upgrade, the upgrade will fail during the migration step and rollback. See Checklist for Upgrade for more information.
  • Only single X-forward-proto will be sent to the server. If the client request contains an X-forward-proto header, then NSX Advanced Load Balancer rewrites it.

  • Search of usable networks in IPAM is now insensitive to case.

  • Jumbo frame support for NSX Advanced Load Balancer environments revised.

  • If user-defined bot mapping is specified in the bot detection policy, no input is required in the system bot mapping reference.

  • The name of individual BotMappingRule objects in a BotMapping object is mandatory. Hence, you will not be able to create any new objects without a name.

  • ControlScripts that make API calls back to the Controller API using localhost must be updated to use the DOCKER_GATEWAY environment variable instead.

  • It is recommended for a Service Engine to have at least 4 GB of memory when GeoDB is in use.

  • Prior to NSX Advanced Load Balancer version 22.1.1, it was only possible to control the update (PUT) action on any resource field. Starting with NSX Advanced Load Balancer version 22.1.1, if the access is disallowed for any field, creation of objects is not permitted as well.
  • Network objects in NSX Advanced Load Balancer now sync with the name of the associated port group in vCenter. Previously, changing name of the port group and name of the network in NSX Advanced Load Balancer was independent of each other.

Ecosystem Changes

  • vCenter Read Access was deprecated as announced in the 21.1.3 release notes. Ensure that any vCenter cloud in Read-Access mode is converted either to Write-Access, Full-Access or No-Access/No Orchestrator mode before upgrading to 22.1.1.

Known Issue in 22.1.1

  • AV-155317: DPDK driver crash with bond interfaces configured on LSC deployments containing a mix of Mellanox and other network interfaces.

Checklist for Upgrade to NSX Advanced Load Balancer Version 22.1.1

Refer to this section before initiating upgrade.

  • Upgrade to NSX Advanced Load Balancer to 22.1.1 is only supported from the following versions:

  • Version 18.2.6 through 18.2.13

  • Version 20.1.1 through 20.1.9

  • Version 21.1.1 through 21.1.4

  • Starting with NSX Advanced Load Balancer version 22.1.1, the minimum memory recommended for an Essentials Controller is 16 GB. Ensure that the memory of an Essentials Controller is at least 16 GB before upgrade.

  • vCenter Read Access is no longer supported. vCenter Read Access was deprecated as announced in the 21.1.3 release notes. Ensure that any vCenter cloud in Read-Access mode is converted either to Write-Access, Full-Access or No-Access/No Orchestrator mode before upgrading to 22.1.1.

  • The ControlScripts framework has been updated. This requires the ControlScripts to be modified prior to upgrade or on upgrade. See the ControlScripts article for more information.

  • As mentioned in the Key Changes, starting with NSX Advanced Load Balancer version 22.1.1, there is an enforcement on the string length in the name field for all objects. Use the script available here to identify all the objects that exceed the name length threshold. Ensure that the object names are modified before upgrading.

For updates on Cloud Services in version 21.1.1, see Cloud Services Release Notes.

Supported Platforms

Refer to System Requirements: Ecosystem

Product Documentation

For more information, please see the following documents, also available within this Knowledge Base.

Installation Guides

Copyrights and Open Source Package Information

For copyright information and packages used, refer to open_source_licenses.pdf.

Avi Networks software, Copyright © 2015-2022 by Avi Networks, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php

Additional Reading

VMware Ports and Protocols