Flexible Upgrades for NSX Advanced Load Balancer
Overview
NSX Advanced Load Balancer supports improved and more flexible methods for upgrading NSX Advanced Load Balancer system. The followings are the additional features for the Flexible Upgrades:
- The upgrade is possible per SE group. The transition of all the SE groups to the new version may occur over a long period.
- Upgrades of different SE groups are supported with different patch versions.
- Rollback to the previous versions of NSX Advanced Load Balancer is non-disruptive.
From NSX Advanced Load Balancer prior to 18.2.6, the only available option was system-level (NSX Advanced Load Balancer Controller and SE Groups) upgrade. With Flexible Upgrades, the following options are available:
| Upgrades | Patch Upgrades | Rollback | Rollback Patch |
|---|---|---|---|
| System (Controller and SE Groups) | System (Controller and SE Groups | System (Controller and SE Groups | System (Controller and SE Groups |
| Controller only | Controller only | Controller only | Controller only |
| Some or all the SE groups | Some or all the SE groups | Some or all the SE groups | Some or all the SE groups |
Use Cases
- Scenarios when it is not possible to upgrade all SE groups to the newer version at the same time due to various business reasons such as logistics, confidence in the new software, etc.
- The configuration is blocked during the entire duration of the Controller and SE upgrade. This is not acceptable in many deployments. With the new upgrade feature, the process is flexible and can be performed per SE group basis. The configuration is blocked for the entire duration if a system upgrade is performed till all Service Engines are upgraded.
- Using SE groups for data plane separation. Based upon the SE group segmentation, the upgrade is performed based upon the following attributes.
- Application or product offering
- Tenant
- Production, pre-production and development environments
- Cloud or environment (AWS, VMware, etc.)
- Provide patches to only applications or SE groups that need them
- Flexible scheduling
- Self-service upgrades
Image Management and Service
Image service is the first step in the flexible upgrade work-flow. It is used to upload the image after which an upgrade operation can be initiated. The Controller hosts images of different versions since SE groups could be potentially in different versions.
The Controller should have additional disk space to host these images.
The Controller images for the major versions include the followings:
- controller.pkg (for VM-based Controller)
- controller_docker.tgz (For Docker-based Controller)
Images for the patches include the following:
- avi_patch.pkg — Full package
- controller_patch.pkg — The Controller package
- se_patch.pkg — SE patch package
As a part of the upload process, image service extracts files, metadata from the package. This information is not only presented to the user but also used in the upgrade process.
Notes:
- Image service provides an ability to upload, query and delete NSX Advanced Load Balancer image(s) to the system.
- Image service supports the upload of NSX Advanced Load Balancer patch packages.
- Image upload can happen only on the cluster leader. It is not allowed from a cluster member.
Image Bundling
NSX Advanced Load Balancer now supports the composite image or the image bundle. The composite image of NSX Advanced Load Balancer consists of the following:
- Base image – Controller image (controller_docker.tgz, controller.pkg, controller ova, controller.qcow2, etc)
- Controller package – It is an optional package
- SE patch image – It is an optional package
The upgrade workflow using the image bundle, or the composite image is the same as using the standard image. When the image bundle is used for upgrade, a patch image can also be applied along with the base image.
Note: When upgrading from NSX Advanced Load Balancer versions 17.x or version lesser than 18.2.6 to NSX Advanced Load Balancer 20.1 and higher, in the Controller, change the DefaultTimeoutStartSec (File: /etc/systemd/system.conf) to 120 seconds to avoid timeout during upgrade.
Uploading Image Using CLI
The CLI provides better control of the upgrade operations leading to a consistent and predictable workflow.
For uploading the package use the upload image filename <path-of-the-package> command as shown below.
[admin:controller]: > upload image filename /tmp/controller.pkg The following show command returns the details of the image metadata.
show image <image-name>
[admin:-controller]: > show image
+-----------------------------+--------------------------------------------+----------------+
| Name | UUID | Status |
+-----------------------------+--------------------------------------------+----------------+
| 18.2.7-5000-20191009.205501 | image-fxxxx22-0f40-45de-8551-15xxxxxxx1fe | SYSERR_SUCCESS |
+-----------------------------+-----
The existing API endpoints (prior to 18.2.6) are not supported. To know more about differences in CLI commands and APIs refer to Comparison Table for Differences in CLIs Commands and APIs.
Uploading Image Service using REST API
A POST operation is used to do an image upload. To get the image details in response, run a GET API request.
-
Use the following REST API to upload image for controller.pkg.
URI :/api/image
Method:POSTroot@admin:-controller# curl -X POST -k https://10.58.3.27/api/image -u "admin:admin" -F file=@controller.pkg -
Use the following REST API to upload image for controller_patch.pkg.
root@admin:-controller-18.2.5-2p3-9002# curl -X POST -k https://10.58.3.27/api/image -u "admin:admin" -F file=@se_patch.pkg
- Use the following API to delete the image provided, if it is not in use.
delete image <image-name>System Report for Upgrade Operations
Starting with NSX Advanced Load Balancer 22.1.6, a system report is generated for every upgrade operation. This system report contains summary of the operation performed and pre-check results from the operation. Unlike the pre-checks tied to the upgradestatusinfo, system report object is retained over multiple iterations.
System Report can be viewed and downloaded using NSX Advanced Load Balancer CLI.
Login to NSX Advanced Load Balancer Controller and use the show systemreport # Prints Summary command to generate a system report.
Command: show systemreport [<report-name> [export]]
- Options:
- Default - List all available report
- Example:
show systemreport
- name:
- Show details of specified report
- Example:
show systemreport upgrade_system_20231108-142156
- export
- Export the report to specified path. [Downloads the full report]
- Example:
show systemreport upgrade_system_20231108-142156 export
Below is a sample output for generating a system report for NSX Advanced Load Balancer Controller.
[admin:10-1-1-1]: > show systemreport
+--------------------------------+----------------------+----------------------+--------------------------------------+
| Name | UUID | State | Image |
+--------------------------------+-----------------------+---------------------+--------------------------------------+
| upgrade_controller_checks_f345 | systemreport-6f499bff-f| UPGRADE_PRE_CHECK_WARNING | 22-1-1-6-43-20240115.080016 |
+--------------------------------+-----------------------+---------------------+--------------------------------------+
[admin:10-1-1-1]: > show systemreport upgrade_controller_checks_f345
+-------------------------+----------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+----------------------------------------------------------------------------------+
| uuid | systemreport-6f499bff-fcbd-4fca-a43a-483d1764eed5 |
| name | upgrade_controller_checks_f345 |
| tenant_ref | admin |
| image_ref | 22.1.6-8143-20240115.080016 |
| archive_ref | report://upgrade_controller_checks_f345.tar.gz |
| state | |
| state | UPGRADE_PRE_CHECK_WARNING |
| last_changed_time | Tue Jan 16 09:02:02 2024 ms(167792166) UTC |
| summary | |
| name | Prechecks for Controller Upgrade |
| description | System evaluation report to perform Controller Upgrade from 22.1.5 to 22.1.6 |
| previews[1] | 'Checking Controller Cluster readiness for upgrade operations.' |
| previews[2] | 'Check if upgrade operation is already in progress.' |
| previews[3] | 'Checking ServiceEngineGroup error recovery options prior to upgrade operations. |
| | ' |
| previews[4] | 'Checking active versions compatibility for upgrade operations.' |
| previews[5] | 'Checking if system in Avi-ESSENTIALS/BASIC mode and allows only permitted opera |
| | tions.' |
| previews[6] | 'Checking if the cloud api versions are compatible after upgrade.' |
| previews[7] | 'Checking ServiceEngineGroup has an ongoing upgrade operation.' |
| previews[8] | 'Checking Controller Cluster disk space for upgrade operations.' |
| previews[9] | 'Checking image version compatibility for upgrade operations.' |
| previews[10] | 'Checking idempotent operations for upgrade operations.' |
| previews[11] | 'Checking Image state across Cluster members for upgrade operations.' |
| previews[12] | 'Checking the system configuration.' |
| previews[13] | 'Checking Patch compatibility for Controller patch operations.' |
| previews[14] | 'Checking config migration.' |
| previews[15] | 'Checking and inform user to take a backup prior to upgrade operations.' |
| previews[16] | 'Checking if Docker version is compatible.' |
| previews[17] | 'Checking for the patch in image bundle.' |
| previews[18] | 'Checking if configured IP type is DHCP or STATIC.' |
| previews[19] | 'Checking if Gslb Feature is enabled and provides feature specific messages.' |
| previews[20] | 'Checking if se linux is enabled on controller nodes.' |
| previews[21] | 'Checking total number of alerts for upgrade operations.' |
| readiness_reports[1] | |
| node_ref | cluster-0-1 |
| name | cluster-0-1 |
| node_type | NODE_CONTROLLER_CLUSTER |
| system_readiness | |
| state | |
| state | UPGRADE_PRE_CHECK_WARNING |
| last_changed_time | Tue Jan 16 09:02:02 2024 ms(165028028) UTC |
| checks[1] | |
| check_code | SYSERR_CHECK_CLUSTER_STATE |
| description | 'Checking Controller Cluster readiness for upgrade operations.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:01:58 |
| duration | 0 sec |
| checks[2] | |
| check_code | SYSERR_CHECK_ACTIVE_VERSIONS |
| description | 'Checking active versions compatibility for upgrade operations.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:01:58 |
| duration | 0 sec |
| checks[3] | |
| check_code | SYSERR_CHECK_SE_GROUP_ERROR_RECOVERY |
| description | 'Checking ServiceEngineGroup error recovery options prior to upgrade operations. |
| | ' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:01:58 |
| duration | 0 sec |
| checks[4] | |
| check_code | SYSERR_AVI_ESSENTIALS_CHECK |
| description | 'Checking if system in Avi-ESSENTIALS/BASIC mode and allows only permitted opera |
| | tions.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:01:58 |
| duration | 0 sec |
| checks[5] | |
| check_code | SYSERR_CHECK_CLOUD_COMPATIBILITY |
| description | 'Checking if the cloud api versions are compatible after upgrade.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:01:58 |
| duration | 0 sec |
| checks[6] | |
| check_code | SYSERR_CHECK_SE_GROUP_UPGRADE_OPS_INPROGRESS |
| description | 'Checking ServiceEngineGroup has an ongoing upgrade operation.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:01:58 |
| duration | 0 sec |
| checks[7] | |
| check_code | SYSERR_CHECK_VERSION_COMPATIBILITY |
| description | 'Checking image version compatibility for upgrade operations.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:01:58 |
| duration | 0 sec |
| checks[8] | |
| check_code | SYSERR_CHECK_IMAGE_VERSION |
| description | 'Checking idempotent operations for upgrade operations.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:01:58 |
| duration | 0 sec |
| checks[9] | |
| check_code | SYSERR_CHECK_IMAGE_COMPATIBILITY |
| description | 'Checking Image state across Cluster members for upgrade operations.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:01:58 |
| duration | 0 sec |
| checks[10] | |
| check_code | SYSERR_CHECK_CONTROLLER_PATCH_COMPATIBILITY |
| description | 'Checking Patch compatibility for Controller patch operations.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:01:58 |
| duration | 0 sec |
| checks[11] | |
| check_code | SYSERR_CONFIGURATION_CHECK |
| description | 'Checking the system configuration.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:01:58 |
| duration | 0 sec |
| checks[12] | |
| check_code | SYSERR_MC_BACKUP_ERR |
| description | Inform User to take configuration backup prior to upgrade operations. |
| details[1] | Please take the backup before starting the upgrade operations. |
| state | UPGRADE_PRE_CHECK_WARNING |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:01:58 |
| duration | 0 sec |
| checks[13] | |
| check_code | SYSERR_DOCKER_VERSION_CHECK |
| description | 'Checking if Docker version is compatible.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:01:59 |
| duration | 0 sec |
| checks[14] | |
| check_code | SYSERR_CHECK_PATCH_IMAGE |
| description | 'Checking for the patch in image bundle.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:59 |
| end_time | 2024-01-16 09:01:59 |
| duration | 0 sec |
| checks[15] | |
| check_code | SYSERR_IP_TYPE_CHECK |
| description | 'Checking if configured IP type is DHCP or STATIC.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:59 |
| end_time | 2024-01-16 09:01:59 |
| duration | 0 sec |
| checks[16] | |
| check_code | SYSERR_GSLB_FEATURE_CHECK |
| description | 'Checking if Gslb Feature is enabled and provides feature specific messages.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:59 |
| end_time | 2024-01-16 09:01:59 |
| duration | 0 sec |
| checks[17] | |
| check_code | SYSERR_CHECK_SE_LINUX_ENABLED |
| description | 'Checking if se linux is enabled on controller nodes.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:59 |
| end_time | 2024-01-16 09:01:59 |
| duration | 0 sec |
| checks[18] | |
| check_code | SYSERR_CHECK_ALERTS |
| description | 'Checking total number of alerts for upgrade operations.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:59 |
| end_time | 2024-01-16 09:01:59 |
| duration | 0 sec |
| checks[19] | |
| check_code | SYSERR_CHECK_CLUSTER_DISK_SPACE |
| description | 'Checking Controller Cluster disk space for upgrade operations.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:01:59 |
| duration | 0 sec |
| checks[20] | |
| check_code | SYSERR_UPGRADE_OPS_IN_PROGRESS |
| description | 'Check if upgrade operation is already in progress.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:02:01 |
| duration | 2 sec |
| checks[21] | |
| check_code | SYSERR_CONFIG_CHECK |
| description | 'Checking config migration.' |
| state | UPGRADE_PRE_CHECK_SUCCESS |
| start_time | 2024-01-16 09:01:58 |
| end_time | 2024-01-16 09:02:02 |
| duration | 3 sec |
| start_time | 2024-01-16 09:01:48 |
| end_time | 2024-01-16 09:02:02 |
| duration | 13 sec |
| upgrade_ops | EVAL_UPGRADE |
| image_ref | 30.2.1-8143-20240115.080016 |
| total_checks | 21 |
| checks_completed | 21 |
+-------------------------+----------------------------------------------------------------------------------+
To download a system report, use the following export option.
[admin:10-1-1-1]: > show systemreport upgrade_controller_checks_f345
<CR> show the object
export download report
[admin:10-1-1-1]: > show systemreport upgrade_controller_checks_f345 export
Downloaded the attachment to /tmp/upgrade_controller_checks_f345.tar.gz
[admin:10-1-1-1]: > show systemreport
+--------------------------------+---------------------+-----------------------------------------------------+
| Name | State | Timestamp |
+--------------------------------+---------------------+-----------------------------------------------------+
| upgrade_controller_checks_8ec5 | SYSTEM_REPORT_ERROR | Wed Feb 21 2024 12:49:08 |
| upgrade_controller_checks_67a0 | SYSTEM_REPORT_ERROR | Tue Feb 27 2024 04:59:28 |
| upgrade_controller_checks_d3f0 | SYSTEM_REPORT_ERROR | Tue Feb 27 2024 05:05:47 |
| upgrade_controller_checks_169b | SYSTEM_REPORT_ERROR | Tue Feb 27 2024 05:08:10 |
| upgrade_controller_checks_601c | SYSTEM_REPORT_ERROR | Wed Feb 28 2024 08:46:55 |
+--------------------------------+---------------------+-----------------------------------------------------+
To delete a specific system report, use the following delete command.
[admin:10-1-1-1]: > delete systemreport upgrade_controller_checks_8ec5
Successfully deleted upgrade_controller_checks_8ec5.
[admin:10-1-1-1]: > show systemreport
+--------------------------------+---------------------+--------------------------+
| Name | State | Timestamp |
+--------------------------------+---------------------+--------------------------+
| upgrade_controller_checks_67a0 | SYSTEM_REPORT_ERROR | Tue Feb 27 2024 04:59:28 |
| upgrade_controller_checks_d3f0 | SYSTEM_REPORT_ERROR | Tue Feb 27 2024 05:05:47 |
| upgrade_controller_checks_169b | SYSTEM_REPORT_ERROR | Tue Feb 27 2024 05:08:10 |
| upgrade_controller_checks_601c | SYSTEM_REPORT_ERROR | Wed Feb 28 2024 08:46:55 |
+--------------------------------+---------------------+--------------------------+
[admin:10-1-1-1]: >
Prechecks_only Flag
Starting with NSX Advanced Load Balancer Controller 22.1.6, a new flag prechecks_only is introduced for all upgrade operations (upgrade, patch upgrade, and rollback). The prechecks_only flag performs upgrade checks prior to an upgrade maintenance window. Executing this flag does not trigger any upgrade operation and it is safe to perform upgrade operations with the prechecks_only flag outside of a maintenance window.
[admin:10-1-1-1]: > upgrade controller image_ref 22.1.6-8143-20240115.080016 prechecks_only
Previewing upgrade
+-------------+-----------------------------------------------------------------------------------------+
| Field | Value |
+-------------+-----------------------------------------------------------------------------------------+
| status_code | SYSERR_UPGRADE_OPS_PREVIEW_RESPONSE |
| status | Checks preview for upgrade operations. |
| checks | |
| | 'Checking Controller Cluster readiness for upgrade operations.' |
| | 'Check if upgrade operation is already in progress.' |
| | 'Checking active versions compatibility for upgrade operations.' |
| | 'Checking if system in Avi-ESSENTIALS/BASIC mode and allows only permitted operations.' |
| | 'Checking if the cloud api versions are compatible after upgrade.' |
| | 'Checking ServiceEngineGroup has an ongoing upgrade operation.' |
| | 'Checking Controller Cluster disk space for upgrade operations.' |
| | 'Checking image version compatibility for upgrade operations.' |
| | 'Checking idempotent operations for upgrade operations.' |
| | 'Checking Image state across Cluster members for upgrade operations.' |
| | 'Checking the system configuration.' |
| | 'Checking config migration.' |
| | 'Checking if Gslb Maintenance Mode is enabled prior to upgrade.' |
| | 'Checking if Gslb Feature is enabled and provides feature specific messages.' |
| | 'Checking and inform user to take a backup prior to upgrade operations.' |
| | 'Checking if Docker version is compatible.' |
| | 'Checking for the mandatory patch in image bundle.' |
| | 'Checking if configured IP type is DHCP or STATIC.' |
| | 'Checking if se linux is enabled on controller nodes.' |
| | 'Checking total number of alerts for upgrade operations.' |
+-------------+-----------------------------------------------------------------------------------------+
Starting upgrade
+--------------------+----------------------------------------------------------------------------------+
| Field | Value |
+--------------------+----------------------------------------------------------------------------------+
| status_code | SYSERR_EVAL_UPGRADE_CONTROLLER_STARTED |
| status | 'Pre-checks for Upgrade of Controller started. Use 'show upgrade status' to check|
| the status.' |
| system_report_uuid | systemreport-6f499bff-fcbd-4fca-a43a-483d1764eed5 |
+--------------------+----------------------------------------------------------------------------------+
Must-Checks for Upgrade
Prior to upgrade operations, various must-checks are run to check the various mandatory and optional requirements for upgrade. The outputs message is exhibited as error message or as Warning message. Warnings can be skipped while ‘Errors’ cannot be over-ridden. API/CLI provides the skip_warnings option to control the above behavior.
For CLI— This is directly integrated into the normal work-flow and there is no separate command.
For the REST API — Add /preview/ at the end of APIs to get previews for that particular flow.
Starting with NSX Advanced Load Balancer 22.1.3, in order to start upgrade operation, all the CLI upgrade request should go with skip_warning option. Without theskip_warning option, the system state for any operation would lead to PRE_CHECK_WARNING and halt.
[admin:10-10-10-1]: > upgrade system image_ref 30.3.3-7235-20230110.035149 skip_warnings
+-------------+------------------------------------------------------------------------------+
| Field | Value |
+-------------+------------------------------------------------------------------------------+
| status_code | SYSERR_UPGRADE_OPS_PREVIEW_RESPONSE |
| status | Checks preview for upgrade operations. |
| checks | |
| | Check Controller Cluster readiness for upgrade operations. |
| | Check and inform user to take a backup prior to upgrade operations. |
| | Check if se linux is enabled on controller nodes. |
| | Check if upgrade operation is already in progress. |
| | Check ServiceEngineGroup has an ongoing upgrade operation. |
| | Check image version compatibility for upgrade operations. |
| | Check ServiceEngine reachability for upgrade operations. |
| | Check ServiceEngine disk space for upgrade operations. |
| | Check Controller Cluster disk space for upgrade operations. |
| | Check and inform Virtual Service(s) disruption for upgrade operations. |
| | Check idempotent operations for upgrade operations. |
| | Check active versions compatibility for upgrade operations. |
| | Check ServiceEngineGroup error recovery options prior to upgrade operations. |
| | Check Image state across Cluster members for upgrade operations. |
| | Checks for the patch in image bundle. |
| | Checks if Gslb Feature is enabled and provides feature specific messages. |
| | Checks the system configuration. |
| | Check total number of alerts for upgrade operations. |
| | Checks if the cloud api versions are compatible after upgrade. |
| | Checks if Docker version is compatible. |
| | Checks if configured IP type is DHCP or STATIC. |
| | Checks if se has a valid license state. |
+-------------+------------------------------------------------------------------------------+
Starting upgrade
+-------------+-----------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------+-----------------------------------------------------------------------------------------------------------+
| status_code | SYSERR_UPGRADE_SYSTEM_STARTED |
| status | 'Upgrade of System (Controller + All SEGroup(s)) started. Use 'show upgrade status' to check the status.' |
+-------------+-----------------------------------------------------------------------------------------------------------+
Similarly, Use the skip warning option while performing the patch upgrade.
[admin:10-10-10-1]: > patch controller controller_patch_ref 23.1.1-7189-2p1-20221216.192828 skip_warnings
Previewing upgrade
Upgrading System (Controller and SE Groups)
The configuration and placement of virtual services are blocked if it is a system-level upgrade till all the Service Engines are upgraded. Once these operations are completed, configuration on the Controller (except the configuration of virtual service and VIP) is allowed, irrespective of the SE group upgrade status.
Note: It is recommended to increase the default timeout value from 90 seconds to 120 seconds before performing upgrade. This is to avoid upgrade going to timeout.
Using CLI
Notes:
- The auto-suggest option in the CLI provides available values on pressing tab on your keyboard.
skip_warnings— Use this option to skip any warnings and optional must checks.
The following are the various options available for the system upgrade.
- Use the
upgrade system image_ref <image name>command to upgrade the system to a base image.[admin:-controller]: >upgrade system image_ref 18.2.6-9000-20191031.063017 - Use the following to upgrade the system to a base image and a controller patch.
[admin:-controller]: >upgrade system image_ref 18.2.6-9134-20191101.042535 controller_patch_ref 18.2.6-9134-2p1-20190806.011824 - Use the following to upgrade the system to a base image and an SE patch.
[admin:-controller]: >upgrade system image_ref 18.2.6-9134-20191101.042535 se_patch_ref 18.2.6-9134-2p1-20190806.011824 - Use the following to upgrade the system to a base image, the Controller patch, and an SE patch
[admin:-controller]: >upgrade system image_ref 18.2.6-9134-20191101.042535 controller_patch_ref 18.2.6-9134-2p1-20190806.011824 se_patch_ref 18.2.6-9134-2p1-20190806.011824
SE Upgrades
The Controller allows you to pick up the number of SE-groups per Controller node.
seupgrade_fabric_pool_size: This property allows the Controller to pickup number of SE groups per Controller to upgrade.
For instance, if seupgrade_fabric_pool_size is set to 3, three SE-groups are picked up per Controller, that means 9 SE groups across the cluster.
The default value of seupgrade_fabric_pool_size is 20. However, you can update this based on the requirement or the load.
seupgrade_copy_pool_size: This parameter defines the number of simultaneous SE image downloads in a SEGroup. It is used to pace the SE downloads so that Controller network/ CPU bandwidth is a bounded operation. A value of zero will disable the pacing scheme and all the SE(s) in the SEGroup will attempt to download the image.
seupgrade_copy_pool_size = n, where ‘n’ is the number of SE within SE group will be picked for copy.
For instance, if seupgrade_copy_pool_size = 3, the three SE in a picked up SE group will be picked for copy.
The default value of seupgrade_copy_pool_size is 5. However, you can update this based on the requirement or the load.
The following are the steps to configure this:
- Configure the Controller properties.
- Set
seupgrade_fabric_pool_size <number>. - Set
seupgrade_copy_pool_size <number>. - Save.
[admin:ctrl]: > configure controller properties
[admin:ctrl]: controllerproperties> seupgrade_fabric_pool_size 2
Overwriting the previously entered value for seupgrade_fabric_pool_size
[admin:ctrl]: controllerproperties> seupgrade_copy_pool_size 2
Overwriting the previously entered value for seupgrade_copy_pool_size
[admin:ctrl]: controllerproperties> save
[admin:ctrl]: > show controller properties |grep pool
| seupgrade_fabric_pool_size | 2 |
| seupgrade_copy_pool_size | 2 |
[admin:ctrl]: >
Using REST API
Image UUID can be obtained by Use the GET /api/image to obtain Image UUID.
The following are the various REST API options available for the system upgrade.
- Use the following API to upgrade the system to a base image.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'system': true } - Use the following API to upgrade the system to a base image and a controller patch.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'controller_patch_uuid': 'image-e3aaad68-5aaf-485a-8bd9-1db3ec562d6a', 'system': true } - Use the following API to upgrade the system to a base image and an SE patch.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'system': true, 'se_patch_uuid': 'image-e3aaad68-5aaf-485a-8bd9-1db3ec562d6a', 'skip_warnings': True } - Use the following API to upgrade the system to a base image, the Controller patch, and an SE patch:
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'controller_patch_uuid': 'image-e3aaad68-5aaf-485a-8bd9-1db3ec562d6a', 'system': true, 'se_patch_uuid': 'image-e88aaad68-5aaf-485a-8bd9-1db3ec562d6a' }
Upgrading Controller
Using CLI
Login to the NSX Advanced Load Balancer shell prompt and use the following upgrade commands for various options.
-
Use the
upgrade controller image_ref <image name>command to upgrade the Controller to a base image.[admin:-controller]: >upgrade controller image_ref 18.2.6-9000-20191031.063017 -
Use the
upgrade controller image_ref <image name>controller_patch_ref <patch name>command to upgrade the Controller to a base image and the Controller patch.[admin:-controller]: >upgrade controller image_ref 18.2.6-9134-20191101.042535 controller_patch_ref 18.2.6-9134-2p1-20190806.011824
Using REST API
- Use the following API to upgrade the Controller to a base image.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4' } - Use the following API to upgrade an Controller to a base image and the Controller patch.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'controller_patch_uuid': 'image-e3aaad68-5aaf-485a-8bd9-1db3ec562d6a' }
Note:
Upgrading SE Group
This interface is used to upgrade all or some of the SE groups.
Using CLI
Login to the NSX Advanced Load Balancer shell prompt to use the various options available for SE group update.
- Use the
upgrade segroup se_group_refs Default-Group image_ref<image name>command to upgrade an SE group to the Controller image.[admin:-controller]: >upgrade segroup se_group_refs Default-Group image_ref 18.2.6-9134-20191101.042535 -
Use the
upgrade segroup se_group_refs Default-Group image_ref <Controller image> se_patch_ref <SE patch name>command to upgrade an SE group to the Controller image and the SE patch image.[admin:-controller]: >upgrade segroup se_group_refs Default-Group image_ref 18.2.6-9134-20191101.042535 se_patch_ref 18.2.6-9134-2p1-20190806.011824
Using REST API
SE Group UUID can be obtained by the GET /api/serviceenginegroup API.
The following are the additional options for SE group upgrade:
-
Disruptive — This is used to disable non-disruptive mechanism to facilitate a faster upgrade. If enabled, the SE(s) are upgraded in a disruptive manner. The default value is false.
-
Suspend-on-failure — This option suspends the upgrade of subsequent SE(s) within a SE-group when a failure is encountered in the SE upgrade path. The default value is false.
The following are the different APIs for the SE group upgrade:
- Use the following API to upgrade the SE group to the Controller image.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'se_group_uuids': [ 'serviceenginegroup-e553b1a6-4851-4e82-ad12-cecc4bbda6c7' ] } - Use the following with the additional SE Group options — Disruptive and Suspend-on-failure.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'se_group_uuids': [ 'serviceenginegroup-e553b1a6-4851-4e82-ad12-cecc4bbda6c7' ], 'disruptive':true, 'suspend_on_failure': true } - Use the following API to upgrade the SE group to the Controller image and the SE patch image.
API:/api/upgrade
Method:POST
JSON Data:{ 'image_uuid': 'image-b8adc2bd-d27f-469d-b78d-5e2bc14a14e4', 'se_patch_uuid': 'image-e3aaad68-5aaf-485a-8bd9-1db3ec562d6a', 'se_group_uuids': [ 'serviceenginegroup-e553b1a6-4851-4e82-ad12-cecc4bbda6c7' ] }
Additional Options for SE Group Upgrade
The following upgrade options are available for upgrading SE group.
| Option | Behaviour | Notes |
|---|---|---|
| SUSPEND_UPGRADE_OPS_ON_FAILURE | This option is used to suspend the upgrade-operations (Upgrade/ Patch) on SE-Group if the SE(s) hit an issue and does NOT come up during the upgrade operations. | It is enabled by default. This option serializes the SE upgrades in the SE group upgrade. It increases the overall upgrade time for the entire SE group. Batch size is used to decrease the upgrade time. Even if the SEs does not have scaled-out virtual services, it still upgrades serially. |
| CONTINUE_UPGRADE_OPS_ON_FAILURE | This option is used to continue the upgrade or patch upgrade operations on SE group even if the SE(s) hit an issue and does not come up during the upgrade operations.
Service disruption can be observed. |
This option parallelizes the SE upgrade in the SE group upgrade if SEs does not have scaled-out virutal services.
If SEs have scaled-out virtual services, then it continue with serial upgrades. |
| Disruptive | This option is used to disable the non-disruptive nature of SE upgrade.
It is used to upgrade all the SE(s) in the group to the next version irrespective of the traffic disruption. |
This option is disabled by default.
All SE(s) will be upgraded in parallel, irrespective of scaled out virtual service existence. Traffic/Service disruption will take place. |
Upgrading using Patch Release
The followings are the available options for patch upgrade:
- System — Patch upgrade for the Controller and all SE groups
- Controller — Patch upgrade for the Controller alone.
- SE group — Patch upgrade for some or all the SE groups.
Notes:
The following are a few points for a patch upgrade process:
- An image along with a patch can be applied.
- The image and the patch must have the same base version.
-
A patch cannot be applied without applying the image.
- Compatibility checks prevent incorrect patches from getting applied to different versions.
Patch Upgrade for NSX Advanced Load Balancer System
Use the following CLI command for the base image upgrade with a patch image.
[admin:controller]: > upgrade system image <image-name> controller_patch <controller-patch-name> se_patch <se-patch-name>
[admin:controller]: > upgrade system image 18.2.6 controller_patch 18.2.6-1p1 se_patch 18.2.6-1p1
- Use the
upgrade system image_ref <image name > controller_patch_ref <SE patch name>command for NSX Advanced Load Balancer system upgrade with a Controller patch.[admin:-controller]: upgrade system image_ref 18.2.6-9000-20191031.063017 controller_patch_ref 18.2.6-2p1-20191031.063017 - Use the
upgrade system image_ref <image name> se_patch_ref <SE patch name>command for NSX Advanced Load Balancer system upgrade with only SE patch.[admin:-controller]: upgrade system image_ref 18.2.6-9000-20191031.063017 se_patch_ref 18.2.6-2p1-20191031.063017 - Use the
upgrade system image_ref <image name> controller_patch_ref <Controller patch image> se_patch_ref <SE patch image>command for the system upgrade with both Controller and SE patch.[admin:-controller]:upgrade system image_ref 18.2.6-9000-20191031.063017 controller_patch_ref 18.2.6-2p1-20191031.063017 se_patch_ref 18.2.6-2p1-20191031.063017
Patch Upgrade for Controller
This interface is used to patch upgrade for the Controller.
Using CLI
Use the upgrade controller image_ref <image name> controller_patch_ref <Controller patch image command to upgrade the Controller with a patch.
[admin:-controller]: upgrade controller image_ref 18.2.6-9000-20191031.063017 controller_patch_ref 18.2.6-2p1-20191031.063017
[admin:controller]: > patch controller <patch-name>
[admin:controller]: > patch controller controller_patch 18.2.5-5p1 Using REST API
POST api/upgrade JSON data:{‘controller_patch_uuid’: <image-uuid>} Patch Upgrade for SE Group
SE groups can be of different versions and different versions of patch can be applied.
Use the upgrade segroup image_ref <image name> se_group_refs Default-Group se_patch_ref <patch for the SE Group> command to upgrade specific SE groups along with a patch.
[admin:-controller]: upgrade segroup image_ref 18.2.6-9000-20191031.063017 se_group_refs Default-Group se_patch_ref 18.2.6-2p1-20191031.063017
Note: Patch name and patch uuid is retrieved from the image service.
Rollback
The rollbacks are non-disruptive in nature.
When a rollback operation is performed, the Controller or SEs will transition to the previous major version of the software. Selective rollback is possible for the Controller and SE groups.
The following options are available:
- Rollback for System
- Rollback for the Controller only
- Rollback for some or all the SE groups
Note: Rollback of the SE Group will be to the previous version.
Rollback for System
Rollback of the system will result in the rollback of the SE(s) followed by the rollback of the Controller. Use the following CLI and REST API for performing rollback for a patch version for system (Controller and SE groups).
Using CLI
[admin:controller]: > rollback system Using REST API
POST api/rollback JSON data:{‘system’:true}
POST api/rollback JSON data:{‘system’:true,‘rollback_type’:2} Rollback for Controller
This interface is used to rollback the Controller.
Using CLI
[admin:controller]: > rollback controllerUsing NSX Advanced Load Balancer REST API
POST api/rollbackRollback for SE Groups
Using CLI
[admin:controller]: > rollback segroup <se-group-name>
[admin:controller]: > rollback segroup seg-a
Using RESt API
POST api/rollback JSON data:{‘se_group_uuids’: [‘seg-a-uuid’]}Rollback - Patch
Rollback of a patch release transitions the software to a version without the specific patch. It will NOT roll back to the previous major version.
Selective ability to rollback the patch on the Controller and SE groups is available.
This interface is used to roll back the patch and not the major version.
The following are the available options:
- System: rollback patch for Controller and all SE groups.
- Controller: rollback patch the Controller only.
- SE-group: rollback patch for all or some of the SE groups.
Rollback Patch for System
Use the following CLI and REST API for performing rollback for System (Controller and SE groups).
Using CLI
[admin:controller]: > rollbackpatch systemUsing REST APIs
POST api/rollback JSON data:{‘rollback_type’:2}Rollback Patch for Controller
Use the following CLI and REST API for performing rollback for a patch version for the Controller.
Using CLI
[admin:controller]: > rollbackpatch controller
Using REST APIs Add here
Rollback Patch for SE Groups
Use the following CLI and REST API for performing rollback for a patch version for an SE group.
Using CLI
[admin:controller]: > rollbackpatch segroup <se-group-name>
[admin:controller]: > rollbackpatch segroup seg-a
Using REST APIs
POST api/rollback JSON data:{‘rollback_type’:2,‘se_group_uuids’: [‘seg-a-uuid’]} Notes: Refer to Additional Options for Flexible Upgrade for the following additional options:
- Rollback - Error Recovery
- Abort Cleanup
- SE Group Resume Option
Show Commands
The following show commands provide software version visibility in the system:
show version controllershow version serviceengineshow version serviceenginegroup
The following commands provide upgrade visibility in the system.
show upgrade status: Various filters will be implemented as per UI work-flow.show upgrade history: This command is deprecated.
Notes:
- The Controller will be at the highest version while the SE groups may be at lower versions. Certain commands may not work due to the Controller version being at the highest version.
- Due to the API version semantics, certain fields may not be available as they are deprecated in annotation.
- Due to API endpoint deprecation, some internal commands may not work.
Alerts and Events
The following events are available to provide visibility:
- Image upload/delete events
- Upgrade-specific events
- Patch-specific events
- Rollback-specific events
- Rollback patch-specific events.
- Failures will translate into alerts.
Additional APIs
The following GET API calls are applicable:
-
The following REST API provides information about all the images present in the system.
Get API: api/image/ -
The following API provides information about a specific image whose UUID is passed as a slug.
Get API: api/image/image-uuid - Use the following API to delete the image provided if not in use.
Delete API: api/image/image-uuid - Inventory API —api/image-inventory. This API provides the image inventory on the system. It provides filtering based on various options such as retrieve all packages for a version etc.