Upgrading a SAS Viya Deployment
In a previous post, I talked about updating SAS Viya. In this one I will look at upgrades. An upgrade adds significant features, changes, or improvements to your deployed software. An upgrade is performed to move from one version of Viya to another, for example from 3.3 to 3.4. The process to do an upgrade is similar to an update. In this post I will look at upgrading from Viya 3.3 to Viya 3.4
NOTE: A change in terminology occurred with the release of Viya 3.4 second edition. This release introduced significant new functionality as a result you need to generate a new order to move an existing Viya 3.4 version to the 3.4 second edition. To quote the documentation “there are two types of upgrade associated with SAS Viya 3.4 software. You can upgrade products in SAS Viya 3.4, such as SAS Econometrics 8.3 to SAS Econometrics 8.4. You can also upgrade to SAS Viya 3.4 from earlier versions of SAS Viya, such as SAS Viya 3.2 or SAS Viya 3.3.” The process for upgrading products within Viya 3.4 has its own section in the documentation “Adding SAS Viya Software to a Deployment and Upgrading Products in SAS Viya 3.4” This blog was written prior to the second edition relase and deals with upgrading from earlier versions of Viya (3.1, 3.2, 3.3) to Viya 3.4.
Upgrade to Viya 3.4 is supported from 3.2 and 3.3 and additional software can be added to the deployment during an upgrade. Upgrade is designed to maintain all content created in the previous version of Viya, so any reports, data plans or models will be preserved.
Step 1 - Plan your Upgrade
You will need a new software order to upgrade your deployed software and an upgrade will most likely require changes to the deployed software’s configuration. You will need to plan for some downtime, upgrading SAS Viya software will require an outage period because SAS Viya services are stopped and restarted automatically during the upgrade process.
You should create a full system backup of your deployment prior to performing an upgrade, there is no rollback from an upgrade, failed or not, so a good backup is critical. In addition, you should also make sure the environment you are upgrading meets all the system requirements for the new version of the software. System requirements do change between releases.
Check the documentation for any manuals steps required to be performed in the current environment prior to the upgrade.
Step 2 - Prepare for the Upgrade
Merge Ansible Files
From your new software order, using the orchestration cli, generate a new playbook and store it in a new location. Make sure you keep a copy of the old playbook, you will need it. Now that you have your new playbook you need to merge the settings from your existing playbook into the new playbook. You must do this to preserve the customizations you made when you deployed the software, and to include any new default settings provided by your new order.
The two important files are:
- vars.yml which contains deployment variables that enable you to customize your deployment to meet your needs.
- inventory.ini which specifies the machines to be included in a deployment and the software to be installed on them.
First, let us look at vars.yml. You can compare the files in a way you are comfortable with. You can use the LINUX diff command, or there is s a nice Linux utility called Meld. There is a Windows version of Meld also, or you can use other Windows tools like WinMerge.
We do the comparison so that we can maintain any customizations you made in the original deployment, and also set values for any new variables in the new vars.yml.
The documentation recommends you use three files in the comparison.
- vars_original.yml the 3.3 vars.yml before any edits were made.
- vars_current.yml the 3.3 vars.yml used to deploy the software
- vars.yml the new 3.4 file for the upgrade
There is no automated way to merge the files. Whatever tools you use, you will still have to go item by item through the comparison and decide what to do in each case. The decisions you make are based on your knowledge of the existing deployment, and of the requirements for the upgraded deployment.
Comparing the vars_current.yml (used to deploy 3.3) with the new vars.yml (before any customizations are made).
diff /root/sas_viya_playbook_33/vars_current.yml /root/sas_viya_playbook/vars.yml
When comparing the variables and values in the old and new vars.yml files a number of scenarios can occur:
- DEPRECATED: In vars_current.yml and not vars.yml: deprecated variables, these should not be brought forward to the new customized vars.yml for 3.4
- NEW: In the new vars.yml for 3,4 but not in 3.3 file (vars_current.yml). New settings for 3.4 set appropriately
- When there are existing variables with different values in the two files, the person doing the upgrade has to make an intelligent decision to update the values for the variable. The choices could be:
- EXISTING-KEEPOLD keep the value from 3.3
- EXISTING-KEEPNEW keep the new default value
- EXISTING-MERGE some variables are more complex and will have to be merged.
Examples always help, right? Each row in the table below has a variable and the scenario for that variable. There are the merge decisions made when upgrading the GEL Administration environment.
The rest of the changes in my vars.yml were the addition of comments and other optional changes.
Next we need to merge the inventory files. To do that compare the old and new inventory.ini files. Ansible uses an inventory file to specify the machines to be included in a deployment and the software to be installed on them.
Each inventory file consists of two parts:
- deployment target definition : a specification of each machine on which SAS Viya software will be deployed
- host group assignment which lists a mapping of the installable groups of software and the machines on which they will be deployed.
The deployment target definition is likely to be the same in the upgraded environment, so you can copy that section of the file. However, host groups will probably change with new ones being added, and potentially old ones being removed.
This picture shows the start of the comparison of the old file (left) to the new file using meld.
meld "/root/sas_viya_playbook_33/inventory.ini" "/root/sas_viya_playbook/inventory.ini"
You can use the arrows in meld to implement the changes in the new inventory.ini file. The tool can help you make these changes but you need to be careful during the process and check what you are doing. For example, in the screenshot below if the arrow was used to make the change the StudioViya host group would be removed from the inventory.ini.
When editing the host groups make the following changes.
- Where a host group exists in the new inventory file make sure you allocate the same machines as you had in the old inventory file.
- If a new host group has been added, consider where it should be placed and add that machine to its host group. In my comparison I had two new host groups StudioViya and GraphBuilderServices.
- Keep any comments that have been added to the new inventory file.
Note: Because of a change in the structure of the host groups between releases, ensure that the inventory.ini file for the upgrade also includes the machine in the [Operations] host group in the [CommandLine] host group.
You should also make sure you transition any changes you made in the ansible,cfg file to the new ansible.cfg file.
The next step is to record the current versions of your software. This is a process I discussed in my previous article where you can find the detailed steps. In this article I am not going to detail that process again.
Manual Preparation Steps
The deployment guide section on upgrading lists quite a few manual steps that need to be performed to prepare for an upgrade. Read the document carefully to determine what manual steps are required in your environment. The steps vary based on the whether your source environment is 3.2 or 3.3 and also what software you have licensed and how it was configured (HA, multi-tenant etc). A few of the more common tasks are listed here and the details of how to do them are in the deployment guide:
- Delete the Default Backup Schedule Job
- Verify the Health of the SAS Infrastructure Data Server
- Remove Repositories
- maintain continuity for auditing after the upgrade (Prepare the [Operations] Host Group for Upgrade)
A final step before the upgrade. Run the system-assessment.yml too assess the readiness of your system for deployment of the new software. Prior to deploying fix any issues that are uncovered.
Step 3 - Upgrade
With all the preparation done the actual upgrade should now be simple. To upgrade the software use the same ansible command that was used to deploy the software. In many cases this step will stop and restart Viya services, as result it is recommended that an outage is scheduled during the upgrade.
ansible-playbook site.yml -i inventory.ini
Step 4 - Post Upgrade Manual Steps
Similar to the pre-upgrade manual steps there may be post upgrade manual steps. Check the deployment guide to determine which ones apply to your deployment.
A common post upgrade manual step is the need to deregister obsolete services. As part of the upgrade process, several services from your old SAS Viya environment will be removed or replaced with other microservices. It is possible that these obsolete microservices might not be fully deregistered from Consul. The Consul log will repeatedly record messages about these failing services. In addition, you will see old instances of the obsolete services in the dashboard and machines area of SAS Environment Manager. This is because they are still registered in consul. Complete the deregistration steps to remove the services.
An additional manual step, listed in the upgrade section that is required is to run a script(add_new_caslib_controls.sh) which makes required changes to permissions on Viya CASLIB's to ensure that internal accounts have the correct level of access.
The documentation also recommends you run an immediate backup of the system. This will be your baseline backup on Viya 3.4. Any backups made prior to the upgrade should not be used and can be deleted.
Step 5 - Test, Test, Test
Ok, your ansible playbook ran, your manual steps are completed. You are not done, the deployment needs to be thoroughly tested. You should also test any existing content such are reports, plans, projects or models. In addition you will need some education of your user community as there are some significant changes with Viya 3.4. Users will notice immediately that SASHome is gone and has been replaced by SASDrive.
The upgrade process for SAS Viya does require some attention to detail. There are quite a few manual steps both prior to and after the upgrade. If you pay attention to the details and plan and prepare for the process, you can successfully upgrade to the latest version of SAS Viya.
This article was a team effort. I would like to thank Erwan Granger, Cindy Taylor, Dawn Schrader, Connie Robison, and Jawna Gardner for their invaluable input.
