Updating and Upgrading SAS Viya: Applying hot fixes

[#1]

With SAS Viya the concept of hot fixes established in previous versions of SAS will continue. SAS provides hot fixes to previously shipped software. A hot fix is created to resolve a number problems in released versions of our software. So, with the adoption of SAS Viya increasing, and the upcoming release of Viya 3.4, it seems like a good time to talk about upgrades and updates in the Viya world.

Firstly, lets look at terminology, in Viya the terms update and upgrade are related but different.

You can update your deployed software environment in order to get the environment’s software to the latest version. As an example, when hot fixes are released for a version of Viya, such as 3.3, you would update your software to get the hot fixes and any other updates to the SAS executables. An update does not require a new software order.

An upgrade adds significant feature changes or improvements to your deployed software. An upgrade is performed to move from one version of Viya to another, for example from 3.3 to 3.4. You will need a new software order to upgrade your deployed software, and an upgrade will most likely require changes to the deployed software’s configuration.

In the first of two posts on this topic I will look at the simpler scenario, updates. One of the most common reasons for doing an update is the availability of a hot fix. The method of staying informed about available hot fixes has not changed from SAS 9.4. For more details see Chris Hemedinger's article on how to stay informed about SAS hot fixes.

I recently updated the deployment we use for the administration section of the GEL Viya Implementers workshop to apply some SAS Environment Manager hot fixes. If you were used to applying hotfixes in SAS 9.4 be prepared for some changes. Unlike SAS 9.4 to apply a hot fix you do not need to specifically download a package of the hotfix. Since SAS Viya software is delivered via RPM's then the same process that was used to install the software can be used to apply updates.

The update process brings your deployed software up-to-date with the latest compatible software. You perform the update with the same command that was used to install SAS Viya, and you use the same software order and the same playbook. Running the playbook updates all software to the latest version (no partial updates). To perform the update process, you must have administrator privileges and sudo access for the machines.

Something to be aware of in relation to Viya is that the nature of the process, where updates are performed with yum via ansible, makes it much easier to do an accidental update. A yum update will get you the latest software, for this reason setting up a mirror repository is a good practice.

The Viya 3,3 deployment that I updated has the Viya services split across 4 separate machines and has a single SMP CAS server. In this scenario, where we need to run commands on each machine in the deployment, ansible is your friend. The process involves comparing the RPM versions on each machine on the deployment before and after the update is installed. To automate this part of the process we created two ensile playbooks.

• gel.compare.update.before.yml: stores the state of the SAS Viya RPM's and repositories prior to the update
• gel.compare.update.after.yml: stores the state of the SAS Viya RPM's after the update and compares them to the before state.

The full process to apply the hot fix is as follows (the storing and comparing rpm states can be automated using the ansible playbooks).

Record the current version of the RPM packages installed

The first step in the process is to record the version of the RPM files currently installed so that we can use them to compare the current deployment to the updated deployment at the end of the process. To do this execute yum list installed on each machine. The command produces a list of all packages installed on each machine that include the string "sas-" in the name of the repository.

yum list installed | sort | grep sas- > /tmp/yum.before.update.txt

The output shown below lists package name, version and release date, and the repository.

You can also use the rpm -qa command to list installed packages.

rpm -qa | sort | grep sas- > /tmp/rpm.before.update.txt

The output of the rpm command is shown below.

Check for user updates in _deployment files

Any Configuration files that include the string _deployment may be overwritten by the update process. These files should not have been updated, however if they have updates, they should be identified and saved for re-implementation. Use the following command to identity all the "*_deployment" files and check them for any changes made by users.

ansible sas-all -b -m shell -a "find /opt/sas/viya -name '*_deployment*' 2> /dev/null -exec ls -al {} \; | grep '_deployment'"

The output below shows a partial list from one of the Viya machines.

Update the mirror

If you are using a mirrored repository you need to re synchronize the mirror prior to performing the update. The re synchronize makes the new software available to the mirrored repository so that it can be installed in the environment.

Use ansible to run the update and apply the hot-fix

To update the software and apply the hot-fixes use the same ansible command that was used to deploy the software. In many cases this step will stop and restart Viya services, as result it is recommended that an outage is scheduled during the update.

ansible-playbook site.yml -i inventory.ini -e "@repo_override.txt"

Record the updated versions of the RPM packages

For comparison purposes record the versions of the RPM's after the update.

yum list installed | sort | grep sas- > /tmp/yum.after.update.txt

rpm -qa | sort | grep sas- > /tmp/rpm.after.update.txt

Compare the before and after for the RPM'S

The playbook runs the commands above on each machine, concatenates all the files and then compares the state of the deployments before and after the update using the diff command.

diff /tmp/yum.before.update.txt /tmp/yum.after.update.txt > /tmp/diff.txt

The output of the diff command comparing the before and after state of the RPM's shows that (among other changes) the SAS Environment Manager RPM versions were updated from version 1.3.31 to 1.3.34 with the date changing from November 20017 to March 2018, the date the hot fix was released.

The diff command shows what files have been changed by the application of the update. It would be nice to see from the hot fix documentation if that matches the expected changes based on the update that is being applied. This would make the validation more robust. Maybe in a later release.

Before you start this process I strongly recommend two things:

• Create a full backup of your Viya Deployment
• Develop a test plan which you will execute to determine the success of the update.

When the process is completed execute your test plan, testing access to applications and any content that has been developed in the environment.

The documentation for the update process can be found in the SAS Viya 3.3 Administration guide. Overall the update process was relatively painless for me. I did have a lot of help though. Thank you Erwan Granger and Gilles Chrzaszcz.

[alancox]

Hi Gerry,

the links to the two ansible playbooks in your informative blog don't work for me (gel.compare.update.before.yml & 
gel.compare.update.after.yml)

Could you check them please?

thanks

Alan

[alancox]

Hi Gerry, could you provide some more information about what goes into the "repo_override.txt" file mentioned here?

ansible-playbook site.yml -i inventory.ini -e "@repo_override.txt"

 Is it to replace the "REPOSITORY_WAREHOUSE: " value that is in the "vars.yml" used for deployment, or something else? This would make sense if a new repo was downloaded to a new location rather than overwriting the old one.

I can't see it mentioned in the blog & the online doc says simply: "When you initiate the update, use the same command and options that you used when you performed the initial deployment"

The only reference to an additional input file is if extra CAS Workers were added

thanks

Alan

[GerryNelson]

Alan,

The blog was written about 3.3 updates. The repo_override was used in 3.3 to point to a different mirror. It was actually replaced in 3.4 by he "REPOSITORY_WAREHOUSE: " value that is in the "vars.yml". The repo_overrode file is documented here in the 3.3 documentation. 

Gerry

[itavco]

Hi Gerry,

it is possible to install only some hotfixes for example by updating or installing only some rpm?

es:

yum update sas-sasvisualanalytics-1.7.89-20190710.1562769564867.x86_64.rpm
yum update sas-report-data-1.10.74-20190628.1561729073993.x86_64.rpm

...

....

Thank you 

Vincenzo
 

[GerryNelson]

Hi Vincenzo,

In Viya 3.4 there was an "update with yum" option. See the documentation here:

https://go.documentation.sas.com/?cdcId=calcdc&cdcVersion=3.4&docsetId=dplyml0phy0lax&docsetTarget=n...

But that is not an option in Viya 3.5

Gerry

[itavco]

Thank you Gerry,

but suppose I wanted to install only this two hotfix:
64155 The sasep-admin script fails with an error on Amazon EMR clusters July 25, 2019 Standard View
64002 The SAS® Embedded Process does not support Hive transactional tables
Red Hat Enterprise Linux 6:
sas-sepcorehadp-16.1.1-20190719.120321556729.x86_64.rpm
sas-hadoopmrep-1.6.16-20190716.1563309940147.x86_64.rpm

I can run only :

yum update sas-sepcorehadp-16.1.1-20190719.120321556729.x86_64.rpm
yum update sas-hadoopmrep-1.6.16-20190716.1563309940147.x86_64.rpm

but suppose I wanted to install only this two hotfix:
64155 The sasep-admin script fails with an error on Amazon EMR clusters July 25, 2019 Standard View
64002 The SAS® Embedded Process does not support Hive transactional tables
Red Hat Enterprise Linux 6:
sas-sepcorehadp-16.1.1-20190719.120321556729.x86_64.rpm
sas-hadoopmrep-1.6.16-20190716.1563309940147.x86_64.rpm

I can run only :

yum update sas-sepcorehadp-16.1.1-20190719.120321556729.x86_64.rpm
yum update sas-hadoopmrep-1.6.16-20190716.1563309940147.x86_64.rpm.

[GerryNelson]

You could do this, but it is not supported, it might work in your case. However, it is recommended to always use the playbook since some components require configuration that installing a package with yum/rpm isn't going to cover.

[itavco]

OK all clear
Thank you very much

V.