CI/CD-ing SAS Viya configuration files

[#1]

Some SAS Viya configuration files can be stored in Git to leverage version control, continuous integration and delivery, workflow branching, and other capabilities offered by Git. This approach could simplify the process of making updates to the Viya configuration and rolling out the changes to one or more servers and/or environments. Consider the application of this CI/CD approach to Viya configuration files; configuration changes can be freely made, continuously integrated, and deployed as and when needed (automatically if required). In this post, we will explore the CI/CD approach to SAS Viya administration, demonstrating the process with an example of moving the CAS configuration file to a GitLab repository, making updates to the file, and then deploying it so that the changes take effect in the Viya platform automatically.

Overview

Continuous Integration/Continuously Delivery (CI/CD) is a concept we will be hearing a lot more of ahead of the release of Viya 2020. CI/CD is a philosophy wherein the steps involved in making changes to and then deploying code are streamlined so that they can be performed freely, continuously and automatically. Git is one of many tools that can help us with this.

Configuration files can be thought of as a form of source code. This suggests that, like code, a configuration file can be version controlled. It also suggests that version-controlled configuration files can be set up for distributed development; that is, multiple people can request changes to the 'master' version of the configuration file, with a record of all changes tracked automatically.

There are a number of ways to address this use case. We will use GitLab as our SCM (source code management) system, but there are alternatives that are also widely used. There are also several options when it comes to deploying a file to where it's needed. We could use simple git commands to pull the latest file(s) from our repo manually, or we can automate that process using CI/CD tools like Jenkins. In this post, we'll use GitLab's own built-in CI/CD functionality.

Preparing for Continuous Integration

In my lab environment, the CAS server is deployed with one controller and two workers. The CAS configuration file, casconfig_usermods.lua is located in /opt/sas/viya/config/etc/cas/default. It's very important to remember that casconfig_usermods.lua can be customized, but casconfig.lua should not be modified. This file contains the majority of your CAS configuration settings, and it is created at deploy time. If you wish to change these settings, the recommended approach is to instead update your vars.yml file and re-run the Ansible playbook to apply the changes. This assures the configuration will persist if you need to deploy again at any point in the future. Any changes made to casconfig_usermods.lua manually will be retained any time CAS is upgraded.

As a preliminary step, we should back up the CAS configuration file. casconfig_usermods.lua is empty immediately after a deployment, but in a real customer environment, the file may contain customizations and it should therefore always be backed up before making changes.

We also need a Git repository in which to put the configuration file. In my lab, I am running GitLab Community Edition in a Docker container, and I simply created a new repository to try out this use case. A quick note on branching; it is generally good practice to store the 'production' version of the file in the master branch and create additional branches for 'development'. If a configuration setting needs to be added or adjusted, a SAS administrator or another 'developer' could make the change in a separate branch from the master. This configuration could be tested and when ready, merged into the 'production' master branch. This also becomes important later when deploying the configuration file. Generally, development branches would need to be reviewed before they are deployed, whereas changes in the master branch might be OK to deploy automatically. We'll look at an example of this later in this post.

Move the casconfig_usermods.lua to the GitLab repository. This can be done in a number of ways. In my lab, I created a clone (an exact copy of the repository created as a local directory) on the CAS controller, copied the file to the clone, committed it to the local repository, and then pushed it to the remote repository.

git clone http://sascas03.example.com/root/sasconfiguration.git /etc/sas/admin/configfiles
cp /opt/sas/viya/config/etc/cas/default/casconfig_usermods.lua /etc/sas/admin/configfiles/
cd /etc/sas/admin/configfiles/
git add casconfig_usermods.lua
git commit casconfig_usermods.lua -m "added cas configuration file"
git push

Use your browser to make sure the file is in the remote repository.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

Now, the CAS server will look for casconfig_usermods.lua the default location on startup, but we want to use the latest version of the file from the GitLab repo. We have a couple of options to address this. We could create a symbolic link in this location to the file that is now in the clone. Of course, this means the symlink would have to persist, and it also means that any changes made to the file in the remote repository would have to be pulled (either automatically or manually) so that CAS starts with the settings from the 'master' version of the configuration file. As per the documentation, any changes made to the CAS configuration file must also be made to other CAS hosts, which means the same setup would have to be performed on all CAS hosts. It would work, but it goes against the DevOps principle of 'automate everything'. Instead, we'll use GitLab's built-in functionality to automate the continuous delivery aspect.

Continuous Delivery with Git

Now that the configuration file is in our remote gitlab repository, let's try making a change to it and then rolling out the change.

Typically, making a change to the CAS configuration would require the following high-level steps to be performed (manually):

1. Back up
2. Make the change on the CAS controller
3. Make the same change on CAS workers
4. Restart the CAS server

We can automate these steps using GitLab's CI/CD pipelines. Pipelines will execute after each commit by default, but there are other options. For instance, the behaviour can be changed to execute the pipeline to run periodically on a schedule, or when there is a merge request, or manually.

First, we need to configure a runner on the CAS controller. This is more or less GitLab's version of a Jenkins agent. It allows GitLab to execute on the remote host on which the runner is installer.

Then we can build a pipeline by creating a .gitlab-ci.yml file in the repository root with the following contents:

stages:
- deploy

deploy-casconfig:
stage: deploy
script:
- if [ "$CI_COMMIT_REF_NAME" == "master" ]
- then 
- cp -p casconfig_usermods.lua /opt/sas/viya/config/etc/cas/default
- scp -p /opt/sas/viya/config/etc/cas/default/casconfig_usermods.lua sas@intcas02:/opt/sas/viya/config/etc/cas/default/casconfig_usermods.lua 
- scp -p /opt/sas/viya/config/etc/cas/default/casconfig_usermods.lua sas@intcas03:/opt/sas/viya/config/etc/cas/default/casconfig_usermods.lua
- sudo systemctl restart sas-viya-cascontroller-default
- if (nc -zv intcas01.example.com 5570 2>&1 >/dev/null); then echo "CAS SERVER ONLINE"; else exit 1; fi
- fi

The above YML file defines structure and order of the pipeline tasks. Reference documentation is available on the GitLab doc pages.

Our example may be a little crude, but it does the job of demonstrating the capabilities. A quick explanation:

• We have defined one stage in our pipeline called deploy.
• deploy has one job defined, called deploy-casconfig.
• The deploy-casconfig job runs some code in the shell on the CAS controller (where the runner is registered). First, we have defined an if-then statement to deploy the file only when commits are made to the master branch. If that condition is met, it copies the config file from the repository root to the default location on the CAS controller.
• The file is then SCP'd to the other CAS hosts (using password-less SSH), and the CAS server is bounced.
• A simple validation check is performed using the CAS server's port number to check if it's running. The job will exit with an error if the check fails.

Note that the preliminary backup step is not in our pipeline. Thanks to GitLab's version control, we can roll back to any version of the file any time, so no manual backups are necessary.

That's it - we now have a simple pipeline configured. Now let's try changing the cas.MAXSESSIONS= option in the CAS config file. We can use any text editor to update the file in the repository. I opted to use Visual Studio Code, as it has some useful built-in features and optional plugins that make light work of this. Once the file is updated, we can save and close the file, and then commit and push it to the remote repository. The pipeline will be triggered immediately, and we can view it's status by clicking on CI/CD > Pipelines. Click the stage name (deploy-casconfig) to see the result of the job.

Verify by SSHing to the CAS hosts and inspecting the contents of the file, which should now be updated with the new setting. Additionally, we can log on to Environment Manager and head to Servers > cas-shared-default > Configuration > CAS Configuration to review the value of the MAXSESSIONS option.

It's worth mentioning that what we've ended up with here is an example of continuous deployment. In GitLab, there is a distinction between continuous delivery and continuous deployment. Both refer to the flow of code from your repository to your application, but continuous delivery refers to manually triggering the push of code to remote hosts after merge requests have been reviewed and accepted, whereas continuous deployment requires no such manual trigger; rather, any changes automatically trigger the pushing of code to the application without human intervention.

Summary

This post turned out a little longer than expected but is still barely scratching the surface. Consider a scenario in which a change to a configuration file must be applied to multiple Viya environments. Perhaps the logging configuration needs to be modified, or a change to user-written SAS program must be rolled out. By storing a central copy of the file in GitLab, the process would be simpler, faster, auditable, and less prone to error. The change would only need to be made once, tested, and then automatically pushed to other servers/environments as necessary. This also simplifies the process of standing up new environments. Obviously, there are some caveats and additional factors to consider in a real-world scenario. For example, security has to be figured out (access to GitLab, the account under which the gitlab-runner is running, elevated privileges, etc.). This post outlines a simple example in an isolated environment, but it could be taken much further by leveraging the full range of capabilites GitLab offers.

Remember that other CI/CD tools such as Jenkins can also be used to perform these kinds of tasks (keep an eye out for a new Communities post coming soon with more information on using Jenkins with SAS Viya). Ansible could also be used to perform similar tasks. Ansible and Git each have slightly different features and functions and come with their own pros and cons when addressing this kind of use case. Often, a combination of the two might be appropriate.

The GitLab Docs pages also contains a vast amount of useful information, often with examples.

Thank you for reading. I hope the information provided in this post has been helpful. Please leave a comment below to ask questions or share your own experiences.