Contemplating Version Pinning for CI/CD

[#1]

The combination of Continuous Improvement / Continuous Delivery (CI/CD) and DevOps has led to significant changes in the way that software today is built, packaged, and delivered. The automation we're able to perform now is extremely powerful, allowing us to create entire deployments, test them, log the results, and tear it all down without any human intervention. The pace of our work is accelerating, but we need to be careful to understand what this means for different scenarios.

One topic to consider is what happens when software that your automation project relies on is updated to a new release. There are two basic approaches we will look at to address this.

Scenario

What are we talking about here? Well, as an example, consider you want to build an automation project (run in Jenkins, Gitlab, or similar CI/CD tool) that will deploy some cloud infrastructure, install SAS Viya software on it, and then run some code through SAS followed by removing everything to reduce cloud costs.

There are a lot of ways to accomplish this - and many outcomes - but for this example, we'll rely on the viya4-iac (-aws, -azure, -gcp) and viya4-deployment projects in Github. The automation project will perform the following general set of tasks:

1. Fire up a script to follow and run it on a local host machine to direct the automation
2. Download and build the viya4-iac container (which itself uses software projects from Terraform and your selected cloud provider)
3. Run viya4-iac to provision hardware in the cloud
4. Download and build the viya4-deployment container (which also uses other 3rd party software projects)
5. Run viya4-deployment to install SAS Viya in your cloud Kubernetes cluster
6. Run whatever suite of SAS code you want
7. Run the viya4-deployment and viya4-iac tools to destroy the cloud environment

As you've probably deduced, there are a lot of dependencies going on here. The events in this chain are tied directly and indirectly to each other and across multiple environments.

In particular though, for this post we're focused on the release versions of the software being employed in the automation project. SAS Viya has requirements for minimum Kubernetes version. The viya4-iac and viya4-deployment projects work with specific cloud CLI tools, Terraform, Ansible, and so on. And of course, your test suite of code might rely on a recent Viya release to access newer features.

Select the image to see a larger version.
Mobile users: To view the image, select the "Full" version at the bottom of the page.

How do we ensure it all plays nice together?

Breaking changes

We want to avoid unexpected changes to the software that break our automation. After all, unless you're specifically testing for problems, you're probably relying on the automation to provide a work product you want.

But changes, they do happen. Often, they're new features or other improvements to the functionality of the software. Or they might be security fixes that (ideally) don't impact the intended functionality. However, more impactful changes do happen - old features are discontinued, syntax is changed, an exploit is so bad that the security fix changes the behavior to access the desired functionality. These last few examples are all changes that could break the automation of our project.

Some changes are more subtle… like when a software release ages out of support. It might continue to work well for some time, but many organizations won't tolerate unsupported software for production use.

Can we avoid breaking changes forever? Probably not. Instead we'll need to be proactive.

Using version pinning

Version pinning is a concept that pre-dates DevOps concepts. It was originally used to describe the practice of explicitly referencing the version of the library your software depends on at build time. Software package managers (like rpm, yum, apt-get) would navigate and reconcile those dependencies for you when installing new software.

Basically, the same concept can be applied at the next higher level - for the disparate software offerings we're using in our example automation project. Consider reviewing the system requirements for SAS Viya. There are minimum (and possibly maximum) version releases of third-party software to accommodate. That's a form of version pinning already.

But really, we want to be even more specific - locking down to a particular version of a software product to ensure we know exactly how it will operate. The viya4-iac utility is built this way. The idea is that the viya4-iac container will provide specific versions of Terraform, kubectl, and the cloud provider's CLI tool. This helps because you don't have to download, install, and resolve the dependencies of these different vendors' products yourself. The viya4-iac tool guarantees you've got the right ones for deploying SAS Viya.

But there's a problem with pinning. Software, unless it's EOL, continues to march forward. Legacy versions eventually fall out of support. And in today's continuous delivery world, the trend is for support lifecycles to get shorter and shorter. This happens even with SAS Viya, only supporting the latest 4 releases… so stable releases are only good for 4 months (and LTS releases for 2 years). And this happens with many other software products, too (e.g. Kubernetes).

In other words, pinning is really just temporary. We must continue to monitor and update our automation project over time.

Using latest

In some ways, this is a simpler approach in that you always pull from latest (or main or whatever) when you download/build a piece of software that’s part of your automation project. This ensures you've always got the newest features, recent stability improvements, and up-to-date security fixes, etc. But what if that software introduces a breaking change? That could cause your automation to fail.

So, for example, when the viya4-iac tool was updated to release 2.0.0, it then specified a newer version of Terraform than before to build into its container. The interesting thing there is that Terraform 1.0.0 introduced a breaking change to its command-line syntax for building a kubeconfig file. So then, the action downstream of the viya4-iac step in our automation project failed to properly build a kubeconfig file until we diagnosed the problem and added the fix.

Another consideration is general baseline and higher support for cross-software dependencies. As you know, SAS has a support policy regarding how this works for our software to explain that while we generally support version X of a product and higher, that support only applies within the documented major release. This is another area to take care to address when using the latest version approach.

Compromises

It's unlikely we'll be able to dodge all change. Instead of trying to avoid it and ignoring the consequences, we can be proactive to identify changes that need addressing so that the work isn't overwhelming with many things to fix all at once. There are options here, too.

For our example automation project, instead of version pinning, we've decided to pull the latest versions of major software elements for several reasons, including that we use this project for day-to-day work, but we also want to stay on top of recent developments. So part of our objective here is to rely on this project as a canary to alert us to new things that people in the field might encounter.

The tradeoff, of course, is handling any new breaking changes in a timely manner to keep our automation running with minimum interruption. To support that, we've documented instructions for troubleshooting and, if necessary, pinning a software version as a workaround until a real fix can be implemented.

If our project was mission critical and could only tolerate planned outages, we might elect to pin all versions of everything wherever possible. But your project plan would still need to allow for time to address the steady progress of underlying products in order to stay in the supported range of versions.

We can try to be a bit more sophisticated in our automation with some more effort. If a software offering follows a form of semantic versioning, then we could possibly allow for some version changes to come through (like those expected to be backward compatible) and not others (those known to include breaking changes). CI/CD software development tends to skew away from this concept, however, and so now there's an increase in use of a version numbering system similar to SAS Viya, simply referencing the year and release (e.g. 2021.3).

Coda

We've talked a lot in reference to automation and repeatable deployments. But even if you're planning on a one-off software deployment, the concepts considered here are still in play. Everything in a CI/CD process is a moving target. By linking together all of these various tools from different vendors, it's possible we could encounter a breaking change (or bug or incompatibility, etc.) one day that wasn't there the day before.

So take the time to explore the various software products in your project, automated or not. Discover the versioning schemes in play and how to watch their release schedules. Experiment with version pinning prior to engaging live with a project at a customer site so you can ensure a known-good approach.