SAS Viya CI/CD Pipelines with Azure DevOps

2 Likes

The post answers the following questions: Can you create SAS Viya CI/CD pipelines with Azure DevOps? What use cases can be covered? How to approach the set-up of a CI/CD pipeline? Read this post to find out more.

The goal is to share a series of recent experience with SAS Viya with Azure DevOps. The post series will cover:

An overview (this post).
A data pipeline.
A model and a model release pipeline.
A decision pipeline.
Azure DevOps agents.
How to protect secrets in a pipeline.
Conclusions and next steps.

CI/CD

A CI/CD pipeline is a sequence of events or jobs that can be executed. CI stands for continuous integration. CD means either continuous delivery or continuous deployment.

My CI/CD interpretation is to get “something developed in SAS Viya” (SAS program, SAS Studio flow, decision, model) into the hands of the end-users.

Why Use CI/CD Pipelines?

Automate the build and release process.
Make the process repeatable and transparent.
Improve quality and consistency.

What is Azure DevOps?

Azure DevOps is a Software as a service (SaaS) DevOps toolchain and orchestration platform from Microsoft. Azure DevOps is language, platform and cloud provider agnostic.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

Several services define Azure DevOps:

Azure Repos: Cloud-hosted private Git repositories.
Azure Pipelines: Continuously build, test, and deploy to any platform and cloud. This is the CI/CD engine in Azure DevOps.

The focus of the post series is on Azure Repos and Azure Pipelines.

Azure DevOps also includes Azure Boards. These are agile tools to plan, track, and discuss work across your teams. Afterward, in Azure Artifacts you can create, host, and share packages with your team. Subsequently, Azure Test Plans help you test and ship with confidence using manual and exploratory testing tools. Lastly, through the Extensions Marketplace you can access hundreds of community-built extensions from Teams and Slack to SonarCloud and other apps and services.

Source: Microsoft Azure DevOps Overview.

Azure DevOps Rankings

Microsoft Azure DevOps is ranked by Peerspot.com:

#1 in Release Automation tools, which includes RedHat Ansible and GitLab.
#1 in Enterprise Agile Planning Tools, before Micro Focus ALM Octane, Planview LeanKit.
#2 in Application Lifecycle Management Suites after Atlassian JIRA.

SAS Viya and Azure DevOps

Pipelines

With Azure DevOps, you can create pipelines operating outside or inside SAS Viya.

Outside

Outside SAS Viya, you could use Azure DevOps to:

- Build a model or decision release pipeline. The release pipeline can deploy a SAS model or decision, stored as a container image, in Azure. The deployment target can be an Azure Web App, Container Instance or Kubernetes Service.
- Install software and packages on a cloud Virtual Machine.
- Deploy cloud resources.
- Deploy SAS Viya.

Inside

Inside SAS Viya, you can use Azure DevOps to:

- Process data.
- Automate model tasks.
- Publish or deploy a decision.
- Score a model or a decision.
- Automate SAS administration tasks.
- Etc.

You are only limited by your imagination (and the amount of code you are willing to add to your Git repository and run inside the pipeline).

The post series will focus on the "inside SAS Viya" CI/CD pipelines.

Steps

You can resume the SAS Viya and Azure DevOps interaction in four steps:

Develop

The key idea is to store “the whole lot” ready for delivery or deployment in a Git repository.

“The whole lot” includes:

SAS artifacts: programs, SAS Studio flows, model and decision files, SAS content in JSON format.
Non-SAS artifacts: Python programs, bash scripts, YAML files, etc. interacting with SAS Viya.
In some cases, a small amount of static, non-private data, for tests, might be acceptable.

“The whole lot” excludes: data, secrets, keys, certificates. We will see in a next post how to use SAS Viya with an Azure Key Vault in an Azure Pipeline.

Version

Version control is handled by a Git repository. In Azure DevOps, the default Git repository is stored in Azure Repos. You can choose GitHub, GitLab, Bitbucket instead of Azure Repos. GitHub, being a Microsoft company, is very well integrated with Azure DevOps.

The Azure pipeline definition, a YAML file, is stored in the same Git repository.

Build

The Azure pipeline clones the Git repository on an agent and executes the steps in the pipeline script.

Execute

Pipeline agents are virtual machines (VM). The agent interacts with a SAS Viya deployment through:

SAS Viya Command Line Interface (CLI).
SAS Viya REST API.

The agents can be Microsoft hosted or self-hosted.

Microsoft hosted: At execution time, the pipeline will fetch, from the Azure cloud, a VM fulfilling certain requirements.

Self-Hosted: You can use a VM you have already deployed. You will have to register it as an agent, in Azure DevOps, before you can use it in a pipeline. Then you must install the software and the packages you need, copy the necessary files, etc.

For Azure DevOps to interact with SAS Viya, a key element is to use and configure a self-hosted agent. We will develop this topic in a future post.

Conclusions

The post briefly explained that you can create SAS Viya CI/CD pipelines with Azure DevOps (Pipelines and Repos). Next, the post looked at possible CI/CD pipelines within SAS Viya. Finally, it provided a four-step approach to set-up a CI/CD pipeline.

Resources

Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about the post content. If you wish to get more information, please write me an email.

Find more articles from SAS Global Enablement and Learning here.