SAS Viya and Azure Pipelines Self-Hosted Agents
The key to working with SAS Viya and Azure Pipelines is a self-hosted agent. The post looks at what a self-hosted agent is, their advantages and disadvantages. The post then explains how a self-hosted agent can work with SAS Viya. Two examples are given: in the first, agents and SAS Viya are deployed on Azure; in the second, agents and SAS Viya are deployed on-premises. Understanding the agents architecture can help you move code and artifacts from one environment to another using pipelines. Read more to understand how this can be achieved.
Self-Hosted Agents
An agent is a system that performs build tasks. Build is the keyword you will find in the Microsoft documentation. Think of a build as of instructions, scripts or programs that must be executed.
The agent is a dedicated server that runs your build process. The self-hosted agent can be a Linux, Windows, macOS virtual machine or a Docker container.
Microsoft explains very well the build and the agents. "Imagine that you have an Azure Pipelines project that receives build requests many times per day. Or perhaps you have multiple projects that can each use the same type of build agent. You can organize build agents into agent pools to help ensure that there's a server ready to process each build request.
When a build is triggered, Azure Pipelines selects an available build agent from the pool. If all agents are busy, the process waits for one to become available."
For example, the next Azure Pipelines definition, captured in a YAML file, indicates that the pipeline must run on an agent, from the “sbxbot-sas-viya-jump-vm” agent pool.
# starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
trigger:
- main
pool:
name: sbxbot-sas-viya-jump-vm
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
- script: |
echo Add other tasks to build, test, and deploy your project.
echo See https://aka.ms/yaml
displayName: 'Run a multi-line script'
Watch the following starter pipeline running on a self-hosted agent.
Self-Hosted Agents and SAS Viya
Let us now look at how a self-hosted agent interacts with SAS Viya.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Azure Pipelines acquires an available agent, from the named pool (Execute).
This agent clones the Git repository associated with the pipeline on the agent machine (Version). It will then run the instructions in the pipeline definition (Build). These instructions can be bash commands invoking the SAS Viya CLI commands or python programs calling SAS Viya REST requests.
When working with SAS Viya, self-hosted agents present advantages and disadvantages compared to Microsoft-Hosted Agents.
Advantages
- You install the software you need only once. For example, a specific Python version or package, SAS Viya Command Line Interface, Azure Command Line Interface, etc. You therefore save time at every run.
- Each time you run a pipeline, it will be on the same VM. This helps with logging, debugging, you can connect to the VM using SSH, explore its folders, files, etc.
- The agent can run as a service. The service can start when you start the machine.
- The agent can accommodate larger disk space needed.
- There is no time limit for a pipeline to run.
Disadvantages
- You manage a self-hosted agent yourself: the deployment, the software installation, the updates and the maintenance, etc.
- You must keep this agent up and running. That means additional costs.
When you put into a balance the advantages and disadvantages, you will prefer self-hosted agents when working with SAS Viya.
Agents and SAS Viya on Azure
Agents are great when you need to move code and artifacts from one environment to another.
The holy grail of CI/CD can be summarized as follows. Write your code once, in a SAS Viya environment, that we will call DEV. Next, deploy and run the code in any number of environments, with a minimum of intervention.
This is where agents can help.
To deploy the code and artifacts in another environment, which we will call TEST, you can simply change the agent pool name, in your pipeline code, from DEV to TEST. At least this is the theory.
This theory assumes you have identical SAS Viya deployments, with no configuration drift.
If your environments are slightly different, you need to parametrize your pipeline definition and your code base. You need to work with parameters, as the SAS Viya URLs will be different, files might have a different name or be in a different location in the two deployments.
One agent pool can contain several agents. For example the DEV agent pool can have two agents, both hosted on Azure:
Agents and SAS Viya on-Premises and on Azure
Azure DevOps supports a hybrid model, with agents running in a cloud or on-premises. You can configure an agent on a VM you are hosting on-prem. This machine must communicate with the Kubernetes cluster where SAS Viya is running.
For example, the DEV agent pool has just one agent, hosted on-premises:
In this example, you can run and test your code using the on-prem agent pool called DEV. When you are ready to promote the code to a SAS Viya on Azure instance, you will change the agent pool name to TEST.
You will not get identical SAS Viya deployments on-prem and on Azure. Therefore the deployment would require more work to map these parameters. The parameters can point to the SAS Viya URL, certificates, file names or their locations.
Azure Networking Considerations
On Azure, the arrow between an agent and the SAS Viya Azure Kubernetes Service cluster, can be depicted more accurately, as in the next figure.
One of the key objectives, when working with self-hosted Azure agents, is to establish a communication line between the agent network and the network where SAS Viya resources are deployed. This is the double orange arrow in the middle. I deliberately over-simplified the components on the right side, where SAS Viya resides.
Think of Network Security Groups (NSG) as the security guarding the doors of an event. To attend the event you must be on the list. The event is the virtual network where resources are deployed. The list is made of the inbound rules in the NSG.
Letting agents attend the SAS Viya “event”, means adding their IPs to the NSG's inbound rules. The agent traffic transits over port 443.
We might look closer at Azure networking for agents in a future post.
Conclusions
In this post, you read what an Azure Pipelines self-hosted agent is, its advantages and disadvantages. A self-hosted agent is probably the best option to work with a SAS Viya deployment. Lastly, the post mentioned how Azure Pipelines agents could help promote code from one environment to another. The next posts will look at agents configuration. First, at a self-hosted CentOS machine, running on-prem. Later, at an Ubuntu virtual machine hosted on Azure.
Azure DevOps and SAS Viya Resources
- SAS Viya Decision Pipeline with Azure DevOps.
- A SAS Viya Model Release Pipeline with Azure DevOps.
- A SAS Viya with Data Pipeline with Azure DevOps.
- SAS Viya CI/CD Pipelines with Azure DevOps.
Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.
Find more articles from SAS Global Enablement and Learning here.
