Read this post to understand how you can build an Azure DevOps self-hosted agent for SAS Viya using scripts. An agent is a virtual machine. This machine can be hosted in a cloud, in Azure, for example. It can also be hosted on-premises.
Overall Picture
The steps to create your own self-hosted agent are:
Pre-Requisites
- Access to an Azure DevOps organization.
- Azure DevOps Organization policies allowing personal access tokens (PAT) use.
- Permissions to create an agent pool in the Azure DevOps organization or in a project.
Azure DevOps Organization
You need access to an Azure DevOps organization.
Azure DevOps Organization Policies
Your Azure DevOps organization policies must allow the usage of personal access tokens (PAT). The Azure DevOps user must have sufficient rights to create an agent pool and a self-hosted agent. Managing an Azure DevOps organization may be discussed in a future post.
Create an Azure DevOps Self Hosted Agent
The steps are:
- Create an Azure DevOps Personal Access Token (PAT).
- Create an agent pool.
- Add an agent to that pool.
Create a Personal Access Token
In your Azure DevOps organization, from User settings > Personal access tokens (PAT), create a new PAT.
To “recruit” the machine as a self-hosted agent, the machine has to initiate the communication. To communicate securely with your Azure DevOps organization a personal access token is required. The token is only needed for the agent configuration.
Select the scopes:
- Custom defined:
- Agent Pools > Read & manage.
- Build > Read & execute.
Create an Agent Pool
Agent pools can regroup several agents. For example, you can add all the development machines in a pool, the test machines in a second pool and so on. You can also have a pool for cloud machines, another for on-premises machines.
Add an Agent in a Pool
In an existing agent pool, add the new machine (New agent). Choose x64 for Linux or CentOS.
The configuration has to be initiated from the VM. SSH to that machine:
cd ~
# adapt the resource group name, the VM name and the SSH key to your environment
export MYUSER=$USER && echo $MYUSER
export JUMPBOXKEY=~/.ssh/gelazuredm-aks-key
JUMPBOXIP=$(az vm list-ip-addresses -g $MYUSER-SCR -n $MYUSER-jump-vm --query "[].virtualMachine.network.publicIpAddresses[0].ipAddress" -o tsv)
echo "jump box Public IP: $JUMPBOXIP"
# SSH connect to your jump VM
cd ~/.ssh
ssh -i $JUMPBOXKEY jumpuser@$JUMPBOXIP
Set Azure DevOps Agent Variables
Scripts are from now on executed on the jump box. To configure the agent, you will need your PAT:
export AZP_TOKEN=paste_here_the_PAT_you_created_in_Az_DevOps
# MYUSER is optional. Only used here as a prefix for other variables.
export MYUSER=your_User_ID # your-6-digit-SAS-userid
Set the following environment variables, adapt them to your environment:
export AZP_POOL=$MYUSER-sas-viya-jump-vm # name of your agent pool
export AZP_AGENT_NAME=$MYUSER-azuredm-jump-vm # your agent's name in Azure DevOps agent pool
export AZP_URL=https://dev.azure.com/geldmdevops # URL to your Azure DevOps organization.
export AZP_WORK=_work # working folder for the agent on this VM
export SUDO_USER=jumpuser # user to run the agent configuration
Install and Update Packages
You might need to do that before you set your agent version.
sudo apt update
sudo apt -y upgrade
sudo apt install jq -y
Set the Agent Version
Set the AZP_AGENT_VERSION environment variable to specify the latest version of the agent.
export AZP_AGENT_VERSION=$(curl -s https://api.github.com/repos/microsoft/azure-pipelines-agent/releases | jq -r '.[0].tag_name' | cut -d "v" -f 2)
echo $AZP_AGENT_VERSION
A YAML pipeline on a Linux machine must be using the latest version of the agent, even if it's pre-release. The agent software is constantly updating, so you can curl the version information from a Microsoft GitHub repo. The command uses jq to read the latest version from the JSON string that's returned.
Build the Agent Script
Create a script to install the Azure DevOps agent on this VM:
cd ~
tee ~/build-agent.sh > /dev/null << EOF #!/bin/bash set -e # Select a default agent version if one is not specified if [ -z "$AZP_AGENT_VERSION" ]; then AZP_AGENT_VERSION=2.200.2 fi # Verify Azure Pipelines token is set if [ -z "$AZP_TOKEN" ]; then echo 1>&2 "error: missing AZP_TOKEN environment variable"
exit 1
fi
# Verify Azure DevOps URL is set
if [ -z "$AZP_URL" ]; then
echo 1>&2 "error: missing AZP_URL environment variable"
exit 1
fi
# If a working directory was specified, create that directory
if [ -n "$AZP_WORK" ]; then
mkdir -p "$AZP_WORK"
fi
# Create the Downloads directory under the user's home directory
if [ -n "$HOME/Downloads" ]; then
mkdir -p "$HOME/Downloads"
fi
# Download the agent package
curl https://vstsagentpackage.azureedge.net/agent/$AZP_AGENT_VERSION/vsts-agent-linux-x64-$AZP_AGENT_VERSION.tar.gz > $HOME/Downloads/vsts-agent-linux-x64-$AZP_AGENT_VERSION.tar.gz
# Create a working directory to extract the agent package to
mkdir -p $HOME/myagent
# Move to the working directory
cd $HOME/myagent
# Extract the agent package to the working directory
tar zxvf $HOME/Downloads/vsts-agent-linux-x64-$AZP_AGENT_VERSION.tar.gz
# Install the agent software
./bin/installdependencies.sh
# Configure the agent as the sudo (non-root) user
sudo chown $SUDO_USER $HOME/myagent
sudo -u $SUDO_USER ./config.sh --unattended \
--agent "${AZP_AGENT_NAME:-$(hostname)}" \
--url "$AZP_URL" \
--auth PAT \
--token "$AZP_TOKEN" \
--pool "${AZP_POOL:-Default}" \
--work "${AZP_WORK:-_work}" \
--replace \
--acceptTeeEula
# Install and start the agent service
sudo $HOME/myagent/svc.sh install $SUDO_USER
sudo $HOME/myagent/svc.sh start
sudo $HOME/myagent/svc.sh status
EOF
ls -la
Make the script executable, and then run it.
chmod u+x build-agent.sh
sudo -E ./build-agent.sh
Sudo enables the script to run as the root user. The -E argument preserves the current environment variables, including the ones you set, so that they're available to the script.
As the script runs, you can see the VM being transformed in an agent.
Test the Self-Hosted Agent with a Pipeline
In your Azure DevOps Organization, select your project, go to pipelines. Edit the pipeline. Copy the pipeline code below and paste it over in your new pipeline:
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
trigger:
- main
pool:
name: myuser-sas-viya-jump-vm
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
- script: |
echo Add other tasks to build, test, and deploy your project.
echo See https://aka.ms/yaml
displayName: 'Run a multi-line script'
What "tells" Azure Pipelines to run on a self-hosted agent are these lines in the pipeline definition:
pool:
name: myuser-sas-viya-jump-vm
The pipeline will pick a suitable, available agent from this agent pool.
Supposing your agent pool has more agents, you can constrain the pipeline to run only on a specific agent, with the “demands” clause.
pool:
name: myuser-sas-viya-jump-vm
demands:
- agent.name -equals myuser-jump-vm
Conclusion
You configured an Azure virtual machine as a self-hosted agent using scripts. You ran a first Azure pipeline on the self-configured agent.
After the build, you can use this virtual machine as an Azure Pipelines self-hosted agent with SAS Viya. This will be the subject of the next post.
Resources Used
Microsoft Learn – Exercise – Create a build agent that runs on Azure.
Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content.
If you wish to get more information, please write me an email.
