How to Create a SAS Viya Azure Pipelines Agent

[#1]

Learn how to create your own self-hosted Azure Pipelines agent. The agent is a Linux Virtual Machine (VM), hosted in Microsoft Azure. This agent performs the build tasks defined in the Azure Pipelines YAML. The YAML can instruct the agent to run a SAS program or SAS Studio flow in batch, import a SAS package, publish a SAS model or a decision to a destination such as Azure or Git.

Overall Picture

The self-hosted agent is the key piece for SAS Viya to work with Azure DevOps. The steps are:

• Create a virtual machine that acts as a self-hosted agent (this post).
• Read How to Make Your Virtual Machine Talk with SAS Viya on Azure to learn how to configure communication with the SAS Viya cluster: define Network Security Groups rules, copy certificates, etc.
• Create an Azure DevOps self-hosted agent.
• Install software and packages on the self-hosted agent: SAS Viya Command Line Interface CLI, utilities such as pyViyaTools, Python, etc.

 

Azure Virtual Network Topology

The self-hosted agent is the virtual machine highlighted in the following Azure virtual network VNET topology. For security and many practical reasons, you will have to create not only the virtual machine, but also many other Azure resources.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

You can write Azure CLI directly in the Cloud Shell or from a machine with cloud access. The commands will create Azure resources.

Create Azure Resources

To realize the above topology, I recommend you follow these steps. As you can see, creating the agent Virtual Machine is actually the last step in this process.

• SSH Keys
  • If You Have the Keys
  • If You Do Not Have the Keys
• Initialize Variables
• Create a Resource Group
• Create a Virtual Network and Subnet
• Create a Public IP Address
• Create a Network Security Group
• Create Network Security Group Rules
• Create a Virtual Network Interface Card
• Create an Availability Set
• Create a Virtual Machine
• Test SSH to Your VM

SSH Keys

To manage the virtual machine you will create, you will need to connect to it using Secure Shell (SSH). To connect using SSH, you will need a set of SSH keys.

If You Have the Keys

First, check you have SSH keys that you can use:

ls -la ~/.ssh
# create the private_key variable to match the key name 
export private_key=~/.ssh/gelazuredm-aks-key

In my case, the key pair was previously generated. It contains:

• The private key gelazuredm-aks-key
• The public key gelazuredm-aks-key.pub
  
  

If You Do Not Have the Keys

If you do not have the keys, you can generate them using the following statement:

cd ~/.ssh
ssh-keygen -m PEM -t rsa -b 4096

Name the new keys. You could use a passphrase, which is more secure.

# adapt the private_key variable to match the key name 
# export private_key=~/.ssh/gelazuredm-aks-key2

Initialize Variables

For the resources you are about to create, it is best to adopt a naming convention. In this example, the resources will be named using the particle $MYUSER:

Create a Resource Group

An Azure resource group is a logical container into which Azure resources are deployed and managed. A resource group must be created before a virtual machine and supporting virtual network resources:

# adapt the private_key variable to match the key name 
az group create --name $RG --location $LOCATION

Create a Virtual Network and Subnet

Create a virtual network in Azure and a subnet into which you can create your VM. Use az network vnet to create a virtual network named $VNET with the Classless Inter-Domain Routing (CIDR) 10.2.0.0/28 (equivalent to 16 IPs). You also add a subnet named $SUBNET with the CIDR 10.2.0.0/29 (8 IPS). This is the minimum and we really do not need more for our purpose:

az network vnet create \
    --resource-group $RG \
    --name $VNET \
    --address-prefix 10.2.0.0/28 \
    --subnet-name $SUBNET \
    --subnet-prefix 10.2.0.0/29

Create a Public IP Address

Now let's create a public IP address with az network public-ip create. This public IP address enables you to connect to your VMs from the Internet. Because the default address is static, create with the --allocation-method Static parameter.

az network public-ip create \
    --resource-group $RG \
    --name $IP \
    --allocation-method Static \
    --sku Basic

Create a Network Security Group

To control the flow of traffic in and out of your VMs, you apply a network security group to a virtual Network Interface Card (NIC) or subnet. The following example uses az network nsg create to create a network security group named $NSG:

az network nsg create \
    --resource-group $RG \
    --name $NSG

Create Network Security Group Rules

You define rules that allow or deny specific traffic. To allow inbound connections on port 22 (to enable SSH access), create an inbound rule with az network nsg rule create. The following example creates a rule named AllowSSH:

az network nsg rule create \
    --resource-group $RG \
    --nsg-name $NSG \
    --name ${MYUSER}AllowSSH \
    --protocol tcp \
    --priority 1000 \
    --source-address-prefixes 148.172.0.0/16 \
    --destination-port-range 22 \
    --access allow

Note: 148.172.0.0/16 is an IP where employees can connect to the internal network using VPN. Therefore, the rule will allow only organization employees who are connected with corporate VPN to SSH the VM.

Create a Virtual Network Interface Card

Virtual network interface cards (NICs) are programmatically available because you can apply rules to their use. Depending on the VM size, you can attach multiple virtual NICs to a VM. In the following az network nic create command, you create a NIC named $NIC and associate it with your network security group. The public IP address $IP is also associated with the virtual NIC.

az network nic create \
    --resource-group $RG \
    --name $NIC \
    --vnet-name $VNET \
    --subnet $SUBNET \
    --public-ip-address $IP \
    --network-security-group $NSG

Create an Availability Set

Availability sets help spread your VMs across fault domains and update domains. Even though you only create one VM right now, it's best practice to use availability sets to make it easier to expand in the future. Create an availability set for your VM with az vm availability-set create. The following example creates an availability set named $AS:

az vm availability-set create \
    --resource-group $RG \
    --name $AS

 

Create a Virtual Machine

You've created the network resources to support Internet-accessible VMs. Now create a VM and secure it with an existing SSH key. In this example, let's create an Ubuntu VM based on the most recent LTS. You can find additional images with az vm image list.

Specify an SSH key to use for authentication. If you do not have an SSH public key pair, you can create them or use the --generate-ssh-keys parameter to create them for you. If you already have a key pair, this parameter uses existing keys in your ~/.ssh folder. Create the VM by bringing all the resources and information together with the az vm create command.

The following example creates a VM named $VM:

az vm create \
    --resource-group $RG \
    --name $VM \
    --location $LOCATION \
    --availability-set $AS \
    --nics $NIC \
    --image UbuntuLTS \
    --admin-username $VM_USER \
    --ssh-key-value ${private_key}.pub \
    --size Standard_B2s

SSH to Your VM

The infrastructure was created. Test the connection, SSH to your VM with the public IP address created:

# SSH connect to your VM
JUMPBOXIP=$(az vm list-ip-addresses -g $RG -n $VM --query "[].virtualMachine.network.publicIpAddresses[0].ipAddress" -o tsv)
echo "${MYUSER} Jump Box Vm's Public IP: $JUMPBOXIP"
cd ~/.ssh
ssh -i $private_key  jumpuser@$JUMPBOXIP

Conclusion

You created a Virtual Machine that will be configured as a self-hosted agent. Read the next posts to learn how to:

• Configure communication with the SAS Viya cluster (Network Security Groups rules, certificates).
• Install software on the self-hosted agent (SAS Viya Command Line Interface CLI, utilities such as pyViyaTools, Python, etc.).

 

Discussion

There are alternatives to the proposed Azure Virtual Network Topology. The SAS Viya deployment on Azure comes with a pre-deployed virtual machine. You can use it to configure it as a self-hosted Azure Pipelines agent too, without creating a new one.

Read the Next Post in the Series

Want to know how to configure this VM to communicate with SAS Viya? Read How to Make Your Virtual Machine Talk with SAS Viya on Azure.

Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.

 

Find more articles from SAS Global Enablement and Learning here.