Read this post to understand how you can build an Azure DevOps self-hosted agent for SAS Viya using scripts. An agent is a virtual machine. This machine can be hosted in a cloud, in Azure, for example. It can also be hosted on-premises. Overall Picture   The steps to create your own self-hosted agent are:   Create a virtual machine on Azure. Read How to Create a SAS Viya Azure Pipelines Agent. Configure communication with the SAS Viya Azure Kubernetes Service cluster. Read How to Make Your Virtual Machine Talk with SAS Viya on Azure. Create an Azure DevOps self-hosted agent (this post). Install software and packages on the self-hosted agent: SAS Viya Command Line Interface (CLI), Python, pyviyatools, utilities, etc. (next post) Pre-Requisites Access to an Azure DevOps organization. Azure DevOps Organization policies allowing personal access tokens (PAT) use. Permissions to create an agent pool in the Azure DevOps organization or in a project. Azure DevOps Organization You need access to an Azure DevOps organization. Azure DevOps Organization Policies Your Azure DevOps organization policies must allow the usage of personal access tokens (PAT). The Azure DevOps user must have sufficient rights to create an agent pool and a self-hosted agent. Managing an Azure DevOps organization may be discussed in a future post. Create an Azure DevOps Self Hosted Agent The steps are: Create an Azure DevOps Personal Access Token (PAT). Create an agent pool. Add an agent to that pool. Create a Personal Access Token In your Azure DevOps organization, from User settings > Personal access tokens (PAT), create a new PAT.   To “recruit” the machine as a self-hosted agent, the machine has to initiate the communication. To communicate securely with your Azure DevOps organization a personal access token is required. The token is only needed for the agent configuration.     Select the scopes: Custom defined: Agent Pools > Read & manage. Build > Read & execute. Create an Agent Pool   Agent pools can regroup several agents. For example, you can add all the development machines in a pool, the test machines in a second pool and so on. You can also have a pool for cloud machines, another for on-premises machines.   Add an Agent in a Pool In an existing agent pool, add the new machine (New agent). Choose x64 for Linux or CentOS.   Jump on the Virtual Machine The configuration has to be initiated from the VM. SSH to that machine: cd ~ # adapt the resource group name, the VM name and the SSH key to your environment export MYUSER=$USER && echo $MYUSER export JUMPBOXKEY=~/.ssh/gelazuredm-aks-key JUMPBOXIP=$(az vm list-ip-addresses -g $MYUSER-SCR -n $MYUSER-jump-vm --query "[].virtualMachine.network.publicIpAddresses[0].ipAddress" -o tsv) echo "jump box Public IP: $JUMPBOXIP" # SSH connect to your jump VM cd ~/.ssh ssh -i $JUMPBOXKEY jumpuser@$JUMPBOXIP   Set Azure DevOps Agent Variables Scripts are from now on executed on the jump box. To configure the agent, you will need your PAT: export AZP_TOKEN=paste_here_the_PAT_you_created_in_Az_DevOps # MYUSER is optional. Only used here as a prefix for other variables. export MYUSER=your_User_ID # your-6-digit-SAS-userid   Set the following environment variables, adapt them to your environment: export AZP_POOL=$MYUSER-sas-viya-jump-vm # name of your agent pool export AZP_AGENT_NAME=$MYUSER-azuredm-jump-vm # your agent's name in Azure DevOps agent pool export AZP_URL=https://dev.azure.com/geldmdevops # URL to your Azure DevOps organization. export AZP_WORK=_work # working folder for the agent on this VM export SUDO_USER=jumpuser # user to run the agent configuration   Install and Update Packages You might need to do that before you set your agent version. sudo apt update sudo apt -y upgrade sudo apt install jq -y   Set the Agent Version Set the AZP_AGENT_VERSION environment variable to specify the latest version of the agent. export AZP_AGENT_VERSION=$(curl -s https://api.github.com/repos/microsoft/azure-pipelines-agent/releases | jq -r '.[0].tag_name' | cut -d "v" -f 2) echo $AZP_AGENT_VERSION A YAML pipeline on a Linux machine must be using the latest version of the agent, even if it's pre-release. The agent software is constantly updating, so you can curl the version information from a Microsoft GitHub repo. The command uses jq to read the latest version from the JSON string that's returned.   Build the Agent Script Create a script to install the Azure DevOps agent on this VM: cd ~ tee ~/build-agent.sh > /dev/null << EOF #!/bin/bash set -e # Select a default agent version if one is not specified if [ -z "$AZP_AGENT_VERSION" ]; then AZP_AGENT_VERSION=2.200.2 fi # Verify Azure Pipelines token is set if [ -z "$AZP_TOKEN" ]; then echo 1>&2 "error: missing AZP_TOKEN environment variable" exit 1 fi # Verify Azure DevOps URL is set if [ -z "$AZP_URL" ]; then echo 1>&2 "error: missing AZP_URL environment variable" exit 1 fi # If a working directory was specified, create that directory if [ -n "$AZP_WORK" ]; then mkdir -p "$AZP_WORK" fi # Create the Downloads directory under the user's home directory if [ -n "$HOME/Downloads" ]; then mkdir -p "$HOME/Downloads" fi # Download the agent package curl https://vstsagentpackage.azureedge.net/agent/$AZP_AGENT_VERSION/vsts-agent-linux-x64-$AZP_AGENT_VERSION.tar.gz > $HOME/Downloads/vsts-agent-linux-x64-$AZP_AGENT_VERSION.tar.gz # Create a working directory to extract the agent package to mkdir -p $HOME/myagent # Move to the working directory cd $HOME/myagent # Extract the agent package to the working directory tar zxvf $HOME/Downloads/vsts-agent-linux-x64-$AZP_AGENT_VERSION.tar.gz # Install the agent software ./bin/installdependencies.sh # Configure the agent as the sudo (non-root) user sudo chown $SUDO_USER $HOME/myagent sudo -u $SUDO_USER ./config.sh --unattended \ --agent "${AZP_AGENT_NAME:-$(hostname)}" \ --url "$AZP_URL" \ --auth PAT \ --token "$AZP_TOKEN" \ --pool "${AZP_POOL:-Default}" \ --work "${AZP_WORK:-_work}" \ --replace \ --acceptTeeEula # Install and start the agent service sudo $HOME/myagent/svc.sh install $SUDO_USER sudo $HOME/myagent/svc.sh start sudo $HOME/myagent/svc.sh status EOF ls -la   Make the script executable, and then run it. chmod u+x build-agent.sh sudo -E ./build-agent.sh Sudo enables the script to run as the root user. The -E argument preserves the current environment variables, including the ones you set, so that they're available to the script.   As the script runs, you can see the VM being transformed in an agent. Test the Self-Hosted Agent with a Pipeline   In your Azure DevOps Organization, select your project, go to pipelines. Edit the pipeline. Copy the pipeline code below and paste it over in your new pipeline: # Starter pipeline # Start with a minimal pipeline that you can customize to build and deploy your code. # Add steps that build, run tests, deploy, and more: # https://aka.ms/yaml trigger: - main pool: name: myuser-sas-viya-jump-vm steps: - script: echo Hello, world! displayName: 'Run a one-line script' - script: | echo Add other tasks to build, test, and deploy your project. echo See https://aka.ms/yaml displayName: 'Run a multi-line script'   What "tells" Azure Pipelines to run on a self-hosted agent are these lines in the pipeline definition:   pool: name: myuser-sas-viya-jump-vm   The pipeline will pick a suitable, available agent from this agent pool.   Supposing your agent pool has more agents, you can constrain the pipeline to run only on a specific agent, with the “demands” clause. pool: name: myuser-sas-viya-jump-vm demands: - agent.name -equals myuser-jump-vm   Conclusion You configured an Azure virtual machine as a self-hosted agent using scripts. You ran a first Azure pipeline on the self-configured agent.   After the build, you can use this virtual machine as an Azure Pipelines self-hosted agent with SAS Viya. This will be the subject of the next post. Resources Used Microsoft Learn – Exercise – Create a build agent that runs on Azure.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content.   If you wish to get more information, please write me an email.   ... View more

This post shows you step by step how to configure a Virtual Machine hosted on Azure to communicate with SAS Viya deployed on Azure. After the configuration, you can use this virtual machine as an Azure Pipelines self-hosted agent. You can also use this machine in a standalone mode, to execute scripts that interact with SAS Viya. The interaction is possible via the SAS Viya Command Line Interface or via SAS Viya REST API requests.   Overall Picture   The post assumes that SAS Viya is deployed on Azure, in an Azure Kubernetes Service (AKS) cluster.   For SAS Viya to work with Azure DevOps and Azure Pipelines, a self-hosted agent is the key piece. A self-hosted agent is a Virtual Machine (VM). The steps to create your own self-hosted agent are:   Create a virtual machine on Azure. Read How to Create a SAS Viya Azure Pipelines Agent. Configure communication with the SAS Viya AKS cluster: create Network Security Groups rules; copy SAS Viya certificates (this post). Create an Azure DevOps self-hosted agent. Install software and packages on the self-hosted agent: SAS Viya Command Line Interface (CLI), Python, pyviyatools, utilities, etc.   Azure Network Security Groups and Rules   One of the key objectives, when working with a self-hosted Azure agent, which can be a VM, is to establish a communication line between:   The virtual network (VNET) where the VM is deployed (left in the figure below). The VNET where SAS Viya resources are deployed (right in the figure below).   The communication line is represented by the double yellow – orange arrow in the middle. I deliberately over-simplified the components on the SAS Viya side, to the right.     To understand a Network Security Groups (NSG), let us use an analogy.   A NSG is the security personnel guarding the doors of an event. To attend the event you must be on the attendees list. The event location is the VNET containing the resources you need to access.   Letting packages from our VM enter the location where SAS Viya “performs”, means adding NSG rules.   A rule has:   A sense: inbound or outbound. Is made of source and destination IPs and ports. A directive: Allow or Deny.   If your objective is to launch SAS Viya CLI commands, from the VM, to interact with SAS Viya, you should know that the CLI traffic transits typically over port 443.   The next step is to create a NSG rule.   Pre-Requisites: Login, set your subscription and location   You need to run this code from a machine which has the az cli and kubectl installed. You can use the Azure portal Cloud Shell or another Linux machine with cloud access.   # Variables - ADAPT them to your environment! export MYUSER=$USER RG=$MYUSER-SCR LOCATION=eastus VM=$MYUSER-jump-vm NSG=$MYUSER-nsg MYSUB=GELDM # Actions # az login # if not logged in already az account list -o table az account set --subscription $MYSUB az config set defaults.location=$LOCATION   SAS Viya Azure Kubernetes Service Network Security Group Rule (Right Side)     The next rule will allow traffic, from the public IP of your VM,  JUMPBOXIP, to the SAS Viya Load Balancer Public IP,  INBOUND_IP, over port 443. The rule is created in the aks-agentpool-yyyyyyyy-nsg. This NSG is associated to the Azure Kubernetes Service AKS virtual network​.   The inbound rule allows outside access, from the VM, through the ingress controller (load balancer), to the SAS Viya pods.   cd ~ # Get the public IP of the Linux Virtual Machine $MYUSER-jump-vm: JUMPBOXIP=$(az vm list-ip-addresses -g $RG -n $VM --query "[].virtualMachine.network.publicIpAddresses[0].ipAddress" -o tsv) echo "${MYUSER} Jump Box Vm's Public IP: $JUMPBOXIP" # find the internal AKS resource group RESOURCEGRP=${MYUSER}-azuredm-rg AKS_RG=$(az aks list -g $RESOURCEGRP --query [].nodeResourceGroup -o tsv) echo "AKS resource group: $AKS_RG" # find the public IP address of the AKS cluster INBOUND_IP=$(az network public-ip list --query "[].{tags: tags.type, address: ipAddress}" -o tsv -g $AKS_RG | grep None | cut -f2) echo "AKS inbound IP: $INBOUND_IP" # Get the NSG of the VM echo "VM NSG: $NSG" echo "Check: $AKS_NSG is filled!" echo "In the https://portal.azure.com navigate to $AKS_RG and search for the aks-agentpool-yyyyyyyy-nsg" # Create inbound NSG rule - port 443 az network nsg rule create -g $AKS_RG --nsg-name $AKS_NSG -n Allow443From-${MYUSER}-jump-vm \ --priority 496 \ --source-address-prefixes $JUMPBOXIP \ --source-port-ranges '*' \ --destination-address-prefixes $INBOUND_IP \ --destination-port-ranges '443' --access Allow \ --protocol Tcp --description "Allow access from New JumpBox on 443" echo "Rule is created in ${AKS_NSG}"   Physically, you will find the aks-agentpool-yyyyyyyy-nsg under the so-called nodeResourceGroup (usually prefixed by MC_ in SAS Viya deployments on Azure).   SAS Viya Certificates   To make your VM “talk” with SAS Viya, you will most likely pass SAS Viya CLI commands or REST API requests.   But before that, you will need to obtain and copy SAS Viya certificate files on the VM. Otherwise any communication attempt with SAS Viya will result in a certificate error.   The process to copy the certificates can be tricky and involves a few steps.   Get the Certificates   ### 0. Setup-Environment-Variables export MYUSER=$USER echo $MYUSER ### 1. Get the SAS Viya certificates from the AKS Cluster #### 1.1 Set the KUBECONFIG file. This is the Kubernetes .conf file allowing you to pass kubectl commands with your SAS Viya deployment. #### On this machine, the file sits in ~/.kube/$MYUSER-aks-kubeconfig.conf cd ~ export KUBECONFIG=~/.kube/$MYUSER-aks-kubeconfig.conf # adapt to your environment echo $KUBECONFIG #### 1.2 Set the current namespace, where SAS Viya is deployed and the context export current_namespace=gelazure # adapt to your environment kubectl config set-context --current --namespace ${current_namespace} kubectl config get-contexts #### 1.3 Get the kube .conf file az aks get-credentials --resource-group $MYUSER-azuredm-rg --name $MYUSER-azuredm-aks --admin -y ls -la ~/.kube #### 1.4 List the SAS Viya Certificates cd ~ ls -la ~/.certs #### You might not have the folder on your machine. Create the folder. mkdir -p ~/.certs #### 1.5 Get the SAS Viya Certificates .PEM File cd ~ kubectl cp $(kubectl get pod -l app=sas-logon-app -o=jsonpath='{.items[0].metadata.name}'):security/trustedcerts.pem ~/.certs/${current_namespace}_trustedcerts.pem ls -la ~/.certs #### 1.6 Backup the .PEM File before the copy cd ~/.certs echo Change permission before overwriting chmod 644 gelazure_trustedcerts.pem chmod 644 gelazure_trustedcerts.bak ls -la & pwd echo Backup the certificates before refresh cp gelazure_trustedcerts.pem gelazure_trustedcerts.bak ls -la #### 1.7 Get the SAS Viya Certificates / Refresh .PEM File cd ~ kubectl cp $(kubectl get pod -l app=sas-logon-app -o=jsonpath='{.items[0].metadata.name}'):security/trustedcerts.pem ~/.certs/${current_namespace}_trustedcerts.pem ls -la ~/.certs   Copy the SAS Viya Certificates to your Virtual Machine   ### 2. Copy SAS Viya Certificates to VM #### 2.1 Initialize variables. The VM and the resource group name. echo $MYUSER echo Assume VM name is $MYUSER-jump-vm # adapt to your environment echo Get the VM Public IP JUMPBOXIP=$(az vm list-ip-addresses -g $MYUSER-SCR -n $MYUSER-jump-vm --query "[].virtualMachine.network.publicIpAddresses[0].ipAddress" -o tsv) echo "JumpBox Public IP: $JUMPBOXIP" #### 2.2 Initialize variables: CURL CA Bundle and SSH Private Key. You will SSH the file to the VM. export CURL_CA_BUNDLE=~/.certs/${current_namespace}_trustedcerts.pem echo $CURL_CA_BUNDLE export private_key=~/.ssh/gelazuredm-aks-key cert=${current_namespace}_trustedcerts.pem echo $cert #### 2.3 Create the .certs folder on jump box VM. Assign permissions. Copy files. Verify copy: cd ~/.certs ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "sudo mkdir -p ~/.certs" wait echo Transfer folder ownership from root to jumpuser: ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "sudo chown jumpuser ~/.certs" wait echo Set folder permissions on the jump box, just in case: ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "sudo chmod -R 755 ~/.certs" wait echo Copy the .pem file: scp -o "StrictHostKeyChecking=no" -i ${private_key} ${cert} jumpuser@${JUMPBOXIP}:~/.certs wait echo Finally, list .pem file permissions: ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "ls -la ~/.certs" wait   Adjust the Permissions   On the machine you copied the certificates from, put the file permissions back to an acceptable range.   ### 3. Certs permissions - VM from where you are copying the certificates and Azure Jump Box VM #### 3.1 Change back file permissions, on your VM ~/.certs folder, after their copy on the jump box: chmod 400 ~/.certs/gelazure_trustedcerts.pem chmod 400 ~/.certs/gelazure_trustedcerts.bak ls -la ~/.certs #### if no longer needed on this machine, you can delete them from here. Do not delete them form the jump box. #### 3.2 On the Jump Box VM: ##### Change the certificate permissions ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "sudo chmod 400 ~/.certs/gelazure_trustedcerts.pem" wait ##### Set .certs folder permissions ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "sudo chmod 700 ~/.certs" wait echo List .pem file and .certs folder permissions ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "ls -la ~/.certs" wait ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "ls -la ~" wait     Conclusions   This post shown how to configure a Virtual Machine hosted on Azure to communicate with SAS Viya deployed on Azure. The first step is to establish Network Security Group rules that allow the communication. The second step is to obtain the SAS Viya .pem file from the SAS Viya Azure Kubernetes Cluster and copy it to the Virtual Machine.   After the configuration, you can use this virtual machine as an Azure Pipelines self-hosted agent. This will be the subject of the next post: Create an Azure DevOps self-hosted agent . Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.   ... View more

Learn how to create your own self-hosted Azure Pipelines agent. The agent is a Linux Virtual Machine (VM), hosted in Microsoft Azure. This agent performs the build tasks defined in the Azure Pipelines YAML. The YAML can instruct the agent to run a SAS program or SAS Studio flow in batch, import a SAS package, publish a SAS model or a decision to a destination such as Azure or Git.   Overall Picture   The self-hosted agent is the key piece for SAS Viya to work with Azure DevOps. The steps are: Create a virtual machine that acts as a self-hosted agent (this post). Read How to Make Your Virtual Machine Talk with SAS Viya on Azure to learn how to configure communication with the SAS Viya cluster: define Network Security Groups rules, copy certificates, etc. Create an Azure DevOps self-hosted agent. Install software and packages on the self-hosted agent: SAS Viya Command Line Interface CLI, utilities such as pyViyaTools, Python, etc.   Azure Virtual Network Topology   The self-hosted agent is the virtual machine highlighted in the following Azure virtual network VNET topology. For security and many practical reasons, you will have to create not only the virtual machine, but also many other Azure resources.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   You can write Azure CLI directly in the Cloud Shell or from a machine with cloud access. The commands will create Azure resources.   Create Azure Resources   To realize the above topology, I recommend you follow these steps. As you can see, creating the agent Virtual Machine is actually the last step in this process. SSH Keys If You Have the Keys If You Do Not Have the Keys Initialize Variables Create a Resource Group Create a Virtual Network and Subnet Create a Public IP Address Create a Network Security Group Create Network Security Group Rules Create a Virtual Network Interface Card Create an Availability Set Create a Virtual Machine Test SSH to Your VM SSH Keys   To manage the virtual machine you will create, you will need to connect to it using Secure Shell (SSH). To connect using SSH, you will need a set of SSH keys.   If You Have the Keys   First, check you have SSH keys that you can use:   ls -la ~/.ssh # create the private_key variable to match the key name export private_key=~/.ssh/gelazuredm-aks-key   In my case, the key pair was previously generated. It contains: The private key gelazuredm-aks-key The public key gelazuredm-aks-key.pub If You Do Not Have the Keys   If you do not have the keys, you can generate them using the following statement:   cd ~/.ssh ssh-keygen -m PEM -t rsa -b 4096   Name the new keys. You could use a passphrase, which is more secure.   # adapt the private_key variable to match the key name # export private_key=~/.ssh/gelazuredm-aks-key2     Initialize Variables   For the resources you are about to create, it is best to adopt a naming convention. In this example, the resources will be named using the particle $MYUSER:   Create a Resource Group   An Azure resource group is a logical container into which Azure resources are deployed and managed. A resource group must be created before a virtual machine and supporting virtual network resources:   # adapt the private_key variable to match the key name az group create --name $RG --location $LOCATION   Create a Virtual Network and Subnet   Create a virtual network in Azure and a subnet into which you can create your VM. Use az network vnet to create a virtual network named $VNET with the Classless Inter-Domain Routing (CIDR) 10.2.0.0/28 (equivalent to 16 IPs). You also add a subnet named $SUBNET with the CIDR 10.2.0.0/29 (8 IPS). This is the minimum and we really do not need more for our purpose:   az network vnet create \ --resource-group $RG \ --name $VNET \ --address-prefix 10.2.0.0/28 \ --subnet-name $SUBNET \ --subnet-prefix 10.2.0.0/29   Create a Public IP Address   Now let's create a public IP address with az network public-ip create . This public IP address enables you to connect to your VMs from the Internet. Because the default address is static, create with the --allocation-method Static parameter.   az network public-ip create \ --resource-group $RG \ --name $IP \ --allocation-method Static \ --sku Basic   Create a Network Security Group   To control the flow of traffic in and out of your VMs, you apply a network security group to a virtual Network Interface Card (NIC) or subnet. The following example uses az network nsg create to create a network security group named $NSG:   az network nsg create \ --resource-group $RG \ --name $NSG Create Network Security Group Rules   You define rules that allow or deny specific traffic. To allow inbound connections on port 22 (to enable SSH access), create an inbound rule with az network nsg rule create . The following example creates a rule named AllowSSH:   az network nsg rule create \ --resource-group $RG \ --nsg-name $NSG \ --name ${MYUSER}AllowSSH \ --protocol tcp \ --priority 1000 \ --source-address-prefixes 148.172.0.0/16 \ --destination-port-range 22 \ --access allow Note: 148.172.0.0/16 is an IP where employees can connect to the internal network using VPN. Therefore, the rule will allow only organization employees who are connected with corporate VPN to SSH the VM.     Create a Virtual Network Interface Card   Virtual network interface cards (NICs) are programmatically available because you can apply rules to their use. Depending on the VM size, you can attach multiple virtual NICs to a VM. In the following az network nic create command, you create a NIC named $NIC and associate it with your network security group. The public IP address $IP is also associated with the virtual NIC.   az network nic create \ --resource-group $RG \ --name $NIC \ --vnet-name $VNET \ --subnet $SUBNET \ --public-ip-address $IP \ --network-security-group $NSG     Create an Availability Set   Availability sets help spread your VMs across fault domains and update domains. Even though you only create one VM right now, it's best practice to use availability sets to make it easier to expand in the future. Create an availability set for your VM with az vm availability-set create . The following example creates an availability set named $AS:   az vm availability-set create \ --resource-group $RG \ --name $AS   Create a Virtual Machine   You've created the network resources to support Internet-accessible VMs. Now create a VM and secure it with an existing SSH key. In this example, let's create an Ubuntu VM based on the most recent LTS. You can find additional images with az vm image list .   Specify an SSH key to use for authentication. If you do not have an SSH public key pair, you can create them or use the --generate-ssh-keys parameter to create them for you. If you already have a key pair, this parameter uses existing keys in your ~/.ssh folder. Create the VM by bringing all the resources and information together with the az vm create command .   The following example creates a VM named $VM:   az vm create \ --resource-group $RG \ --name $VM \ --location $LOCATION \ --availability-set $AS \ --nics $NIC \ --image UbuntuLTS \ --admin-username $VM_USER \ --ssh-key-value ${private_key}.pub \ --size Standard_B2s     SSH to Your VM   The infrastructure was created. Test the connection, SSH to your VM with the public IP address created:   # SSH connect to your VM JUMPBOXIP=$(az vm list-ip-addresses -g $RG -n $VM --query "[].virtualMachine.network.publicIpAddresses[0].ipAddress" -o tsv) echo "${MYUSER} Jump Box Vm's Public IP: $JUMPBOXIP" cd ~/.ssh ssh -i $private_key jumpuser@$JUMPBOXIP   Conclusion   You created a Virtual Machine that will be configured as a self-hosted agent. Read the next posts to learn how to: Configure communication with the SAS Viya cluster (Network Security Groups rules, certificates). Install software on the self-hosted agent (SAS Viya Command Line Interface CLI, utilities such as pyViyaTools, Python, etc.).   Discussion   There are alternatives to the proposed Azure Virtual Network Topology. The SAS Viya deployment on Azure comes with a pre-deployed virtual machine. You can use it to configure it as a self-hosted Azure Pipelines agent too, without creating a new one.     Read the Next Post in the Series Want to know how to configure this VM to communicate with SAS Viya? Read How to Make Your Virtual Machine Talk with SAS Viya on Azure.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

With faceted search you can look at your information assets, from multiple lenses, called facets. You can search assets by name, asset type, description, by date or by user. You can explore data assets by library, file type, table type, number of columns or rows, size. You can search by column information such as name, label, keyword, or completeness percentage. More surprisingly you can search by language detected, semantic type calculated or data privacy status. Watch the video to understand how faceted search can help you find what you need or discover something new about your information assets.   What Is Faceted Search?   The faceted search in SAS Information Catalog is like a photographer’s lens kit. The facets are the lenses. Each has its own characteristics. They can capture different angles, colors, levels of detail. You switch them, until you can get the perfect shot. The perfect shot can be a unique perspective over your information assets in SAS Viya.     These facets are calculated when assets are automatically indexed or discovered with an agent.   The automatic index keeps track of models, reports, decisions, SAS Studio flows, data plans and CASLIB in-memory or SASHDAT tables available in your SAS Viya.   The discovery agent analyzes libraries: CASLIBs or SAS Compute. It produces tens of metrics for each table or file included in the discovery. These calculated metrics are then associated with facets, through which you can search.     Faceted Search Short Demonstration   Watch the following short video to understand what faceted search can help you discover in SAS Viya.   Asset Type Facets   Try a faceted search, AssetType, and it will list the asset types catalogued.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The results are the union of assets catalogued via: Automatic indexing, e.g. model will list all catalogued models. Discovery agents, e.g. inmemorytable will list all the CAS in-memory tables.   Semantic Types Facets   SAS Information Catalog is using a Quality Knowledge Base to calculate semantic types for columns in a table or file, based on their name or content. It also labels the columns from a data privacy perspective. For example: Column.informationPrivacy: "private" lists all tables that contain private data.     Column.semanticType: "individual" shows tables or files where columns have been classified as belonging to a private person.   Detected Language Facets   More recently, language detection is also performed in long text columns:     Languages: "en" will show you tables that have columns with English language content.   Languages: "zh" will show you which tables have Chinese language content.   Languages: "tl" will indicate assets where Tagalog was detected in a column.   To search all CAS tables that contain Korean language data try: +Languages: "ko" +AssetType: "cas" . The + indicates a MUST type of search, which means every facet criterion must be satisfied in the search results.     Combine Faceted and Free Text   SAS Information Catalog supports two types of searches, free text and faceted, which you can mix and match in a single query, for even better precision or more surprising findings.   Software Version   The above examples were produced using the SAS Viya Long Term Support Release 2022.1 (May 2022), although facet search has been around since at least October 2021. You will require a SAS Information Governance license for language, semantic type and information privacy features.   Additional Resources   For more information, see SAS® Information Catalog | Faceted Search. Alternatively, see SAS® Information Catalog | Free Text Search.   Conclusion   The faceted search allows you to look at your information assets, from multiple lenses, called facets. You can search assets by name, asset type, library, file type, table type, language detected, semantic type calculated or information privacy status, just to name a few.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

The key to working with SAS Viya and Azure Pipelines is a self-hosted agent. The post looks at what a self-hosted agent is, their advantages and disadvantages. The post then explains how a self-hosted agent can work with SAS Viya. Two examples are given: in the first, agents and SAS Viya are deployed on Azure; in the second, agents and SAS Viya are deployed on-premises. Understanding the agents architecture can help you move code and artifacts from one environment to another using pipelines. Read more to understand how this can be achieved.   Self-Hosted Agents   An agent is a system that performs build tasks. Build is the keyword you will find in the Microsoft documentation. Think of a build as of instructions, scripts or programs that must be executed.   The agent is a dedicated server that runs your build process. The self-hosted agent can be a Linux, Windows, macOS virtual machine or a Docker container.   Microsoft explains very well the build and the agents. "Imagine that you have an Azure Pipelines project that receives build requests many times per day. Or perhaps you have multiple projects that can each use the same type of build agent. You can organize build agents into agent pools to help ensure that there's a server ready to process each build request.   When a build is triggered, Azure Pipelines selects an available build agent from the pool. If all agents are busy, the process waits for one to become available."   For example, the next Azure Pipelines definition, captured in a YAML file, indicates that the pipeline must run on an agent, from the “sbxbot-sas-viya-jump-vm” agent pool.   # starter pipeline # Start with a minimal pipeline that you can customize to build and deploy your code. # Add steps that build, run tests, deploy, and more: # https://aka.ms/yaml trigger: - main pool: name: sbxbot-sas-viya-jump-vm steps: - script: echo Hello, world! displayName: 'Run a one-line script' - script: | echo Add other tasks to build, test, and deploy your project. echo See https://aka.ms/yaml displayName: 'Run a multi-line script'   ​Watch the following starter pipeline running on a self-hosted agent.     Self-Hosted Agents and SAS Viya   Let us now look at how a self-hosted agent interacts with SAS Viya. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Azure Pipelines acquires an available agent, from the named pool (Execute).   This agent clones the Git repository associated with the pipeline on the agent machine (Version). It will then run the instructions in the pipeline definition (Build). These instructions can be bash commands invoking the SAS Viya CLI commands or python programs calling SAS Viya REST requests.   When working with SAS Viya, self-hosted agents present advantages and disadvantages compared to Microsoft-Hosted Agents.   Advantages   You install the software you need only once. For example, a specific Python version or package, SAS Viya Command Line Interface, Azure Command Line Interface, etc. You therefore save time at every run. Each time you run a pipeline, it will be on the same VM.​ This helps with logging, debugging, you can connect to the VM using SSH, explore its folders, files, etc.​ The agent can run as a service.​ The service can start when you start the machine. The agent can accommodate larger disk space needed. There is no time limit for a pipeline to run.   Disadvantages   You manage a self-hosted agent yourself: the deployment, the software installation, the updates and the maintenance, etc.​ You must keep this agent up and running. That means additional costs.   When you put into a balance the advantages and disadvantages, you will prefer self-hosted agents when working with SAS Viya.   Agents and SAS Viya on Azure   Agents are great when you need to move code and artifacts from one environment to another.   The holy grail of CI/CD can be summarized as follows. Write your code once, in a SAS Viya environment, that we will call DEV. Next, deploy and run the code in any number of environments, with a minimum of intervention.   This is where agents can help.   To deploy the code and artifacts in another environment, which we will call TEST, you can simply change the agent pool name, in your pipeline code, from DEV to TEST. At least this is the theory.   This theory assumes you have identical SAS Viya deployments, with no configuration drift.   If your environments are slightly different, you need to parametrize your pipeline definition and your code base. You need to work with parameters, as the SAS Viya URLs will be different, files might have a different name or be in a different location in the two deployments.     One agent pool can contain several agents. For example the DEV agent pool can have two agents, both hosted on Azure:     Agents and SAS Viya on-Premises and on Azure   Azure DevOps supports a hybrid model, with agents running in a cloud or on-premises. You can configure an agent on a VM you are hosting on-prem. This machine must communicate with the Kubernetes cluster where SAS Viya is running.     For example, the DEV agent pool has just one agent, hosted on-premises:     In this example, you can run and test your code using the on-prem agent pool called DEV. When you are ready to promote the code to a SAS Viya on Azure instance, you will change the agent pool name to TEST.   You will not get identical SAS Viya deployments on-prem and on Azure. Therefore the deployment would require more work to map these parameters. The parameters can point to the SAS Viya URL, certificates, file names or their locations.   Azure Networking Considerations   On Azure, the arrow between an agent and the SAS Viya Azure Kubernetes Service cluster, can be depicted more accurately, as in the next figure.     One of the key objectives, when working with self-hosted Azure agents, is to establish a communication line between the agent network and the network where SAS Viya resources are deployed. This is the double orange arrow in the middle. I deliberately over-simplified the components on the right side, where SAS Viya resides.   Think of Network Security Groups (NSG) as the security guarding the doors of an event. To attend the event you must be on the list. The event is the virtual network where resources are deployed. The list is made of the inbound rules in the NSG.   Letting agents attend the SAS Viya “event”, means adding their IPs to the NSG's inbound rules. The agent traffic transits over port 443.   We might look closer at Azure networking for agents in a future post.    Conclusions   In this post, you read what an Azure Pipelines self-hosted agent is, its advantages and disadvantages. A self-hosted agent is probably the best option to work with a SAS Viya deployment. Lastly, the post mentioned how Azure Pipelines agents could help promote code from one environment to another. The next posts will look at agents configuration. First, at a self-hosted CentOS machine, running on-prem. Later, at an Ubuntu virtual machine hosted on Azure.   Azure DevOps and SAS Viya Resources SAS Viya Decision Pipeline   with Azure DevOps. A   SAS Viya Model Release Pipeline   with Azure DevOps. A   SAS Viya with Data Pipeline   with Azure DevOps. SAS Viya CI/CD Pipelines with Azure DevOps . SAS Viya and Azure Pipelines Microsoft-Hosted Agents   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

The key to working with SAS Viya and Azure Pipelines from Azure DevOps is the agent that you will use. The post explains what an agent is. It then looks at the different types of Azure DevOps agents, Microsoft-hosted and self-hosted. The post focuses on Microsoft-hosted agents, their advantages and disadvantages.   What Are Agents?   ​Agents are Virtual Machines or containers that run an Azure pipeline.​ Agents can run on Linux, Windows or macOS Virtual Machines (VM). They can run in a Docker container​, as well. The agents are typically grouped in an agent pool. An agent pool is a regrouping of agents that can run pipelines, just like in the example below.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The communication with Azure DevOps is initiated by the agent​. The agents state their availability to your Azure DevOps agent pool. In Azure DevOps we distinguish two types of agents:   Microsoft-hosted. Self-hosted.   Microsoft-Hosted Agents   The easiest way to use Azure DevOps is with Microsoft-hosted agents. When you run a pipeline, Azure Pipelines does several things:   First, it determines the type of agent that can run your pipeline. Second, it takes stock of the agents available and acquires an available agent, from the Azure cloud. Third, it clones the Git repository linked to your pipeline on this agent. Fourth, it starts running the instructions in the pipeline definition file.   Watch a starter pipeline running on a Microsoft hosted agent.         The Azure pipeline definition is a YAML file. The file indicates that the pipeline must run on a Microsoft hosted agent, an ubuntu VM (pool: vmImage: ubuntu-latest).   # starter pipeline # Start with a minimal pipeline that you can customize to build and deploy your code. # Add steps that build, run tests, deploy, and more: # https://aka.ms/yaml trigger: - main pool: vmImage: ubuntu-latest steps: - script: echo Hello, world! displayName: 'Run a one-line script' - script: | echo Add other tasks to build, test, and deploy your project. echo See https://aka.ms/yaml displayName: 'Run a multi-line script' Advantages   Using this type of agent is very convenient. Microsoft handles the installation, the maintenance and the upgrade.​ Certain software is pre-installed: e.g.: Python, AZ CLI, etc.​ It is simple and easy to use. In your Azure DevOps organization, you have X “run” minutes you can consume every month.​   Disadvantages   Each time you run a pipeline you get a fresh Virtual Machine from the Azure cloud.​ If any software you need is missing, it is up to you to script the install of that software.​ The machine is discarded after each run, including the software you had to install!   Can Microsoft-Hosted Agents Be Used with SAS Viya?   In theory, yes. Although, you must consider the following aspects:   Software and Packages   The first reason is related to existing software on the agent. For an Azure pipeline to work with SAS Viya, your agent must pass SAS Viya command-line interface (CLI) commands or SAS Viya REST API requests. To pass SAS Viya CLI commands, you will need to have the SAS Viya CLI installed on the agent. To invoke SAS Viya REST APIs, the agent will require specific Python packages installed. Now, imagine installing the CLIs and the python packages, again and again, every time you want to run something. Obviously, this will add to the execution time.   Networking   A second reason to consider is networking. On Azure, the agent machine must communicate with your SAS Viya Azure Kubernetes Service (AKS) cluster. The two components sit in different virtual networks (vnet). By default, the SAS Viya deployment allows the access only from a very specific IP address range. Your Microsoft-hosted agent will have an IP unknown to the vnet where SAS Viya is running. Any call you will make from this agent will be blocked by the Network Security Group of the vnet. To overcome the issue, you will have to add a new rule in the Network Security Group. This adds more time to the execution time. Also your MS-hosted agent will have a different IP, at the next run.   Authorizations   A third reason is related to authorizations. SAS Viya is using Transport Layer Security TLS. You would need a set of certificates on the agent,  for SAS Viya to allow any communication. Your Microsoft hosted agent will not know about these certificates. You will need to copy them, from somewhere secure, on this agent, every time you are running a pipeline.   For all these impracticalities, you should use a self-hosted agent. We will discuss self-hosted agents in the next post.   Conclusions   In this post, you read what an Azure Pipelines MS-hosted agent is and its advantages and disadvantages. MS-hosted agents are the go-to option for a variety of tasks. However, for working with SAS Viya, using a self-hosted agent is probably the best option. Self-hosted agents will be discussed in the next post.   Resources   Read more about: Azure Pipelines agents. Microsoft-hosted agents. Azure DevOps and SAS Viya:   SAS Viya and Azure Pipelines Self-Hosted Agents SAS Viya Decision Pipeline with Azure DevOps. A   SAS Viya Model Release Pipeline   with Azure DevOps. A   SAS Viya with Data Pipeline   with Azure DevOps. SAS Viya CI/CD Pipelines with Azure DevOps .   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

If you wondered how to pass parameters to a SAS program running in batch in SAS Viya (2020.1 or later), this post shows you a possible solution. The alternative titles could have been “How to write your SAS program for a CI/CD pipeline, when you have parameters” or "How does the autoexec work in batch with SAS Viya".   The SAS Program   Consider the following SAS program, that defines a PATH CASLIB. Suppose you wrote and tested the program in an environment, which we will call DEV.   * CAS host ; cas casauto host="controller.sas-cas-server-default.gelazure.svc.cluster.local" port=5570 sessopts=(metrics=true); * CASLIB path and name; %let SRC_PATH=/azuredm/gitrepo/content/; %let DM_src=source; * define a CASLIB on this specific path ; proc cas; table.dropcaslib caslib="&DM_SRC" quiet=true; quit; proc casutil; caslib &DM_SRC path="&SRC_PATH" global; list files; quit; cas casauto terminate ;   The Problem   Suppose you want to run the same program, in batch, in another SAS Viya environment, which we will call TEST. If the two SAS Viya environments are perfectly the same, then you do not have to do anything. However, what if in TEST, you have a different CAS host and CASLIB path?   %let cashost=controller.sas-cas-server-default.gelenv.svc.cluster.local; %let SRC_PATH=/gelcontent/home/sasadm/gitrepo/content/;   Your program will fail in TEST.   To solve the problem you could create a "TEST version" of your program. What if you have hundreds of programs you need to promote? Do you multiply them by the number of environments?    Solution   A better solution might be to define the parameters that change with the environment, in a second file. With other words, you avoid hardcoding elements that change with the environment.   For DEV, create a myauto-dev.sas program:   %let cashost=controller.sas-cas-server-default.gelazure.svc.cluster.local; %let SRC_PATH=/azuredm/gitrepo/content/;   For TEST, create a myauto-test.sas program:   %let cashost=controller.sas-cas-server-default.gelenv.svc.cluster.local; %let SRC_PATH=/gelcontent/home/sasadm/gitrepo/content/;   Your main program will use the parameters from the myauto-$env.sas program, where env is dev, test, etc. Let us name this program 1.sas :   * Print parameters from myauto file; %put &cashost; %put &SRC_PATH; * CAS host ; cas casauto host="&cashost" port=5570 sessopts=(metrics=true); * CASLIB path and name; %let DM_src=source; * define a CASLIB on this specific path ; proc cas; table.dropcaslib caslib="&DM_SRC" quiet=true; quit; proc casutil; caslib &DM_SRC path="&SRC_PATH" global; list files; quit; cas casauto terminate ;   The Batch Execution   To run this program, in batch, use the SAS Viya Command Line Interface (CLI), submit-pgm command. Create a variable env  with values dev, test, etc., that you will use in your parameter file name (one per environment): myauto-dev.sas, myauto-test.sas, etc. When you submit the programs for execution, specify --job-file myauto-$env.sas. In bash, it would be:   clidir=/opt/sas/viya/home/bin # the folder where the SAS Viya CLI is installed cd ~ # the folder where 1.sas and myauto.sas programs are echo Run the main SAS program 1.sas with myauto.sas as autoexec file export env=test cat ~/myauto-${env}.sas > ~/myauto.sas $clidir/sas-viya batch jobs submit-pgm --pgm-path ~/1.sas --context default --job-file myauto.sas --sasoption '-autoexec !BATCHJOBDIR/myauto.sas' --watch-output --wait-log-list --results-dir ~/ echo Reprint the execution log cat ~/1.log   The file myauto.sas is used as an autoexec file for the SAS session. The --job-file option is used to upload the myauto.sas file to the file set, where it is then copied to the batch job directory. There, it is accessible as !BATCHJOBDIR . The --sasoption option is used to specify the -autoexec SAS option that uses the myauto.sas file.   With SAS Viya on Kubernetes, the execution happens in a new pod. The pod is created when you run the submit-pgm command. The pod is destroyed when the job is complete. You therefore need to pass both files at the execution time.   How to Create a SAS Viya CLI Profile and Get a SAS Viya Access Token   Previous posts comments asked to add the code to create a SAS Viya CLI profile and get a token. There it is:   echo Create SAS-Viya CLI Profile export SAS_CLI_PROFILE=gelenv # gelenv is the Kubernetes namespace where SAS Viya is running export current_namespace=$SAS_CLI_PROFILE export CURL_CA_BUNDLE=~/.certs/${current_namespace}_trustedcerts.pem export SSL_CERT_FILE=~/.certs/${current_namespace}_trustedcerts.pem export REQUESTS_CA_BUNDLE=${SSL_CERT_FILE} echo Adapt the SAS Viya INGRESS_URL to match your environment export INGRESS_URL=https://Your-SAS-Viya_URL-here/ # for example https://gelenv.pdcesx08880.race.sas.com/ clidir=/opt/sas/viya/home/bin echo This is the folder where the SAS-Viya CLI is installed $clidir $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile set-endpoint ${INGRESS_URL} $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile toggle-color on $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile set-output fulljson $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile show echo Login in $clidir/sas-viya --profile ${SAS_CLI_PROFILE} auth login -user sasadm -password ********* export SAS_OUTPUT=text echo You can now run any sas-viya cli commands, including the submit-pgm   Conclusions   To execute a SAS program, in batch, using parameters, you can split your program in two: a program containing the parameters and a second program using them. Submit both programs using the SAS Viya CLI, the batch plug-in.   Acknowledgements   Many thanks to my SAS colleague @AjmalFarzam.   References   SAS® Viya® Programming Documentation | Submit SAS Programs in Batch You can find a submit-pgm command example in How to Run SAS Programs in Batch in the New SAS Viya. How to Run a SAS Studio Flow in Batch Thank you for your time reading this post. If you liked the post, give it a thumbs up!   Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

What if you could combine SAS Intelligent Decisioning with Azure DevOps to create a decision pipeline? A pipeline that imports decisions stored in Git, loads data needed for tests, guides the user to test and publish the decision and finally, deploys the decision. Watch the video demonstration in this post to find out more.   This is the fourth post in the series. The previous posts covered:   SAS Viya Model Release Pipeline with Azure DevOps SAS Viya with Azure DevOps Data Pipeline SAS Viya CI/CD Pipelines with Azure DevOps Requirements   Suppose you have a customer that works with SAS Intelligent Decisioning. The customer requires:   Automatic processes to import, export and deploy the decision. Decisions are scored by issuing REST API calls. An automatic process to load the data needed for tests. Easily track the changes from one decision version to another. Manual review steps, where the user can test and confirm the test results before deployment. Logs and tracking of the above steps.   You could solve the above requirements with SAS Viya and Azure Pipelines. SAS Viya Decision Pipeline   Watch this short video demonstration:       The decision pipeline has multiple stages. Each of the stage has jobs. Every job takes care of a task and fulfills a part of the requirements. For instance:   Develop stage: Import a decision from a package file, stored in a Git repository (Azure Repos). Load the test data in Cloud Analytic Services (CAS). Ask the user to test the decision with the test data. Publish stage: If the test is successful, publish the decision to a Git publishing destination, such as GitHub. When you publish a decision to Git, you can do two things: first, version the decision files in a human readable format; second, prepare its deployment. After publishing, a job in this stage lists the Git repository folders and the files created. Deploy stage: Use the decision files published to Git to deploy to another publishing destination, SAS Micro Analytic Service (MAS). From Git, you can publish to MAS or CAS. Publishing to MAS allows you to score with a REST API.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Conclusions   The pipeline demonstrated only a subset of what is possible. You can go even further and add other jobs or pipelines before or after. Before, to create publishing destinations. After, to score the decisions deployed and check the results. You can deploy the decision as a container image in Azure, as shown in the SAS Viya Model Release Pipeline with Azure DevOps.   You can combine jobs managing decisions inside SAS Viya with other Azure Pipelines tasks and scripts. You can therefore place decisions developed in SAS Viya in a larger enterprise-wide scope.   Resources SAS Viya and Azure Pipelines Self-Hosted Agents SAS Viya and Azure Pipelines Microsoft-Hosted Agents    Thank you for your time reading this post. If you liked the post, give it a thumbs up!   Please comment and tell us what you think about this topic. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

What if you could use the SAS Model Manager “Publish” model button to automatically deploy your SAS model to Azure? It can be done, thanks to SAS Container Runtime (SCR) and an Azure DevOps release pipeline. SCR allows you to publish a SAS model as a container image in Azure. A release pipeline starts when a new image is published. It takes the image, and it automatically deploys it to a back-end Azure Web App. The app's role is to allow you to score the SAS model running in a container. When more models are published from SAS, the release pipeline handles their deployment. Scoring can continue, without interruption. Watch the video demonstrations in this post to find out more.   This is the third post in the series. The previous post demonstrated a SAS Viya with Azure DevOps Data Pipeline.   Watch the Release Pipeline in Action   This example shows how the Publish button in SAS Model Manager is "transformed" in an automatic deployment process. An Azure DevOps pipeline deploys the model to Azure.     How the Model Release Pipeline Works?   To summarize what you have seen in the video:   A first SAS model or a new SAS model version is ready to be deployed. Publish this model to an Azure publishing destination. The SAS model gets published as a Docker container image in an Azure Container Registry (ACR). The Azure destination is using SAS Container Runtime (SCR). An Azure DevOps release pipeline is triggered when you publish a new container image in the ACR. When the release pipeline is triggered, the web app is immediately updated. Its production slot will be refreshed with the latest container image. Therefore, the SAS model is running inside a SCR container, in the Azure app production slot. Scoring against the model can continue. We observe a certain delay when the production slot is refreshed. However, everything goes back to normal soon after. When a new SAS model is published, the release pipeline repeats the process all over again. The scoring at the end demonstrates that the new model replaced the old one, in the production slot, without any human intervention.   What Are the Steps to Create a Model Release Pipeline?   This video shows the steps to set-up a model release pipeline:     To summarize the steps in the video:   Create an Azure publishing destination. Publish at least one SAS model to Azure. Create an Azure App service plan. The plan allows you to use deployment slots with Azure Web Apps. Create an Azure Web App in your service plan. Create a release pipeline in Azure DevOps. The release pipeline has a trigger, an artifact and at least a stage, with one task. The trigger is when a new Docker container image is published to the Azure Container Registry (ACR). The artifact is the container image itself. The task is to publish the container image to an Azure web app, in the production slot.   Conclusions   The post demonstrated an automatic model release process from SAS Model Manager to Azure. SAS Container Runtime in SAS Viya and an Azure DevOps release pipeline were the key ingredients.   What about decisions from SAS Intelligent Decisioning? Is a decision release pipeline feasible? Yes, it is. You can apply the same approach to deploy decisions to Azure.   Additional Resources SAS Viya with Azure DevOps: SAS Viya with Azure DevOps Data Pipeline. SAS Viya CI/CD Pipelines with Azure DevOps. SAS Viya Decision Pipeline with Azure DevOps SAS Viya and Azure Pipelines Self-Hosted Agents SAS Viya and Azure Pipelines Microsoft-Hosted Agents SAS Container Runtime How to Publish a SAS Model to Azure with SCR: A Start-to-Finish Guide. How to Score a SAS Model in an Azure Container Instance How to Publish a Decision to Azure with SAS Container Runtime. How to Score a SAS Decision Published to Azure with SCR. Using the SAS Container Runtime for publishing SAS models to Kubernetes on the Azure cloud. How to Deploy and Score a SAS Model or Decision in Azure Kubernetes Service.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about this topic.   If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

Watch the short video demonstration in this post. The video shows a SAS Viya and Azure DevOps pipeline. The pipeline focuses on data management activities in SAS Viya, such as running SAS programs and SAS Studio flows.   This is the second post in the series. The previous post offered an overview of SAS Viya CI/CD Pipelines with Azure DevOps.   Video Demonstration   This example shows an Azure pipeline which combines several data management tasks in SAS Viya:    The Git Repository   The "trick" is to add everything that you need to run in a Git repository. Azure Repos are used in this example.   First, add the Azure Repo in SAS Studio. Second, create and test the SAS programs and SAS Studio flows in SAS Viya. Third, push the files containing the SAS programs and flows to the remote Git repository.   You can export the SAS Studio flow as a SAS package (JSON file) and then store it in the Git repository. You could also save the flow file directly (FLW file), in the same repository.   The Azure Pipeline   Several jobs define the pipeline. Each job performs a task:   Load a CAS table from a SAS program. Import a SAS Studio flow from a package. Validate manually the flow. Run the flow to query the table and create an output table. Save the output table, as a file, in Azure Data Lake Storage Gen 2 (ADLS2).   The Azure pipeline definition, a YAML file, defines the tasks, their order, where they should run and when.   You can view and change this YAML file directly from SAS Studio.   When Will The Pipeline Run?   The trigger type in the pipeline definition sets when the pipeline will start. For instance, every time you push a change to the main branch of the Git repository.   Where Will the Pipeline Run?   The pipeline runs on a self-hosted agent, a Virtual Machine, which can communicate with the SAS Viya deployment. More about agents in a future post.   What Will the Pipeline Run?   The pipeline uses SAS-VIYA CLI (Command Line Interface) to interact with SAS Viya:   To run the SAS programs: sas-viya batch jobs submit-pgm .   To import the SAS studio flow: sas-viya transfer import .   To run the SAS studio flow: sas-viya job requests execute .   For the flow to run, it needs a job request. The easiest way to generate a job request is to schedule the job request. If you change the SAS Studio Flow, do not forget to re-schedule it, to get a new job request. Work is in progress for a REST API to run flows directly from the SAS code they generate.   Considerations and Open Discussion What can you achieve with such a pipeline? You can:   Group all the related technical jobs by objective. Adopt a process-oriented approach. Decouple the jobs. Suppose we have three developers working together to deliver the “data product”, the output file, in this example. Two prefer to code in SAS, one prefers SAS Studio flows. All of them have different working schedules. A pipeline with a Git repository would keep everything together. The same pipeline would give them the flexibility to work independently and incrementally on their programs or flows. Their customer will receive the "data product" earlier. Test if the whole is still working. The pipeline can run after each incremental change. The pipeline "build" implicitly tests if one of the jobs still works after changes were brought to another. Put SAS Viya in an enterprise perspective. The tasks in SAS Viya might be dependent on other tasks or scripts operating outside SAS Viya. For example, a bash script might be needed to copy the input files in a mount accessible in SAS Viya. Azure CLI commands might be required to copy the output files somewhere else, assign user permissions and so on. An Azure pipeline can contain tasks operating inside and outside SAS Viya. Combine different types of jobs. You can mix execute jobs with import jobs and manual review steps. Sequence and condition the execution of jobs. Start a job only after other jobs have completed successfully or not.   Conclusions   The post demonstrated a SAS Viya and Azure DevOps pipeline. The pipeline ran data management tasks in SAS Viya. All the pipeline code was stored in a Git repository (Azure Repos). The pipeline interacts with SAS Viya through the SAS Viya CLI. The post ended with an open discussion, when and why you might use SAS Viya and Azure Pipelines for data management tasks.   Next In a next post, we will look closer at a SAS Viya model release pipeline with Azure DevOps.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Resources   SAS Viya Decision Pipeline with Azure DevOps SAS Viya with Azure DevOps Data Pipeline SAS Viya CI/CD Pipelines with Azure DevOps SAS Viya and Azure Pipelines Self-Hosted Agents SAS Viya and Azure Pipelines Microsoft-Hosted Agents    Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about this topic. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

The post answers the following questions: Can you create SAS Viya CI/CD pipelines with Azure DevOps? What use cases can be covered? How to approach the set-up of a CI/CD pipeline? Read this post to find out more.   The goal is to share a series of recent experience with SAS Viya with Azure DevOps. The post series will cover:   An overview (this post). A data pipeline. A model and a model release pipeline. A decision pipeline. Azure DevOps agents. How to protect secrets in a pipeline. Conclusions and next steps.   CI/CD   A CI/CD pipeline is a sequence of events or jobs that can be executed. CI stands for continuous integration. CD means either continuous delivery or continuous deployment.   My CI/CD interpretation is to get “something developed in SAS Viya” (SAS program, SAS Studio flow, decision, model) into the hands of the end-users.   Why Use CI/CD Pipelines?   Automate the build and release process. Make the process repeatable and transparent. Improve quality and consistency.   What is Azure DevOps?   Azure DevOps is a Software as a service (SaaS) DevOps toolchain and orchestration platform from Microsoft. Azure DevOps is language, platform and cloud provider agnostic.       Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     Several services define Azure DevOps:   Azure Repos: Cloud-hosted private Git repositories. Azure Pipelines: Continuously build, test, and deploy to any platform and cloud. This is the CI/CD engine in Azure DevOps.   The focus of the post series is on Azure Repos and Azure Pipelines.   Azure DevOps also includes Azure Boards. These are agile tools to plan, track, and discuss work across your teams. Afterward, in Azure Artifacts you can create, host, and share packages with your team. Subsequently, Azure Test Plans help you test and ship with confidence using manual and exploratory testing tools. Lastly, through the Extensions Marketplace you can access hundreds of community-built extensions from Teams and Slack to SonarCloud and other apps and services.   Source: Microsoft Azure DevOps Overview.   Azure DevOps Rankings   Microsoft Azure DevOps is ranked by Peerspot.com: #1 in Release Automation tools, which includes RedHat Ansible and GitLab. #1 in Enterprise Agile Planning Tools, before Micro Focus ALM Octane, Planview LeanKit. #2 in Application Lifecycle Management Suites after Atlassian JIRA.   SAS Viya and Azure DevOps   Pipelines   With Azure DevOps, you can create pipelines operating outside or inside SAS Viya.   Outside   Outside SAS Viya, you could use Azure DevOps to:   Build a model or decision release pipeline. The release pipeline can deploy a SAS model or decision, stored as a container image, in Azure. The deployment target can be an Azure Web App, Container Instance or Kubernetes Service. Install software and packages on a cloud Virtual Machine. Deploy cloud resources. Deploy SAS Viya.   Inside   Inside SAS Viya, you can use Azure DevOps to:   Process data. Automate model tasks. Publish or deploy a decision. Score a model or a decision. Automate SAS administration tasks. Etc.   You are only limited by your imagination (and the amount of code you are willing to add to your Git repository and run inside the pipeline).   The post series will focus on the "inside SAS Viya" CI/CD pipelines.   Steps   You can resume the SAS Viya and Azure DevOps interaction in four steps:     Develop   The key idea is to store “the whole lot” ready for delivery or deployment in a Git repository.   “The whole lot” includes:   SAS artifacts: programs, SAS Studio flows, model and decision files, SAS content in JSON format. Non-SAS artifacts: Python programs, bash scripts, YAML files, etc. interacting with SAS Viya. In some cases, a small amount of static, non-private data, for tests, might be acceptable. “The whole lot” excludes: data, secrets, keys, certificates. We will see in a next post how to use SAS Viya with an Azure Key Vault in an Azure Pipeline.   Version   Version control is handled by a Git repository. In Azure DevOps, the default Git repository is stored in Azure Repos. You can choose GitHub, GitLab, Bitbucket instead of Azure Repos. GitHub, being a Microsoft company, is very well integrated with Azure DevOps.   The Azure pipeline definition, a YAML file, is stored in the same Git repository.   Build   The Azure pipeline clones the Git repository on an agent and executes the steps in the pipeline script.     Execute   Pipeline agents are virtual machines (VM). The agent interacts with a SAS Viya deployment through:   SAS Viya Command Line Interface (CLI). SAS Viya REST API.   The agents can be Microsoft hosted or self-hosted.   Microsoft hosted: At execution time, the pipeline will fetch, from the Azure cloud, a VM fulfilling certain requirements.   Self-Hosted: You can use a VM you have already deployed. You will have to register it as an agent, in Azure DevOps, before you can use it in a pipeline. Then you must install the software and the packages you need, copy the necessary files, etc.   For Azure DevOps to interact with SAS Viya, a key element is to use and configure a self-hosted agent. We will develop this topic in a future post.   Conclusions   The post briefly explained that you can create SAS Viya CI/CD pipelines with Azure DevOps (Pipelines and Repos). Next, the post looked at possible CI/CD pipelines within SAS Viya. Finally, it provided a four-step approach to set-up a CI/CD pipeline.   Next   In a next post, we will look closer at a SAS Viya with Azure DevOps Data Pipeline.     Resources   SAS Viya Model Release Pipeline with Azure DevOps SAS Viya Decision Pipeline with Azure DevOps SAS Viya with Azure DevOps Data Pipeline SAS Viya and Azure Pipelines Self-Hosted Agents SAS Viya and Azure Pipelines Microsoft-Hosted Agents  Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about the post content. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

Bitbucket is a Git-based source code repository hosting service. It is the third largest source code management software product. Some SAS customers are actively using Bitbucket. Which brings us to their question: can SAS Studio, on SAS Viya, integrate with Bitbucket? Read the post to find out the answer.   Bitbucket   Atlassian Account   Bitbucket is developed by Atlassian, the same company making the world-famous JIRA. To access Bitbucket you will require an Atlassian account. To open an Atlassian account is easy. It takes a few minutes to register a free account, which includes Bitbucket. You can use your corporate email to register an account.   Bitbucket Repository   To create a Bitbucket repository, you must create a project first. Within a project, you can register as many repositories as you like. The concept is similar with GitHub or GitLab. In fact, Bitbucket is using a “Git-based source code repository”. With Bitbucket, you will be on familiar territory, if you used GitHub or GitLab, at some point, in the past.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Bitbucket and SAS Viya   With the Bitbucket repository initialized, you can start thinking how to add it in SAS Viya. There are several questions you need to answer first:   How do I authenticate from SAS? You can choose from two options: HTTPS or SSH. I will be choosing HTTPS for this post. HTTPS communicates over port 443 and requires a user and a password. Do I prefer a visual interface, or am I OK to use git commands? If you require an interface, then the only obvious place to add a Bitbucket repository is in SAS Studio. Where do I clone the repository? You will need a folder on a network file share accessible from SAS Studio. The folder must have enough permissions to allow read, write and execute operations. Authentication: Bitbucket App Password   To access the Bitbucket repository, via HTTPS, you will require an app password. You can register one from your account’s personal settings. Following, you will have to select the permitted operations in Bitbucket.   Create the password and store it somewhere safe. You will not be able to retrieve the generated password later.   A Bitbucket app password is similar with a GitHub personal access token.       Bitbucket Repository URL   To clone the repository using HTTPS, your will require a repository URL, for example: https://sbxbot-admin@bitbucket.org/sbxbot/sasviya.git     Bitbucket User and Email   The user in this example is sbxbot-admin. It can be identified in the repository URL https://sbxbot-admin@bitbucket.org/sbxbot/sasviya.git as the part before `@bitbucket.org`.   The email is the one you used to register your Atlassian account.   SAS Viya   SAS Studio Settings   To clone the repository using HTTPS, your SAS Administrator must enable the following SAS Studio configuration options in SAS Environment Manager:   allowGit: to use the Git plug-in in SAS Studio. allowGitPassword: to use HTTPS as an authentication method. showServerFiles: to clone the repository on a network file share folder and see the cloned files.   As previously mentioned, Bitbucket is a “Git-based source code repository”, therefore the Git settings will work here just fine.       SAS Studio   In SAS Studio, to access the Bitbucket repository: Create a Git profile. Clone or add the repository. Create a Git Profile   Choose HTTPS, indicate the Bitbucket user, the user account email and copy the app password created earlier.     Clone or Add a Repository   Clone a Git Repository   Indicate the repository URL, an empty folder, on a Network File Share (NFS), accessible from SAS Studio and the Git profile.     Successful Clone   When you clone the repository, you will know you are on the right path when you will see something similar:     Cloned Repository History   In SAS Studio check the cloned repository history. You will see your earlier repository activity.     Explorer   To access the repository files, check in Explorer, under the folder where you cloned the repository.       You are now ready to version control your code into the Bitbucket repository.   Conclusion   Can SAS Studio on SAS Viya integrate with Bitbucket? Yes, it can. You can add a Bitbucket repository in SAS Studio, the same way you would add any other Git repository.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

You can use the decisiongitdeploy SAS Viya Command Line Interface plug-in to deploy decisions or rule sets from a Git repository to SAS Viya. The decisions can include SAS models in their composition. You can deploy the decisions as SAS Micro Analytic Service (MAS) modules in any number of SAS Viya environments. When you deploy SAS decisions to MAS, your decision can be scored through a REST API. You can call the REST API from any application, programming language or script.   Publish to Git - Deploy from Git   The deployment process, which is the focus of this post, is highlighted by the orange block in the diagram.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Publish to Git   A series of posts have already discussed the publishing from SAS Viya to a Git repository:   How to Publish SAS Decisions to Git: Published Files looked at the publishing and the files created. How to Publish SAS Decisions to Git: Publishing Destination created a Git destination. How to Publish SAS Decisions to Git: Required Configuration detailed the SAS Viya and Git configuration. How to Integrate SAS Intelligent Decisioning with Git gave an overview of the process.   Deploy from Git   The deployment process: Make accessible the Git files containing the published decisions to the SAS Viya Command Line Interface (CLI). Select the publishing destination. Use the decisiongitdeploy SAS Viya CLI to deploy these decisions, to the SAS Viya publishing destination.   The decisiongitdeploy SAS Viya CLI   In SAS Viya, a command-line interface (CLI) is a user interface to the SAS Viya REST services. In this interface, you enter commands on a command line and receive a response back from the system. With the decisiongitdeploy CLI you can deploy decisions and rule sets from a local Git repository to a SAS Micro Analytic Service (MAS) environment or to a SAS Cloud Analytic Services (CAS) environment. You must publish the decisions and rule sets to a Git publishing destination, then, copy them into your local Git repository before you use the CLI. The decisiongitdeploy CLI, was first released in the SAS Viya 2021.2 (November 2021) long term support.   Make Accessible the Git Files to the SAS Viya CLI   To access the files from your Git repository, an easy option is to clone the Git repository. If you decide to clone a private GitHub repository with SAS Studio:   Create a Git profile. HTTPS can be used as an authentication method. The GitHub Personal Access Token acts as a password. Choose a folder for the cloned repository. The example in the video uses a folder called gitDeploy.     Select the Publishing Destination   A Micro Analytic Service must exist prior to publishing. In most SAS Viya deployments, a MAS destination is created by default.   Use the decisiongitdeploy SAS Viya CLI   To deploy the decisions from the cloned files, you must use the SAS Viya Command Line Interface (CLI). The base assumption is that you have already installed the SAS Viya CLI.   Create a SAS Viya CLI Profile   First, you need to create a SAS Viya CLI profile. In our environment, the required script would look like:   # Get namespace and host kubectl get ns # Key variables – SAS Viya on RACE # The SAS Viya Ingress URL in this example points to https://gelenv.pdcesx1234.race.sas.com current_namespace=gelenv INGRESS_SUFFIX=$(hostname -f) INGRESS_URL=https://${current_namespace}.${INGRESS_SUFFIX} echo ${INGRESS_URL} # Get certificates (TLS support) export SSL_CERT_FILE=~/.certs/${current_namespace}_trustedcerts.pem export REQUESTS_CA_BUNDLE=${SSL_CERT_FILE} echo ${SSL_CERT_FILE} # Confirm the sas-viya CLI folder clidir=/opt/sas/viya/home/bin cd $clidir pwd # Create a SAS Viya CLI profile export SAS_CLI_PROFILE=${current_namespace} ./sas-viya --profile ${SAS_CLI_PROFILE} profile set-endpoint "${INGRESS_URL}" ./sas-viya --profile ${SAS_CLI_PROFILE} profile toggle-color off ./sas-viya --profile ${SAS_CLI_PROFILE} profile set-output fulljson ./sas-viya --profile ${SAS_CLI_PROFILE} profile show   SAS Viya CLI Login   Second, you need to login with your SAS credentials. This will allow you to get a SAS Viya access token and perform the operations required. My recommendation is to use the pyviyatools for authentication.   tee ~/.authinfo > /dev/null << EOF default user sasadm password ****** EOF chmod 600 ~/.authinfo # Login and create a token cd ~/admin/pyviyatools ./loginviauthinfo.py -f ~/.authinfo   The less elegant option is to directly employ the user and the password.   # Login and create a token ./sas-viya --profile ${SAS_CLI_PROFILE} auth login -user sasadm -password ******   Deploy to MAS   You must use the decisiongitdeploy plug-in to deploy the decision.   When you call the deploy command, you must be in the folder containing the decision files. This folder is where the Git repository was cloned locally.   In the example below, the gitDeploy folder contains two subfolders, one for each published rule set: autoAuction1_0 and autoAuctionMake1_0. The subfolder name is passed in the decisiongitdeploy command.     The deployment code looks like:   # Deploy from GIT to MAS # cd to the folder where the repository containing published was cloned cd /shared/gelcontent/home/sasadm/gitDeploy pwd # Confirm the folder where sas-viya CLI is installed ls /opt/sas/viya/home/bin/ # Deploy a decision from Git /opt/sas/viya/home/bin/sas-viya --profile ${SAS_CLI_PROFILE} decisiongitdeploy deploy --force true --destinationtype=MAS autoAuction1_0 # Deploy another decision from Git /opt/sas/viya/home/bin/sas-viya --profile ${SAS_CLI_PROFILE} decisiongitdeploy deploy --force true --destinationtype=MAS autoAuctionMake1_0 # Notes ## --destinationtype=MAS parameter is optional, as MAS is the default deployment destination ## --force true, allows to overwrite a MAS module if it already exists; results in an error if the module exists and the parameter is not specified   Conclusions   To deploy rule sets or decisions, from a Git repository, to a SAS Viya Micro Analytic Service (MAS) publishing destination: First, make the Git files accessible to the SAS Viya Command Line Interface (CLI). Second, select the existing publishing destination. Third, use the SAS Viya CLI decisiongitdeploy plug-in, to deploy to the chosen publishing destination.     Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about the post content. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more