@StephenFoerster  mentioned in Connecting Viya in Azure to On-Prem with Azure VPN, ExpressRoute (Intro): “Microsoft recommends two different mechanisms for connecting SAS Viya in Azure to on-premises resources […] Both methods require Azure Virtual Network (VNET) gateways to facilitate communication. Site to Site VPN connections also require VPN devices configured on premises.”   Read this post to learn how you can connect a SAS Viya on Azure deployment to an on-premises network by using site-to-site VPN gateways. When the connection is in place, you can access from SAS Viya resources from your on-premises data centre, for example, a database.   Context and Objective The previous post, How to Connect SAS Viya in Azure to On-Prem with VPN Gateways – Part 1: Introduced site-to-site VPN gateway connections. Simulated an on-premises data centre in Azure, to which we want to connect from SAS Viya. Provided the Azure CLI commands to create: Gateway subnets for each virtual network (VNET) you want to connect with Virtual Network Gateways. Local Network Gateways. Architecture Diagram As a reminder, by the end of the post series, we want to realize the following architecture diagram:     To reiterate the objective, we want to connect to the on-premises database from SAS Viya, through the VPN tunnel.   Assumptions SAS Viya deployed on Azure, in the $PREFIX-vnet mentioned in the diagram. You created the resources mentioned in How to Connect SAS Viya in Azure to On-Prem with VPN Gateways – Part 1. Create the Azure-side VPN gateway First, you'll create the VPN gateway for the Azure end of the connection. It can take up to 45 minutes to create a virtual network gateway. To save time, you can use Azure CLI commands with the --no-wait parameter to minimize the overall time required. This parameter lets you create both virtual network gateways simultaneously.   Public IP for the Virtual Network Gateway The VPN gateway must be associated with a public IP. Create the PIP-VNG-$PREFIX-VNet public IP address.   az network public-ip create \ --resource-group $RG \ --name PIP-VNG-${PREFIX}-vnet \ --allocation-method Dynamic   Virtual Network Gateway Create the VNG-$PREFIX-VNet virtual network gateway.   az network vnet-gateway create \ --resource-group $RG \ --name VNG-${PREFIX}-vnet \ --public-ip-addresses PIP-VNG-${PREFIX}-vnet \ --vnet ${PREFIX}-vnet \ --gateway-type Vpn \ --vpn-type RouteBased \ --sku VpnGw1 \ --no-wait     Note: you need to create a route-based gateway. Route-based VPNs use "routes" in the IP forwarding or routing table to direct packets into their corresponding tunnel interfaces. The tunnel interfaces then encrypt or decrypt the packets in and out of the tunnels.   Create the On-premises VPN gateway Next, you'll create a VPN gateway in HQ-Network to simulate an on-premises VPN device.   Public IP for the Virtual Network Gateway Create the PIP-VNG-HQ-Network public IP address.   az network public-ip create \ --resource-group $RGHQ \ --name PIP-VNG-HQ-Network \ --allocation-method Dynamic   Virtual Network Gateway Create the VNG-HQ-Network virtual network gateway   az network vnet-gateway create \ --resource-group $RGHQ \ --name VNG-HQ-Network \ --public-ip-addresses PIP-VNG-HQ-Network \ --vnet HQ-Network \ --gateway-type Vpn \ --vpn-type RouteBased \ --sku VpnGw1 \ --no-wait   Monitor the Creation Gateway creation takes approximately 45 minutes to complete. Run the command to check whether both virtual network gateways have been created.   az network vnet-gateway list \ --resource-group $RG \ --output table az network vnet-gateway list \ --resource-group $RGHQ \ --output table   The initial state will show Updating. You want to see Succeeded on both VNGs.     After each VPN gateway shows a ProvisioningState of Succeeded, you're ready to continue.   When Both Virtual Network Gateways Have Been Created The local network gateway IP must be updated to reflect the virtual network gateway IP from the vnet we want to connect to.   Effectively, you are “crossing” the IP addresses: The IP address of the VNG-HQ-Network from the HQ-Network is assigned to the LNG-HQ-Network from $PREFIX-vnet. The IP address of the VNG-$PREFIX-vnet from $PREFIX-vnet is assigned to the LNG-$PREFIX-vnet from HQ-Network.   Update the Azure Local Network Gateway IP Retrieve the IPv4 address assigned to PIP-VNG-HQ-Network and store it in a variable.   PIPVNGAZUREVNET=$(az network public-ip show \ --resource-group $RGHQ \ --name PIP-VNG-HQ-Network \ --query "[ipAddress]" \ --output tsv) echo $PIPVNGAZUREVNET   Update the LNG-HQ-Network local network gateway so that it points to the public IP address attached to the VNG-$PREFIX-VNet virtual network gateway.   az network local-gateway update \ --resource-group $RG \ --name LNG-HQ-Network \ --gateway-ip-address $PIPVNGAZUREVNET   Update the On-premises Local Network Gateway IP Retrieve the IPv4 address assigned to PIP-$PREFIX-VNet and store it in a variable.   PIPVNGHQNETWORK=$(az network public-ip show \ --resource-group $RG \ --name PIP-VNG-${PREFIX}-vnet \ --query "[ipAddress]" \ --output tsv) echo $PIPVNGHQNETWORK   Update the LNG-$PREFIX-VNet local network gateway so that it points to the public IP address attached to the VNG-$PREFIX-VNet virtual network gateway.   az network local-gateway update \ --resource-group $RGHQ \ --name LNG-${PREFIX}-vnet \ --gateway-ip-address $PIPVNGHQNETWORK     Create the Connections You'll now complete the configuration by creating the connections from each VPN gateway to the pair local network gateway. Shared Key Create the shared key to use for the connections. In the following command, replace  with a text string to use for the IPSec pre-shared key. The pre-shared key is a string of printable ASCII characters no longer than 128 characters. It can't contain special characters, like hyphens and tildes. You'll use this pre-shared key on both connections.   SHAREDKEY=1A2b3C4D5e6^7%8   In this example, any set of numbers will work for a shared key. In production environments, we recommend using a string no longer than 128 characters.   IKE: Internet Key Exchange (IKE) is a standard protocol used to set up a secure and authenticated communication channel between two parties via a virtual private network (VPN). The protocol ensures security for VPN negotiation, remote host and network access.   IPSec: IKE is a part of IPsec, a suite of protocols and algorithms used to secure sensitive data transmitted across a network. The Internet Engineering Task Force (IETF) developed IPsec to provide security through authentication and encryption of IP network packets and secure VPNs.   Create the First Connection Create a connection from VNG-$PREFIX-VNet to LNG-HQ-Network.   az network vpn-connection create \ --resource-group $RG \ --name ${PREFIX}-vnet-To-HQ-Network \ --vnet-gateway1 VNG-${PREFIX}-vnet \ --shared-key $SHAREDKEY \ --local-gateway2 LNG-HQ-Network   Create the Second Connection Create a connection from VNG-HQ-Network to LNG-$PREFIX-VNet.   az network vpn-connection create \ --resource-group $RGHQ \ --name HQ-Network-To-${PREFIX}-vnet \ --vnet-gateway1 VNG-HQ-Network \ --shared-key $SHAREDKEY \ --local-gateway2 LNG-${PREFIX}-vnet   LNG-$PREFIX-VNet contains a reference to the public IP address associated with the VNG-$PREFIX-VNet VPN gateway. This connection would normally be created from your on-premises device.   You've now finished the configuration of the site-to-site connection. This will take a few minutes, but the tunnels should automatically connect and become active.   Verify the Connections Let's confirm that the VPN tunnels are connected.   az network vpn-connection show \ --resource-group $RG \ --name ${PREFIX}-vnet-To-HQ-Network \ --output table \ --query '{Name:name,ConnectionStatus:connectionStatus}'     az network vpn-connection show \ --resource-group $RGHQ \ --name HQ-Network-To-${PREFIX}-vnet \ --output table \ --query '{Name:name,ConnectionStatus:connectionStatus}'     You shall see:     Conclusions The site-to-site VPN tunnel is now complete. Virtual network gateways were created in Azure and "on-premises" (in the simulated data centre). The local network gateway IP was updated to reflect the virtual network gateway IP from the VNET we want to connect to. Site-to-site VPN gateway connection was configured. VPN connections have been successfully established. Resources in SAS Viya subnets $PREFIX-misc-subnet and $PREFIX-aks-subnet can now access resources from the Applications subnet. Read the next post, How to Connect SAS Viya in Azure to On-Prem with VPN Gateways - Part 3, where you will learn how to test the access to the on-premises database from SAS Viya and verify the traffic over the VPN tunnel.   Useful Resources How to Connect SAS Viya in Azure to On-Prem with VPN Gateways – Part 1 Connecting Viya in Azure to On-Prem with Azure VPN, ExpressRoute (Intro) SAS Cloud Data Exchange for the SAS Viya Platform Connect on-premises networks to Azure by using site-to-site VPN gateways Exercise - Prepare Azure and on-premises virtual networks by using Azure CLI commands Exercise - Create a site-to-site VPN gateway by using Azure CLI commands CIDR to IP range   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about access to on-premises datacentres using VPN gateways.  If you wish to get more information, please write me an email. ... View more

@StephenFoerster mentioned in Connecting Viya in Azure to On-Prem with Azure VPN, ExpressRoute (Intro): “Microsoft recommends two different mechanisms for connecting SAS Viya in Azure to on-premises resources […] Both methods require Azure Virtual Network (VNET) gateways to facilitate communication. Site to Site VPN connections also require VPN devices configured on premises.”   Read this post to learn how you can connect a SAS Viya on Azure deployment to an on-premises network by using site-to-site VPN gateways. When the connection is in place, you can access from SAS Viya resources in your on-premises data centre, a database, for example.   Context and Objective You have SAS Viya deployed on Azure. All resources are deployed in a virtual network. You also have an on-premises data centre with one database. The database is fenced by its own virtual network.   The objective is to access this database from SAS Viya on Azure, through the VPN tunnel.   One way to achieve the objective is to connect the on-premises datacentre to the Azure virtual network (VNET) through a site-to-site VPN gateway connection.   VPN: A virtual private network (VPN) is a type of private interconnected network. VPNs use an encrypted tunnel within another network. They're typically deployed to connect two or more trusted private networks to one another over an untrusted network, typically the public Internet. Traffic is encrypted while traveling over the untrusted network to prevent eavesdropping or other attacks.   Azure VPN gateway: A VPN gateway is a type of Virtual Network Gateway (VNG). VPN gateways are deployed in Azure virtual networks and can connect on-premises datacentres to Azure virtual networks through a site-to-site connection.   Simulated Data Centre In this post we are going to simulate the on-premises data centre. The on-premises datacentre will be represented by a database in an Azure virtual network, called HQ-Network.   Why we chose to simulate the data centre? There are many types of on-premises VPN devices, and it is not possible to describe their configuration in a single post.   What you need to remember is the logic and the configuration steps. You just need to replace the steps for the simulated network with steps tailored to your on-premises VPN device.   Architecture Diagram By the end of the post series, we want to realize the following architecture diagram:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Where: $PREFIX-vnet is the SAS Viya virtual network: $PREFIX-aks-subnet is the sub-network of the SAS Viya Azure Kubernetes Service (AKS) cluster and its resources. $PREFIX-misc-subnet is the sub-network of the several virtual machines, including $PREFIX-jump-VM. A separate gateway subnet must be created for the Azure Virtual Network Gateway (VNG). HQ-Network is the on-premises network that you want to access from Azure: It has an Applications subnet. In this subnet, a database server and a private endpoint are hosted. We created a private endpoint, to force the database traffic, network-to-network, through the VPN tunnel. We want to avoid SAS Viya connections directly to the database over the Public Internet. A separate gateway subnet must be created for the on-premises VNG. $PREFIX-vnet will be connected to HQ-Network via a Virtual Network Gateway (VNG) / Local Network Gateway (LNG) pair. HQ-Network will be connected to $PREFIX-vnet via a second VNG / LNG pair. The connections are establishing the VPN tunnel. To reiterate the objective, we want to connect to the on-premises database from SAS Viya, through the VPN tunnel.   Assumption SAS Viya deployed on Azure, in the $PREFIX-vnet mentioned in the diagram.   Azure Side Resources   You can use the Azure CLI to create the resources: Variables   RGHQ=HQ-rg RG=resource_group_for_SAS_Viya_resources PREFIX=prefix_for_SAS_Viya_resources     Add a Gateway Subnet Add a gateway subnet to $PREFIX-vnet. Every VNG needs a gateway subnet. $PREFIX-vnet has an address space of 192.168.0.0/16 (65,536 IPs), therefore you must choose at least a /27 address space (32 IPs):   az network vnet subnet create \ --resource-group $RG \ --vnet-name $PREFIX-vnet \ --address-prefixes 192.168.255.0/27 \ --name GatewaySubnet         Create a Local Network Gateway A local network gateway (LNG) is a specific object that represents your on-premises location (the site) for routing purposes. You give the site a name by which Azure can refer to. You then specify the IP address of the on-premises VPN device to which you'll connect.   You must specify the IP address prefixes that can be routed through the VPN gateway to the VPN device.   Create the LNG-HQ-Network local network gateway. The local address prefixes describe your on-premises vnet or subnets. For example, 172.16.0.0/24 corresponds to the Applications subnet in the HQ-Network vnet.   az network local-gateway create \ --resource-group $RG \ --gateway-ip-address 94.0.252.160 \ --name LNG-HQ-Network \ --local-address-prefixes 172.16.0.0/24   This local gateway represents the on-premises network that you’re connecting to. The IP address specified as the remote gateway, the simulated on-premises network, needs to be updated later because it has not yet been created.   The address prefixes are very important! They define the local resources you can reach through the gateway from Azure.       Simulated On-premises Resources   Create the HQ-Network Virtual Network and the Applications Subnet Create the HQ-Network virtual network and the Applications subnet in a separate resource group.   az group create --name $RGHQ --location eastus az network vnet create \ --resource-group $RGHQ \ --name HQ-Network \ --address-prefixes 172.16.0.0/16 \ --subnet-name Applications \ --subnet-prefixes 172.16.0.0/24   Add a Gateway Subnet to HQ-Network Add GatewaySubnet to HQ-Network.   az network vnet subnet create \ --resource-group $RGHQ \ --address-prefixes 172.16.255.0/27 \ --name GatewaySubnet \ --vnet-name HQ-Network   Create a Local Network Gateway Create the LNG-$PREFIX-VNet local network gateway.   az network local-gateway create \ --resource-group $RGHQ \ --gateway-ip-address 94.0.252.160 \ --name LNG-$PREFIX-vnet \ --local-address-prefixes 192.168.2.0/24 192.168.0.0/23   The gateway’s --local-address-prefixes points to the Azure network and subnets you're connecting to. As you can see, the address space points to $PREFIX-misc-subnet and $PREFIX-aks-subnet address ranges: 192.168.2.0/24 and 192.168.0.0/23.         You'll update later the IP address specified as the remote gateway, which is in Azure.       Verify the Topology   Verify the Virtual Networks Verify that the virtual networks have been successfully created.   az network vnet list \ --resource-group $RG \ --output table az network vnet list \ --resource-group $RGHQ \ --output table         Verify the Local Network Gateways Verify the local network gateways have been successfully created.   az network local-gateway list \ --resource-group $RG \ --output table az network local-gateway list \ --resource-group $RGHQ \ --output table         Conclusions For a site-to-site VPN gateway connection: In each virtual network (VNET) you want to connect, you need a Gateway Subnet where the Virtual Network Gateway will be hosted There must not be addresses that overlap in the VNETs you want to connect The address prefixes in each Local Network Gateway are very important! They define the local resources you can reach through the gateway from Azure. In Part 2   Read the next post, How to Connect SAS Viya in Azure to On-Prem with VPN Gateways - Part 2, where you will learn how to: Create the Azure side and the on-premises Virtual Network Gateways. Connect the two Virtual Network Gateways. Useful Resources Connecting Viya in Azure to On-Prem with Azure VPN, ExpressRoute (Intro) How to Connect SAS Viya in Azure to On-Prem with VPN Gateways - Part 2 SAS Cloud Data Exchange for the SAS Viya Platform Connect on-premises networks to Azure by using site-to-site VPN gateways Exercise - Prepare Azure and on-premises virtual networks by using Azure CLI commands Exercise - Create a site-to-site VPN gateway by using Azure CLI commands CIDR to IP range Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about access to on-premises datacentres using VPN gateways.  If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here.   ... View more

Read the post and watch the videos to learn how to successfully clone a GitHub repository in SAS Studio on SAS Viya using SSH authentication. The content of the post was developed with SAS Viya Long Term Support Release 2022.09, deployed to Azure. You will also learn a few tricks that will save you plenty of research time.   Initially, I tried many times to clone a GitHub repository in SAS Studio on SAS Viya using SSH and failed every time. What worked with SAS Viya 3.5 seemed not to work anymore, for some reason. When I finally succeeded, I learnt that there are a few elements that make all the difference between failure and success.   Learnings The type of SSH keys you generate matters: ECDSA keys are accepted by SAS and GitHub. The ownership of the folder where the SSH keys are stored is particularly important. The clone and the SSH keys must be stored on a server file (network file share or mounted path). It will not work if you use SAS Content. Pre-Requisites   Physical Infrastructure   In this example, SAS Viya is deployed to Azure. On Azure, the SAS Viya deployment comes with a virtual machine called jump-vm.   To generate the SSH keys, I am going to use this virtual machine (VM). The VM has access to folders on a Network File Share (NFS). The NFS has been mounted in the SAS Compute pod at deployment time. SAS Studio Settings   In SAS Environment Manager, your SAS Administrator must configure these properties for the Git Integration. Follow the link for more details. Steps to Clone using SSH Generate the SSH Keys The SAS Note 69005 - An SSH authentication fails and you see "...The specified URL or Public SSH key is invalid for remote repository" (sas.com) suggests that you are better off generating ECDSA keys when working with GitHub. ECDSA stands for Elliptic Curve Digital Signature Algorithm. RSA SHA-1 keys are no longer supported by GitHub, while ed25519 keys are not supported by SAS. On the Linux machine, for example the jump-vm, generate the ECDSA key:     ssh-keygen -t ecdsa -b 521 -C 'your_email@domain.com'   I chose an empty passphrase, although phrases are supported in SAS Studio.   Copy the Content of the SSH Public Key On the Linux machine where you generated the keys, copy the content of the public key:     ls -la .ssh cat ~/.ssh/id_ecdsa.pub   Add the SSH Public Key in GitHub To use SSH with a GitHub repository, you have to add your public key in your GitHub account. For more information, see Adding a new SSH key to your GitHub account.   The Key text box hints you what type of keys GitHub accepts. I tried initially with a ssh-rsa key and it failed. I tried with an ed25519 key and it also failed, as these are not supported by SAS. Use ecdsa keys.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page. Get the GitHub Repository Details To create Git profile in SAS Studio, you would need: The SSH URL, for example: git@github.com:bteleuca/git-demo.git The GitHub user, for example bteleuca The email registered with your GitHub account. Test Clone on the Jump VM To know if you can reach the GitHub repository and if you are allowed to clone, execute a quick test on your Jump VM:     # clone the git demo repository in a new folder cd /viya-share/azuredm/code ls -la mkdir -p gitrepo1 chmod -R 777 gitrepo1 ls -la cd gitrepo1 git clone git@github.com:bteleuca/git-demo.git cd git-demo ls -la     You should see the cloned repository in a VM folder, proving the SSH clone works.   Download – Upload Your SSH Keys   You can either:    Download the SSH keys from the VM and upload them in SAS Studio. This will automatically set the right file ownership. Copy the SSH keys on the Jump VM. Do not forget to set the right ownership.   Supposing you go with the first option:     Download the SSH keys from your VM. In SAS Studio: Create a folder e.g. sshkeys. Upload the public and the private SSH keys.   Note that this folder has to be on a Network File Share (SAS Server files). It will not work if your keys are in a folder under SAS Content or My Folder.     Create a SSH Profile From Manage Git connections > Add a Profile. Choose SSH:     Fill in:   The SSH URL, for example: git@github.com:bteleuca/git-demo.git The GitHub user, for example bteleuca The email registered with your GitHub account. The passphrase is empty, in this example. Otherwise, fill in the passphrase if you chose one when you created the SSH keys. Select the public and the private key from the server folder. Clone the Repository   From Manage Git connections > Repositories > Clone a repository. Repository: Your GitHub SSH URL. E.g.: git@github.com:bteleuca/git-demo.git Server location: choose an empty folder or create a new one. Profile: Choose the profile you defined earlier.   Success Looks Like This     Folder and SSH Keys Files Ownership   On the Jump VM check the permissions and file ownership.     You will need sudo for this operation:     sudo ls -la /viya-share/azuredm/sshkeys     The ownership of the folder is really important! The number 1803... corresponds to the SAS Viya user identity that created the folder.   Adjust the Permissions   At this point the permissions are too wide for SSH keys. To be safe, restrict them to: Folder: chmod 700 -rwx------ Owner: read, write and execute. Public key: chmod 644 -rw-r--r-- Owner: read, write and execute. Group: read. Other: read. Private key: chmod 600 -rw------- Owner: read, write.   sudo chmod 700 /viya-share/azuredm/sshkeys sudo chmod 644 /viya-share/azuredm/sshkeys/id_ecdsa.pub sudo chmod 600 /viya-share/azuredm/sshkeys/id_ecdsa sudo ls -la /viya-share/azuredm/ sudo ls -la /viya-share/azuredm/sshkeys   Push Some Files Remotely   The objective is to test if you can push some files remotely. Although the clone operation works, you can be sure of your setup only when you manage to push.     Create a File   Write a SAS program and save it in a folder under your cloned repository:     PROC SORT DATA=sashelp.cars OUT=auto2 ; BY Model ; RUN ; PROC PRINT DATA=auto2 ; RUN ;     Push Process   Optional: Create a new branch. Checkout to that branch. Stage the changes. Commit with a message. Push the code. Optional: Merge the branch back into main. Look at the Git repository History tab in SAS Studio or, check remotely the changes (on the respective branch).   Additional Git Operations in SAS Studio   In this bonus section, you can watch how to add a repository in SAS Studio or delete a repository.     Conclusions   You can clone a GitHub repository in SAS Studio on SAS Viya. Attention points: The type of SSH keys you generate matters: ECDSA keys are accepted by SAS and GitHub. The ownership of the folder where the SSH keys are stored is particularly important. The clone and the SSH keys must be stored on a server file (network file share or mounted path). It will not work if you use SAS Content. Additional Resources Understanding Git Integration in SAS Studio - SAS Documentation. For SAS Viya 3.5 / Studio 5.2 see the YouTube by @XavierBizoux : Setting up Git repository in SAS Studio 5.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about SAS Studio and Git SSH. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

Learn how to import, register an analytical model in SAS Viya, publish and deploy automatically to Azure, where it will be ready for scoring requests, in less than eight minutes. Read on for the details.   Why SAS Viya and Azure Pipelines?   Stack Overflow wrote:   Data scientists excel at creating models that represent and predict real-world data, but effectively deploying machine learning models is more of an art than science. Deployment requires skills more commonly found in software engineering and DevOps. Venturebeat reports that 87% of data science projects never make it to production.   With SAS Viya and the Azure cloud platform you can tip the balance in your favor. Models that do not get deployed can become the exception, not the rule.   Watch the Pipelines in Action   In just under eight minutes, you can convert a dormant ZIP file into a registered model in SAS Model Manager, a container image, and a functional REST API that is prepared for scoring on Azure.       What is Required   SAS Viya   SAS Viya Long Term Support version 2022.09, 2021.2 or 2021.1. For convenience, the SAS Viya in our examples is deployed in Azure. SAS Model Manager licensed. You must create a project in SAS Model Manager prior to importing models. An Azure publishing destination in SAS Viya. SAS Container Runtime (SCR) is a key component for converting a model into a container image.   Azure Cloud Access to an Azure subscription and sufficient permissions. An Azure Container Registry is required for the Azure publishing destination. See How to Publish a SAS Model to Azure with SCR: A Start-to-Finish Guide or Creating model publishing destinations using the SAS Viya CLI. An Azure Web App. The web app serves as a target for the model deployment.   Azure DevOps Access to an Azure DevOps Organization. Create a release pipeline that deploys the model to an Azure Web App. For more information, refer to SAS Viya Model Release Pipeline with Azure DevOps. A self-hosted agent pool with at least one agent that can “communicate” with your SAS Viya deployment. The self-hosted agent is a virtual machine and must have sas-viya CLI installed, Python, certificates copied, etc. Refer to the following resources for instructions: An Azure pipeline that uses the self-hosted agent. A Git repository.   How It All Works     The Detailed Steps   The Model   Create and test an analytical model. Export your model to a ZIP file. As an alternative, you can use a zipped model form the SAS Model Manager Quickstart Tutorial.   The Git Repository   Upload the model ZIP file to a Git repository, such as Azure Repos, GitHub, GitLab, Bitbucket. Create a requirements.txt file for Python packages. Insert just one word inside: sasctl. The sasctl package enables easy communication between the SAS Viya platform and a Python runtime. Read Sophia Rowland’s post or the python-sasctl SAS Software GitHub project for more information.   Register Model Program   Create a Python program called SCRModelRegister.py, that uses sasctl to register a model in SAS Model Manager from the ZIP file in your Git repository.   # Import necessary packages import sys import os import requests import shutil from sasctl import Session from sasctl.services import model_repository as mr, model_management as mm import urllib3 import argparse # Create an ArgumentParser object to handle command-line arguments parser = argparse.ArgumentParser() parser.add_argument('--host', type=str, required=True) parser.add_argument('--user', type=str, required=True) parser.add_argument('--password', type=str, required=True) parser.add_argument('--protocol', type=str, required=True) parser.add_argument('--repository_name', type=str, required=True) parser.add_argument('--project_name', type=str, required=True) parser.add_argument('--model_name', type=str, required=True) parser.add_argument('--model_folder', type=str, required=True) args = parser.parse_args() # Print the command-line arguments print('SAS Viya URL: ', args.host) print('SAS User Name: ', args.user) #print('SAS User Password: ', args.password) print('Protocol: ', args.protocol) print('SAS Model Repository Name: ', args.repository_name) print('SAS Model Project Name: ', args.project_name) print('SAS Model Name: ', args.model_name) print('SAS Model Folder: ', args.model_folder) # Set the path to the SSL certificates pem_path='/home/jumpuser/.certs/my_trustedcerts.pem' # Create a session using the specified arguments s = Session(args.host, args.user, args.password, args.protocol, verify_ssl=pem_path) # List object properties print("Object doc: s") print(dir(s)) print("Object doc: s._request_token_with_oauth") print(dir(s._request_token_with_oauth)) # Get an access token using OAuth at=s._request_token_with_oauth(args.user, args.password).access_token # Retrieve a repository and project from SAS Model Manager repository = mr.get_repository(args.repository_name) print(f"Repository retrieved: {repository.name} with {repository.id}") project = mr.get_project(args.project_name) print(f"Project retrieved: {project.name} with {project.id}") # Prepare to post the model to the specified URL url = f"{args.host}/modelRepository/models" print("Will POST the model", args.model_name, " to this url: ", url) headers = {'Authorization': f'Bearer {s._request_token_with_oauth(args.user, args.password).access_token}'} payload={ 'name': args.model_name, 'type': 'GENERIC', 'projectId': project.id } print("This model will be posted: ") print(args.model_folder + "/" + args.model_name + ".zip") files={'files' : open(args.model_folder + "/" + args.model_name + ".zip",'rb')} # Post the model to the specified URL register_model = requests.post( url, data = payload, headers = headers, files = files, verify=pem_path ) # Print the content of the response print(register_model.content)   Publish Model Program   Create a Python program called SCRModelPublish.py, that publishes the registered model from SAS Model Manager, to Azure, through the Azure publishing destination.   # Import necessary packages import sys import os import requests import shutil from sasctl import Session from sasctl.services import model_repository as mr, model_management as mm import urllib3 import argparse # Create an ArgumentParser object to handle command-line arguments parser = argparse.ArgumentParser() parser.add_argument('--host', type=str, required=True) parser.add_argument('--user', type=str, required=True) parser.add_argument('--password', type=str, required=True) parser.add_argument('--protocol', type=str, required=True) parser.add_argument('--repository_name', type=str, required=False) parser.add_argument('--project_name', type=str, required=False) parser.add_argument('--model_name', type=str, required=True) parser.add_argument('--model_folder', type=str, required=False) parser.add_argument('--dest_name', type=str, required=True) parser.add_argument('--model_publish_name', type=str, required=True) parser.add_argument('--model_publish_notes', type=str, required=True) args = parser.parse_args() # Print the command-line arguments print('SAS Viya URL: ', args.host) print('SAS User Name: ', args.user) #print('SAS User Password: ', args.password) print('Protocol: ', args.protocol) print('SAS Model Repository Name: ', args.repository_name) print('SAS Model Project Name: ', args.project_name) print('SAS Model Name: ', args.model_name) print('SAS Model Folder: ', args.model_folder) print('Azure Publishing Destination Name: ', args.dest_name) print('SAS Model Publish Name: ', args.model_publish_name) print('SAS Model Publish Notes: ', args.model_publish_notes) # Set the path to the SSL certificates pem_path='/home/jumpuser/.certs/my_trustedcerts.pem' # Create a session using the specified arguments s = Session(args.host, args.user, args.password, args.protocol, verify_ssl=pem_path) # Get an access token using OAuth at=s._request_token_with_oauth(args.user, args.password).access_token # Get the model from SAS Model Manager model = mr.get_model(args.model_name) print(model) # Prepare to publish the model to the specified URL headers = { "Content-Type": "application/vnd.sas.models.publishing.request.asynchronous+json", "Authorization": f"Bearer {s._request_token_with_oauth(args.user, args.password).access_token}" } # to publish model under a different name can use "modelName": args.model_publish_name # to publish model under the same name can use "modelName": args.model_name payload = { "destinationName": args.dest_name, "modelContents": [ { "modelName": args.model_publish_name, "publishLevel": "model", "sourceUri": f"/modelRepository/models/{model.id}" } ], "name": args.model_publish_name, "notes": args.model_publish_notes } # Post/publish the model to the specified URL model_publish = requests.post( url=f'{args.host}/modelManagement/publish?force=true', headers=headers, data= json.dumps(payload), verify=pem_path ) # Print the content of the response print(model_publish.content)   The Azure Pipeline   Create a pipeline in Azure DevOps that uses the Git repository. Create a YAML script for the Azure pipeline that includes instructions to: Install pip on the agent machine. Install the packages defined in txt. Run the SCRModelRegister.py program with several arguments. Run the SCRModelPublish.py program with several arguments. Use sas-viya models CLI to list the SAS Model Manager models, published models, projects, and repositories. Run or trigger the pipeline. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Pipeline YML   # Python script importing a model from a .zip in SAS Viya Model Manager # A second Python script publishes the model from SAS Viya Model Manager to Azure # the SAS Model Manager repository and project must be created before # the Azure publishing destination must be created before # the pipeline passes parameters to the python scripts. name: 06_051_Model_Automation parameters: - name: MODEL_NAME displayName: Model file name without the .zip at the end type: string default: QS_Reg1 trigger: - main pool: name: $(pool) stages: - stage: models displayName: Manage Models in SAS Model Manager jobs: - job: importModel displayName: Import Model in SAS Model Manager steps: - checkout: self persistCredentials: true - task: AzureKeyVault@2 inputs: azureSubscription: '$(azureSubscriptionSC)' KeyVaultName: '$(keyvault)' SecretsFilter: '*' RunAsPreJob: true - script: | echo Check python python3 --version echo Python is installed echo Install pip3 sudo apt install python3-pip -y displayName: 'Install Pip 3.x' - script: | echo 'List directory where Git Repo was cloned: ' && pwd ls -la echo Get the folder where the Git repo was cloned export GITFOLDER=$(pwd) echo Git Pull git pull origin main ls -la echo Install Requirements pip3 install -r $GITFOLDER/requirements.txt displayName: 'Install Python Requirements' - task: Bash@3 inputs: targetType: 'inline' script: | echo 'List the folder where Git Repo was cloned: ' && pwd ls -la echo Get the folder where the Git repo was cloned export GITFOLDER=$(pwd) echo List the Python programs folder ls -la $GITFOLDER/programs echo List the models folder ls -la $GITFOLDER/programs echo Arguments to pass to the python script export INGRESS_URL=https://my_viya_url echo Viya URL: ${INGRESS_URL} SAS_USER=sas_user_here #SASPASS Will be retrieved from the Azure Key Vault $(keyvault) as a variable $(SASPASS) PROTOCOL=https REPO_NAME=Public PROJECT_NAME=QS_HMEQ echo Path to certificates - ADAPT PATH if NEEDED REQUESTS_CA_BUNDLE=~/.certs/my_trustedcerts.pem echo Execute Python Program with Parameters python3 $GITFOLDER/programs/SCRModelRegister.py --host $INGRESS_URL --user $SAS_USER --password $(SASPASS) --protocol $PROTOCOL --repository_name $REPO_NAME --project_name $PROJECT_NAME --model_name ${{ parameters.MODEL_NAME }} --model_folder $GITFOLDER/content > ~/SCRModelRegister.log echo List Execution Log cat ~/SCRModelRegister.log displayName: Import Model in SAS Model Manager - job: publishModel displayName: Publish Model to Azure dependsOn: importModel steps: - checkout: self persistCredentials: true - task: AzureKeyVault@2 inputs: azureSubscription: '$(azureSubscriptionSC)' KeyVaultName: '$(keyvault)' SecretsFilter: '*' RunAsPreJob: true - task: Bash@3 inputs: targetType: 'inline' script: | echo 'List the folder where Git Repo was cloned: ' && pwd ls -la echo Get the folder where the Git repo was cloned export GITFOLDER=$(pwd) echo List the Python programs folder ls -la $GITFOLDER/programs echo List the models folder ls -la $GITFOLDER/programs echo Arguments to pass to the python script export INGRESS_URL=https://my_viya_url echo Viya URL: ${INGRESS_URL} SAS_USER=sas_user_here #SASPASS Will be retrieved from the Azure Key Vault $(keyvault) as a variable $(SASPASS) PROTOCOL=https REPO_NAME=Public PROJECT_NAME=QS_HMEQ DEST_NAME=AzureCLI echo Path to certificates - ADAPT PATH if NEEDED REQUESTS_CA_BUNDLE=~/.certs/my_trustedcerts.pem echo Execute Python Program with Parameters python3 $GITFOLDER/programs/SCRModelPublish.py --host $INGRESS_URL --user $SAS_USER --password $(SASPASS) --protocol $PROTOCOL --model_name ${{ parameters.MODEL_NAME }} --dest_name $DEST_NAME --model_publish_name "sasmodel" --model_publish_notes "Published using REST API" > ~/SCRModelPublish.log echo List Execution Log cat ~/SCRModelPublish.log displayName: Publish Model from SAS Model Manager to Azure - job: listModels displayName: List Models in SAS Model Manager dependsOn: publishModel steps: - checkout: self persistCredentials: true - task: AzureKeyVault@2 inputs: azureSubscription: '$(azureSubscriptionSC)' KeyVaultName: '$(keyvault)' SecretsFilter: '*' RunAsPreJob: true - script: | echo 'List the folder where the Git Repo was cloned: ' && pwd ls -la echo Get the folder where the Git repo was cloned in a variable export GITFOLDER=$(pwd) echo List the cloned Git Repo programs folder ls -la $GITFOLDER/programs echo Create SAS-Viya CLI Profile echo Initialize needed variables pwd && ls -la export SAS_CLI_PROFILE=gelenv echo $SAS_CLI_PROFILE export current_namespace=my export CURL_CA_BUNDLE=~/.certs/${current_namespace}_trustedcerts.pem echo $CURL_CA_BUNDLE export SSL_CERT_FILE=~/.certs/${current_namespace}_trustedcerts.pem echo ${SSL_CERT_FILE} export REQUESTS_CA_BUNDLE=${SSL_CERT_FILE} echo ${REQUESTS_CA_BUNDLE} export INGRESS_URL=https://my_viya_url echo Viya URL: ${INGRESS_URL} clidir=/opt/sas/viya/home/bin echo SAS-Viya CLI directory is: $clidir echo Create SAS-Viya CLI Profile $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile set-endpoint ${INGRESS_URL} $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile toggle-color on $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile set-output fulljson echo Show profile $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile show echo Login in SAS_USER=sas_user_here $clidir/sas-viya --profile ${SAS_CLI_PROFILE} auth login -user $SAS_USER -password $(SASPASS) echo *************** END COMMON TRUNK ******************* echo Model Management Module echo Switching SAS Output to text export SAS_OUTPUT=text echo SAS Model Manager echo Get a List of Repositories $clidir/sas-viya --profile ${SAS_CLI_PROFILE} --output text models repository list echo Get a List of Projects $clidir/sas-viya --profile ${SAS_CLI_PROFILE} --output text models project list echo List models $clidir/sas-viya --profile ${SAS_CLI_PROFILE} --output text models list echo Get a List of Publishing Destinations $clidir/sas-viya --profile ${SAS_CLI_PROFILE} --output text models destination list echo Get a List of Published Objects $clidir/sas-viya --profile ${SAS_CLI_PROFILE} --output text models published list displayName: SAS Model Manager Lists   The pipeline requires several variables to be adapted: MODEL_NAME: name of the model without extension. pool: name of the self-hosted pool, configured to work with SAS Viya. azureSubscriptionSC: the service connection authorizing the Azure DevOps project to access resources in an Azure subscription. keyvault: the keyvault where secrets are stored. SAS_USER:user. INGRESS_URL: SAS Viya URL without / at the end. current_namespace: Kubernetes namespace where SAS Viya is deployed. SASPASS: sas user password stored in the keyvault above; retrieved at runtime. REQUESTS_CA_BUNDLE: path to the SSL certificates. Should be the same as the pem_path in the Python programs.   The Release Pipeline   The pipeline publishes the model to Azure and triggers a release pipeline that deploys the model in a Web App. The release pipeline includes a trigger, artifact, and stage with a task to deploy the container image to an Azure web app when a new Docker container image sasmodel:latest arrives in the Azure Container Registry. For more information, refer to SAS Viya Model Release Pipeline with Azure DevOps.   Traceability   The model registration and publishing is captured in SAS Model Manager.       The publishing captures the published container image name and the tag.       From this point forward, you have to look at the release pipeline logs in Azure DevOps or at the Azure Web App logs.   Conclusions   Before you know it, you can successfully import and register your analytical model in SAS Viya, and then have it published to Azure. It is amazing how quick and easy the entireprocess is. And you can start scoring your newly-deployed model straight away.   There are some serious benefits for data scientists everywhere using this method. It can help them deploy their models faster to production.   Acknowledgements   Thanks to my colleague @YiJianChing for helping out with sasctl.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about Python and SAS Viya REST APIs. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

SAS customers need to run a SAS Studio Flow in batch or from a CI/CD pipeline. They also need to version the flow code, possibly in a Git repository and compare the changes from one version to another. To achieve these goals, you need the SAS code behind the SAS Studio Flow. A REST API helps you generate the code. Read the post to find out how you can call the SAS Viya REST API from Python.   The post How To Get the Code From a SAS Studio Flow and Why It Matters explained how to get the SAS code from a SAS Studio Flow by writing a SAS program (proc http). You can write a Python program to achieve the same result.   Python Program Calling SAS Viya REST APIs   The studioDevelopment REST API has been available since the SAS Viya 2022.09 Long Term Support. I tested the program below with Python version 3.8, 3.9 and 3.10.   SAS Studio Flow   Suppose you developed a flow that performs a de-duplication of an existing table.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   To generate the SAS code form this flow you could write a Python program:   The first part of the program returns a SAS Viya Token. The second part generates the SAS code from a SAS Studio Flow.   Get a SAS Viya Access Token   There are several ways to get a token. I will just pick one method I favor for several reasons.   Packages   Several packages are required for this program to run.   # 1 Packages import requests import urllib3 import json, os, pprint import getpass import base64   Variables   The unsafe choice is to hardcode credentials in the program. It might be suitable for a quick test, although you should never save it in a program file. The file can end up in a Git repository. The Git repository can become public therefore compromising your credentials.   Never hardcode credentials directly in the program.   Instead of hardcoding variables you can pass them as arguments. Two reasons for using arguments:   Avoid hardcoding. In production environments, Python programs are executed from the command line most of the time. Therefore, it helps if you pass the “changing” part as an argument.   In this example:   The first argument is the Python program name. More about it in the execution part. The RG is the second argument ([1] in Python) in the argument array. The client_id is the third argument ([2] in Python) in the argument array. The client_secret is the fourth argument ([3] in Python) in the argument array. The pem_path is the path to the SSL client-side certificate. E.g. '/path_to/my_trustedcerts.pem'. The FLOW_STR points to the path and name of the flow in SAS Viya. E.g.: Public/Flow.flw.  # 2. Variables import sys print ("Number of arguments:", len(sys.argv), "arguments") #print ("Argument List:", str(sys.argv)) RG=str(sys.argv[1]) client_id=str(sys.argv[2]) client_secret=str(sys.argv[3]) pem_path=str(sys.argv[4]) FLOW_STR=str(sys.argv[5]) # Viya URLs VIYAURL='https://' + RG +'.sas.com' url = VIYAURL + "/studioDevelopment/code" authUri="/SASLogon/oauth/token" Client and Secret Method   The authentication method in this example uses a SAS Viya registered client, called gel_client3 and a client secret. A SAS Administrator registered the client with  "authorized_grant_types": ["client_credentials"] .   How to register such a client and the secret? See for more details:   Authentication to SAS Viya: a couple of approaches. How to Score a SAS Decision Published to MAS Using Shell Commands.   It simply is a more elegant choice to use a client and a client secret to get a SAS Viya access token.   Reasons:   You can use this grant type in the payload sent to the SAS Viya REST API. This is better than using your user and the password. The password grant type is not considered secure and should not be used in production environments. There are ways to rotate secrets automatically. If the secret is exposed, you can just refresh it. Because you are passing it as a parameter, you do not need to update the program. You can scope the client registration. The scope links the client to a group and its authorization rules. This acts as a ring-fence around what the client can be used for.   # 3. This is using a client id and a secret - no user and password in the payload print("The Auth is using This is using a client id and a secret. Grant type: client_credentials") # client_id must be registered using client_credentials grant type # base 64 encode the api client id and secret and pass it in the authorization header auth_str = f'{client_id}:{client_secret}' auth_bytes = auth_str.encode('ascii') auth_header = base64.b64encode(auth_bytes).decode('ascii') my_headers = {'Authorization': f'Basic {auth_header}'} payload = { 'grant_type': 'client_credentials', 'client_id': client_id, 'client_secret': client_secret } print(f'Logging in to {VIYAURL}') # post as form response = requests.post( f'{VIYAURL}/SASLogon/oauth/token', data=payload, headers=my_headers, verify=pem_path) print(response) token = response.json()['access_token'] #print (token)   In addition, we are using SSL certificate verification in the request: verify=pem_path resolves to  verify='/path_to/my_trustedcerts_.pem' . You should specify a local certificate to use as a client-side certificate. The client-side certificate can be obtained using a kubectl command. See Get the SAS Viya Certificates in How to Score a SAS Decision Published to MAS Using Shell Commands.   Generate the SAS Code from a SAS Studio Flow   Now that you have a SAS Viya access token, you can use the /studioDevelopment/code endpoint to extract the SAS code from the SAS Studio Flow.   # 4. Headers containing the token headers = { 'Authorization': 'bearer ' + token, 'Accept': 'application/json', 'Content-Type': 'application/json;charset=utf-8' } #print(headers) # 5. Payload to be passed with the REST API. The flow is passed as a variable inside. ## FLOW='/path_to/flow_name.flw'. E.g. #FLOW='/Public/Flow.flw' ## We add the '/' to the FLOW_STR inside the program. ### As of 2022.09 the Flow MUST HAVE A TABLE step inside. FLOW = '/' + FLOW_STR print(FLOW) payload_dict = { 'reference': { 'type': "content", 'path': FLOW, 'mediaType': "application/vnd.sas.dataflow" }, 'initCode': False, 'wrapperCode': False } payload = json.dumps(payload_dict) print(payload) # 6. POST REQUEST response = requests.request("POST", url, headers=headers, data=payload, verify=pem_path) print(response.text) print(response.json()) # 7. Write the output to a JSON file jsonString = json.dumps(response.json()) jsonFile = open("Flow.json", "w") jsonFile.write(jsonString) jsonFile.close()   The headers contain the SAS Viya token.   The SAS Studio Flow we want to obtain the code from is stored in /Public/Flow.flw.   A JSON payload is prepared from a dictionary variable payload_dict. The SAS Studio Flow path and name are passed as an argument from the FLOW variable. We then “dump” the dictionary variable in JSON.   { "reference": { "type": "content", "path": "/Public/Flow.flw", "mediaType": "application/vnd.sas.dataflow" }, "initCode": false, "wrapperCode": false }   The payload is passed, or posted, to the SAS Viya REST API. The response comes back in JSON format. We write the output to a JSON file.   Execute the Python Program   Suppose you will save the whole Python program as codeGen.py. On a Linux machine, you can call the above script from the command line using the python command, the Python program name, followed by the arguments.   Hardcoded secret   You could simply pass the secret in the command:   python3 codeGen_ssl_args.py 'viya_url_prefix' 'gel_client3' 'the_secret' '/path_to/my_trustedcerts.pem' 'Public/Flow.flw'   Vaulted Secret   If you are calling the script from an Azure Virtual Machine, you should store the secret in an Azure Key Vault. You can then retrieve the secret using the az keyvault secret show command.   secret=$(az keyvault secret show --name THE_SECRET --vault-name THE_KEY_VAULT --query "value" -o tsv)   Then, you can call the Python program using:   python3 codeGen_ssl_args.py 'viya_url_prefix' 'gel_client3' $secret '/path_to/my_trustedcerts.pem' 'Public/Flow.flw'   This approach has the advantage that the secret will not be directly written to an output or captured in a log file.   Clean the JSON Output   The output is written to the Flow.json file. If you display it, it will look a bit like The Matrix code with Latin letters. There are no sushi recipes encoded inside, just SAS code:   cat Flow.json | jq SAS Studio Flow Code Output in JSON format   Or, you will certainly want the SAS code stripped of separators like \n :   cat Flow.json | jq --raw-output '.code' | sed 's/\\n/\n/g' > Flow.sas cat Flow.sas Now, not only it looks better, but you can now run the SAS file in batch, store it to a Git repository and so on.    PyViyaTools   And since we are discussing Python and code generation, Gerry Nelson has just created a new pyviyatool named exportstudioflowcode.py. The nice thing about it is that it can loop through a folder and convert all the SAS Studio Flows from that folder.   # usage: exportstudioflowcode.py [-h] -t {Flow,Folder} -n NAME -d DIRECTORY [--includeinitcode] [--includewrappercode]   The pyviyatools are a set of command-line tools that call SAS Viya REST APIs from Python. These tools simplify common administration tasks, provide more complex functionality and are a great complement to the SAS Viya CLI. The tools are available on the public SAS Software GitHub. They are easy to install and use.    Conclusions   The post provided you a Python program you can use to interact with SAS Viya REST APIs.   First, we discussed the importance of avoiding to hardcode credentials in programs, the advantages of clients and secrets and how to pass the secrets as arguments in the Python program. Second, we obtained a SAS Viya token. Third, we discussed the SAS Viya codeGen REST API and how to generate the SAS code from a SAS Studio Flow. Fourth, you executed the Python program using shell and passed the arguments. Lastly, you cleansed the output and obtained the SAS program from the JSON file.   Acknowledgements   Thank you, Alexey Vodilin and Richard Frazer for several Python code snippets I used as inspiration.   Resources SAS.COM codeGeneration REST API. SAS Documentation SAS Viya 2022.09 Long Term Support. How to Score a SAS Decision Published to MAS Using Shell Commands. Authentication to SAS Viya: a couple of approaches. How To Get the Code From a SAS Studio Flow and Why It Matters.   Find more articles from SAS Global Enablement and Learning here. ... View more

SAS customers need to run a SAS Studio Flow in batch or from a CI/CD pipeline. They also need to version the flow code, possibly in a Git repository and compare the changes from one version to another. To achieve these goals, you need the SAS code behind the SAS Studio Flow. A REST API helps you generate the code. Read the post to find out more.   Why is This Important?   SAS Customers want to run SAS Studio Flows: In batch, from the command line. From a CI/CD pipeline, for example from Azure Pipelines or Jenkins. From open-source programming languages, such as Python.   They also want to: Version the flows in Git repositories. Compare the code changes from one version to another.   The Key   To run a SAS Studio Flow in batch or in a pipeline, you need the SAS code behind the flow.   To compare two versions of the same SAS Studio Flow, you need the SAS code behind the flow.   The Answer: a REST API   To generate the SAS code, use the /studioDevelopment/code REST API. The API is also known as the "CodeGen" or the “Code Generation” API, because it “generates” the SAS code from a SAS Studio Flow file.   The studioDevelopment REST API has been available since the SAS Viya 2022.09 Long Term Support.   Practical Example   Watch the following video for an overview of the approach:     SAS Studio Flow   Suppose you developed a flow that performs a de-duplication of an existing table.      To generate the SAS code form this flow you could write a SAS program:   SAS Code Generation Program   You need to initialize several files:   input – will be posted to the REST API endpoint. Perhaps, the most important in this JSON file is the path to your SAS Studio Flow. resp – has the REST API response. flowcode – holds the generated SAS code. filename resp clear; filename input clear; filename flowcode clear; filename input temp; data _null_; file input recfm=f lrecl=1; put "{"; put """reference"": {"; put """type"": ""content"","; put """path"": ""/Users/sasuser/My Folder/Flow.flw"","; put """mediaType"": ""application/vnd.sas.dataflow"""; put "},"; put """initCode"": true,"; put """wrapperCode"": false"; put "}"; run;   You then POST to the studioDevelopment/code REST API the input JSON file. For Authentication, you can reuse the SAS_SERVICES token, if you are writing the program inside SAS Studio.   %let myviyaurl = %sysfunc(getoption(servicesbaseurl)); %put NOTE: &=myviyaurl; filename resp temp; proc http url = "&myviyaurl/studioDevelopment/code" in=input out= resp method="POST" OAUTH_BEARER=SAS_SERVICES VERBOSE; headers "Content-Type" = "application/json;charset=utf-8" "Accept" = "application/json" ; run; libname resp clear; libname resp json; /* OUTPUT: resp json libname resp json: ALLDATA with columns: P, P1, V, Value ROOT with columns: ordinal_root, code Both root.code and alldata.value = the SAS Program generated from the source FLOW */ Run Result   When you run the program and it is successful, you should see a 200 OK. It means, the REST API generated the code from the SAS Studio Flow.     You can then define a JSON library on top of the resp JSON file. From Libraries, open RESP.ROOT and notice the SAS Studio Flow code is stored in the code field.     SAS Code to a File   You can write this program to extract the code to a file called flowcode.   * Save the code in a file you want to execute:; * The flowcode must not have references to paths unknown to the Batch service; filename flowcode temp; data _null_; set resp.root; file flowcode; put code; run; quit;   Save and Version the SAS Code   Alternatively, you can save the code to a SAS file, on the file system.   filename flowcode '/azuredm/gitrepo/programs/Flow.sas'; data _null_; set resp.root; file flowcode; put code; run; quit;   You can also save it directly to a Git repository, provided you added one in SAS Studio.       Execute the SAS Code   Depending on your use case, you can execute the code, using an %include statement, effectively executing the flow code.   * Skip errors if any from the flow code; options nosyntaxcheck; * Execute the flow file; %include flowcode; quit; I also noticed that the generated code sometimes throws small errors when executed, hence the nosyntaxcheck option will help.   This is not the only way to run the SAS code. According to  my colleague Alexey Vodilin, you can [use the] Compute API or Job Execution Service API depending on [your] use case. All these APIs are publicly available.   PyViyaTools And since we are discussing Python and code generation, Gerry Nelson has just created a new pyviyatool named exportstudioflowcode.py. The nice thing about it is that it can loop through a folder and convert all the SAS Studio Flows from that folder. # usage: exportstudioflowcode.py [-h] -t {Flow,Folder} -n NAME -d DIRECTORY [--includeinitcode] [--includewrappercode] The pyviyatools are a set of command-line tools that call SAS Viya REST APIs from Python. These tools simplify common administration tasks, provide more complex functionality and are a great complement to the SAS Viya CLI. The tools are available on the public SAS Software GitHub.   Conclusion The /studioDevelopment/code REST API, also known as "CodeGen" can help you generate SAS code from a SAS Studio Flow. With the code, you can further run a SAS Studio Flow in batch or from a CI/CD pipeline. Or, you can version the flow code using Git repositories and compare the changes from one version to another. In a next post, we will look How to Get the Code From a SAS Studio Flow Using Python. Acknowledgements Thank you, Alexey Vodilin, Mark Escauriaga, Patric Hamilton, Cecily Hoffritz and Bruno Mueller for helping me figure out how this SAS REST API works and suggesting code improvements.  Resources How to Get the Code From a SAS Studio Flow Using Python. pyviyatools. SAS.COM codeGeneration REST API. SAS Documentation SAS Viya 2022.09 Long Term Support. Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about SAS codeGen REST API. If you wish to get more information, please write me an email. ... View more

With Python popularity still soaring, and with Python allowed in a SAS decision, you no longer have to choose between SAS or Python. You can use both in SAS Intelligent Decisioning. Your decision can contain only Python code or, can be combined with models, rule sets, SAS Data Step 2 (DS2) code, branching logic and much more. While the Python integration is not new and has been there since SAS Viya 3.5, we are looking at an example with SAS Viya, the Long-Term Support Release 2022.09 (November 2022). Read the post to find out how to use Python code in SAS Intelligent Decisioning.   Take the following scenario: you might want to add to a decision, a simple if-then-else logic, written in Python.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Prerequisite The main prerequisite is to Configure Python integration with SAS Viya. Read Scott McCauley’s detailed post for more information.   Steps to Use Python in a SAS Decision Write the Python code. Test the Python code: On a machine with Python With SAS Studio: Write a Python program directly, or Write a SAS program with PROC PYTHON. Write the Python code file in SAS Intelligent Decisioning. Test the Python code file. Add the Python code file to a decision. Test the decision.   Publishing Notes   You can publish decisions that contain Python code files to SAS Micro Analytic Service destinations, SAS Cloud Analytic Services (CAS) destinations, Git and container destinations, such as Microsoft Azure, Amazon Web Services, and private Docker repositories. The publishing capabilities can help you move faster the decisions you developed, in a production environment.   Write the Python Code   The code uses two input variables Bid and state to create an output variable bidCommand.   if Bid == 0: bidCommand = 'Do NOT bid on this car!!!' else: bidCommand = 'Bid on this car!!!' if state == 'CA': bidCommand = 'Buy anything from California!'   It is deliberately a very basic Python program, to focus more on the steps, rather on the code itself.   The code is written with Python 3.9. The same version was used for the Python integration with SAS Viya.   Write the Python Code for PyMAS   When you publish a decision that contains a Python code file, the Python code is injected into a PyMAS package. The PyMAS package is accepted by PROC DS2 in SAS servers, SAS Cloud Analytic Services (CAS), and the SAS Micro Analytic Service REST service.   When you are developing your Python code, for SAS Intelligent Decisioning, follow these rules:   The Python logic must be part of an execute function. The function can have input variables, such as Bid and state. The output variable bidCommand must be specifically declared after "Output: “ , for example, "Output: bidCommand" .   The Python code subsequently becomes:   # CalifoniaOverride PyMAS function def execute(Bid, state): "Output: bidCommand" if Bid == 0: bidCommand = 'Do NOT bid on this car!!!' else: bidCommand = 'Bid on this car!!!' if state == 'CA': bidCommand = 'Buy anything from California!' return bidCommand   Test the Python Code   Watch the following video where the Python code is tested:   First, on a Linux machine with Python installed. Second, in SAS Studio in a Python program. Third, in a SAS program with PROC PYTHON.   On a Linux Machine with Python   You can write a bash script to add the Python code to a .py file, and further, add three test cases after the execute function, to validate the logic.   cd ~ # Create CalifoniaOverride PyMAS function tee ~/pymas.py > /dev/null <<EOF #!/usr/bin/env python # coding: utf-8 # CalifoniaOverride PyMAS function def execute(Bid, state): "Output: bidCommand" if Bid == 0: bidCommand = 'Do NOT bid on this car!!!' else: bidCommand = 'Bid on this car!!!' if state == 'CA': bidCommand = 'Buy anything from California!' return bidCommand # test the PyMAS function (python 3) Bid=0 state='PA' bidCommand=execute(Bid,state) print(f"With bid {Bid} and state {state} the bidCommand is: {bidCommand}") Bid=1 state='PA' bidCommand=execute(Bid,state) print(f"With bid {Bid} and state {state} the bidCommand is: {bidCommand}") Bid=0 state='CA' bidCommand=execute(Bid,state) print(f"With bid {Bid} and state {state} the bidCommand is: {bidCommand}") EOF # Test Python python3 pymas.py   The output will show:   With bid 0 and state PA the bidCommand is: Do NOT bid on this car!!! With bid 1 and state PA the bidCommand is: Bid on this car!!! With bid 0 and state CA the bidCommand is: Buy anything from California!   With SAS Studio   Write a Python Program   With the October 2021 release of SAS Viya, SAS introduced the Python Code Editor. Data Scientists and Python programmers can now code, execute and schedule Python scripts from SAS Studio.   Copy the above Python code from # CalifoniaOverride PyMAS function until the line before EOF .   Write a SAS Program with PROC PYTHON   With the December 2021 release of SAS Viya, a user can wrap Python code in a SAS program using PROC PYTHON.   proc python; submit; # CalifoniaOverride PyMAS function here… endsubmit; run; Write the Python Code File in SAS Intelligent Decisioning   Watch the following video where the Python code is written to a SAS Intelligent Decisioning code file.     In SAS Intelligent Decisioning, create a new Python code file stateBid . Open the code file editor and paste the Python code. Note we removed the test cases form the code. You can test directly in SAS Intelligent Decisioning.   # CalifoniaOverride PyMAS function def execute(Bid, state): "Output: bidCommand" if Bid == 0: bidCommand = 'Do NOT bid on this car!!!' else: bidCommand = 'Bid on this car!!!' if state == 'CA': bidCommand = 'Buy anything from California!' return bidCommand   Because you wrote the code as a PyMAS function, when you save the code file, the variables will be automatically get created from the code. That really saves you time when you have too many variables.   Test the Python Code file   When testing, you can choose: Tests, where you can use a CAS table for the input data. Scenarios, where you can add manually the inputs and as an option, the expected output:   The result can help you validate the Python logic directly in SAS Intelligent Decisioning.     The submitted code will show the Python code wrapped in DS2 code, in a PyMAS module.     Add the Python Code File to a Decision   The decision in this example is called autoAuctionDecPy.   Variable Mapping   If your Python code input variables have the same name as the decision variables, they will be automatically mapped. I strongly advise you to create your Python code with your variable names in mind.   Even if the input variables have different names in your Python code, you can easily map them using the decision interface.   Watch the following video where a decision with a Python code file is tested in SAS Intelligent Decisioning.      Conclusions   With SAS Viya configured for Python integration, it is easy to create a Python code file and use it in a SAS decision. The Python code has to follow the PyMAS guidelines,. However, this is a small price to pay for automatic variable mapping, multiple SAS server runtime compatibility and the ability to publish to different destinations.   There are many ways to test your Python code file, directly in SAS Intelligent Decisioning, line by line, in batch. You can also test directly in SAS Studio. Or simply, on another machine with Python.   Given Python’s popularity, I hope this will encourage you to use more SAS Intelligent Decisioning and use more Python in your SAS decisions.     Additional Resources Configure Python integration with SAS Viya by Scott McCauley Exploring the use of Python Code with SAS Intelligent Decisioning by Gerrit Van Wyngaard. Very good use case calling REST API from Python. Using PROC PYTHON to augment your SAS programs by Nicolas Robert. SAS® Intelligent Decisioning | LTS 2022 | Python Code Files documentation. Python Popularity Still Soaring, Info World, August 2022.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about Python in SAS Intelligent Decisioning. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

Since the SAS Viya Stable release 2022.09, you can use the SAS Viya Command Line Interface (CLI) to create a Git publishing destinations. A Git publishing destination allows you to publish a SAS model, decision or rule set to a Git repository. The SAS Viya CLI is faster and less complex than using Python to call SAS Viya REST APIs. Because of these factors, in a few minutes, you will be ready to publish models from SAS Model Manager, or decisions and rule sets from SAS Intelligent Decisioning to any Git repository on GitHub, Bitbucket, Azure Repos, GitLab, etc. Read the post to learn how to use the SAS Viya CLI to create Git publishing destinations.   Why the Git Publishing Destinations?   Models and decisions are stored in a Git repository, which sits outside SAS Viya. Specifically for SAS Viya cloud deployments, it helps SAS customers decouple development and promotion, share with a wider audience the SAS models or decisions code, while reducing cloud costs.   From the Git repository, you can deploy the published models or decisions to SAS Cloud Analytics Service (CAS) or SAS Micro Analytic Service (MAS). There, you can score them in batch or using a REST API. These two destinations can be in the same SAS Viya environment, or in a different environment, for example test, staging, production, etc.   How to Create Git Publishing Destinations?   After Michael Goddard mentioned in his post Creating model publishing destinations using the SAS Viya CLI : “Using the SAS Viya CLI is a much better approach and hides the complexity of working with the REST API”, I decided to give the SAS Viya CLI a go for a Git destination.   The video summarizes the approach.      It took less than five minutes to define a Git publishing destination, which is great.   Prerequisites First of all, there are some prerequisites for your software and environment.   SAS Viya Use SAS Viya version 2022.09 or later.  Configure a SAS Viya Persistent Volume Claim (PVC) called git. Read more in SAS Model Manager: Git Publishing Destinations section. Git Repository I am using GitHub as the Git repository flavor. If you are using GitHub, you must: Have  GitHub account. Obtain a GitHub Personal Access Token (PAT) to interact with a Git repository from SAS. Points two and four are detailed in How to Publish SAS Decisions to Git: Required Configuration.   Client Machine On a client machine, for example a Linux Virtual Machine: Install the latest SAS Viya CLI version. See the SAS Downloads page. I am using the 1.21.0 version in the following example. Create a SAS Viya CLI profile. Create the Git Publishing Destination You can use the sample code provided to define the publishing destination. Adapt the variables to your context: # Gather SAS Viya Variables export GELENV_NS=gelenv # your SAS Viya Namespace NS=$GELENV_NS echo $RG INGRESS_FQDN=${RG}.gelenable.sas.com # this is your SAS Viya URL without the https:// DOMAIN_NAME=GitHubDomainCLIGamma # credential domain name that will be created DOMAIN_DESC=GitHubDomainCLIGamma # credential domain description DEST_NAME=GitHub # publishing destination name DEST_DESC=GitHub # publishing destination description CODE_GEN=MAS # From Git you can later deploy to a MAS or CAS publishing destination. Code generation can be MAS or CAS. # Gather GIT Variables - from your Git repository GITHUB_REPO_URL= fill_in_here # e.g., GITHUB_REPO_URL=https://github.com/bteleuca/sas-viya-az-devops.git GITHUB_USER= fill_in_here # GITHUB_USER Your GitHub user, typically after https://github.com/ e.g., GITHUB_USER=bteleuca GITHUB_EMAIL= fill_in_here # GITHUB_EMAIL=Your_GitHub_Account_email GITHUB_BRANCH= fill_in_here # e.g. main # Secret GITHUB_PAT=fill_in_here # e.g., GITHUB_PAT=ghp_z*jc2y*Z*8bG91Y*P9bSJy8MZ8j*Ne3h*uHu # Check the variables printf "\n We need the following variables for the CLI \n " echo "SAS Viya URL: https://$INGRESS_FQDN" echo "SAS User Name: ${SASUSER}" # the SAS identity user or group creating the publishing destination echo "SAS User Password: ${AZUPW}" # the credential for the SAS identity echo "SAS Domain Name: ${DOMAIN_NAME}" echo "SAS Domain Description: $DOMAIN_DESC" echo "Git User Name: $GITHUB_USER" echo "SAS Publishing Destination Name: $DEST_NAME" echo "SAS Publishing Destination Description: $DEST_DESC" echo "Git User Email: $GITHUB_EMAIL" echo "Git Repository URL: $GITHUB_REPO_URL" echo "Git Repository Branch: $GITHUB_BRANCH" echo "Code Generation Mode MAS or CAS: $CODE_GEN" #echo "Git Personal Access Token: $GITHUB_PAT" echo "Git Personal Access Token: well... it's secret..." # Login with the sas-viya cli. Profile {SAS_CLI_PROFILE} has already been created. cd ~ export SAS_CLI_PROFILE=${GELENV_NS:-Default} export SSL_CERT_FILE=~/.certs/${GELENV_NS}_trustedcerts.pem sas-viya -k auth login --user sassmth --password ********** # mask # List the variables used by the SAS VIYA CLI sas-viya -k --profile ${SAS_CLI_PROFILE} models destination createGit --help # create the Git publishing destination ## all code on one line ; descriptions must be double quoted sas-viya -k --profile ${SAS_CLI_PROFILE} models destination createGit --name ${DEST_NAME} --description "${DEST_DESC}" --remoteRepoURL ${GITHUB_REPO_URL} --userEmail ${GITHUB_EMAIL} --gitUserId ${GITHUB_USER} --gitBranch ${GITHUB_BRANCH} --gitAccessToken ${GITHUB_PAT} --codeGenerationMode ${CODE_GEN} --userName ${GITHUB_USER} --identityType user --identityId ${SASUSER} --credDomainID ${DOMAIN_NAME} --credDescription "${DOMAIN_DESC}" --localRepoLocation " /mmprojectpublic" After you run the code, you can check in SAS Environment Manager the Git destination properties.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.       Publish a Model to Git   If you would publish from SAS Model Manager, the decision tree model QS_Tree1 to the recently created Git publishing destination, you would see:         The Git repository used as a publishing destination will contain several folders and files. The folders are a combination of project name, model version and model name.             Publish a Decision to Git   If you would publish from SAS Intelligent Decisioning, the SAS decision autoAuctionDec to the recently created Git publishing destination, you would see:         The decision is made of a SAS model, several rule sets and code files. The Git repository used as a publishing destination will contain several folders and files.   If you are curious what is in the published files, see How to Publish SAS Decisions to Git: Published Files.   Conclusions   Since the SAS Viya Stable release 2022.09, you can use the SAS Viya CLI to create the Git publishing destination. The CLI simplifies greatly the task for the end user. Because it is now less complex, we should more people publishing models and decisions (or rule sets) from SAS Viya to a Git repository.   From the Git repository, you can deploy the published models or decisions to CAS or MAS, where you can score them. These two destinations can be in the same SAS Viya environment, or in a different one (test, staging, production).   Models and decisions are stored in a Git repository, which sits outside SAS Viya. Specifically for SAS Viya cloud deployments, it helps SAS customers decouple development and promotion, share with a wider audience the SAS models or decisions code and reduce cloud costs.   Additional Resources Creating model publishing destinations using the SAS Viya CLI How to Integrate SAS Intelligent Decisioning with Git How to Publish SAS Decisions to Git: Required Configuration How to Publish SAS Decisions to Git: Publishing Destination How to Publish SAS Decisions to Git: Published Files   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

Read the post to understand how you can use Azure Pipelines to install packages, tools and then pass commands from the SAS Viya Command Line Interface (CLI) or pyviyatools. The goal is to interact with SAS Viya hosted in Microsoft Azure from an Azure hosted virtual machine. You can use for this purpose the self-hosted agent you created and registered.   Overall Picture   The steps to create your own self-hosted agent are: Create a virtual machine on Azure. Read How to Create a SAS Viya Azure Pipelines Agent. Configure communication with the SAS Viya Azure Kubernetes Service cluster. Read How to Make Your Virtual Machine Talk with SAS Viya on Azure. Create an Azure DevOps self-hosted agent. Install software and packages on the self-hosted agent: SAS Viya Command Line Interface (CLI), Python, pyviyatools, utilities, etc. (this post).   Azure Pipeline Code   The Azure Pipeline code provided further will focus on four things: Install and update needed packages on the agent. Download and install the SAS Viya CLI on the agent. Clone and install the pyviyatools. The pyviyatools are a set of command-line tools that call the SAS Viya REST APIs from Python. These tools simplify SAS Viya administration tasks. The source code is hosted in GitHub. Test the SAS Viya CLIs and pyviyatools.   The Azure Pipeline code is written in yaml.   Pipeline Header Parameters   The parameters section contains: MYUSER : a string used to prefix the SAS Viya URL and Azure resources. sasviyaclitgz : a compressed archive containing the sas-viya (CLI) executable. This points to the file name sas-viya-cli-1.21.0-linux-amd64.tgz. You can download it from the SAS Downloads Page and upload this file in the Git repository accompanying your Azure pipeline. Your Git repository can be hosted in Azure Repos, GitHub, Bitbucket, Gitlab, just to name a few options. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Variable   The variable defines the self-hosted agent pool name created in How to Build an Azure Self-Hosted Agent for SAS Viya.   You can use parameters in variables. For example, ${{ parameters.MYUSER }}-sas-viya-jump-vm  becomes sbxbot-sas-viya-jump-vm when MYUSER=sbxbot .   Azure Pipelines does not allow you to use agent pool names fully stored in a parameter.   # this pipeline configures the SAS Viya self-hosted agent parameters: - name: MYUSER displayName: SAS User ID type: string default: - name: sasviyaclitgz displayName: SAS Viya CLI TGZ Name type: string default: sas-viya-cli-1.21.0-linux-amd64.tgz # Nov 2022 variables: - name: pool value: ${{ parameters.MYUSER }}-sas-viya-jump-vm trigger: - main pool: name: $(pool)   The trigger tells Azure Pipelines to start a build, a new pipeline run, when somebody pushes a change to the main branch of the underlying Git repository.   Pool   The pool tells the pipeline to build on (use) a self-hosted agent from your agent pool.   Pipeline Stage   You can structure an Azure pipeline in a hierarchy of stages, jobs, scripts or tasks. The hierarchy helps you keep the pipeline code clear and allows you to build-in dependencies between jobs or stages.   Install Job   There is only one stage called install . The installPackages job clones the Git repository on the agent in the checkout step. You will notice the checkout step repeated in every job. Further, the next four bash scripts will run on the agent, which is a Linux Ubuntu VM. The scripts will:   Update packages. Install Python. Install jq (a command-line JSON processor). List the versions of the components installed. stages: - stage: install displayName: Install jobs: - job: installPackages displayName: Install and Update Needed Packages steps: - checkout: self persistCredentials: true - script: | echo 'List directory where Git Repo was cloned: ' && pwd ls -la echo Get the folder where the Git repo was cloned export GITFOLDER=$(pwd) echo Update Advanced Package Tool, or APT sudo apt update sudo apt -y upgrade displayName: 'Install and Update APT' - script: | echo Install Python 3.x with pip3 sudo apt install python3-pip -y displayName: 'Install Python 3.x' - script: | echo Install a Linux Json parser named jq sudo apt install jq -y displayName: 'Install JQ' - script: | echo Check python python3 --version echo Check pip3 is installed pip3 --version echo Check jq jq --version displayName: 'Check install'     Install SAS Viya CLI Job   The next job has also four scripts. Their purpose is to: Un-compress the archive containing the sas-viya CLI executable. Move the executable in the $clidir folder. Install the SAS Viya CLI in that folder. Test the installation, by listing the plug-ins. - job: sasviyacli dependsOn: installPackages displayName: Install the SAS Viya CLI steps: - checkout: self persistCredentials: true - script: | echo 'List directory where Git Repo was cloned: ' && pwd ls -la echo Get the folder where the Git repo was cloned export GITFOLDER=$(pwd) echo Unpack the tgz tar zxvf ${{ parameters.sasviyaclitgz }} # confirm you have the sas-viya executable in your folder ls -la displayName: 'Unpack the SAS Viya CLI' - script: | echo 'List directory where Git Repo was cloned: ' && pwd ls -la echo Get the folder where the Git repo was cloned export GITFOLDER=$(pwd) clidir=/opt/sas/viya/home/bin echo Create $clidir folder sudo mkdir -p $clidir echo Move the sas-viya program to its default folder sudo mv sas-viya $clidir echo Give exec permissions to sas-viya in $clidir folder sudo chmod +x $clidir/sas-viya echo List $clidir folder ls -la $clidir displayName: 'Move the SAS Viya CLI' - script: | echo Install the SAS-VIYA CLI plugins clidir=/opt/sas/viya/home/bin echo Set viya_cli_repository viya_cli_repository=https://support.sas.com/repos/viyacli/ echo $viya_cli_repository echo INSTALLING ALL plugins for p in $($clidir/sas-viya plugins list-repo-plugins --repo SAS | awk 'NR > 3 { print $1 }' ) do $clidir/sas-viya plugins install --repo SAS ${p#"*"} done echo Done installing SAS_VIYA CLI echo List plugins $clidir/sas-viya plugins list displayName: 'Install the SAS Viya CLI' - script: | clidir=/opt/sas/viya/home/bin ls -la $clidir echo Test the sas-viya cli $clidir/sas-viya --help echo List sas-viya cli plugins $clidir/sas-viya plugins list displayName: 'Test the SAS Viya CLI'       Create a SAS Viya CLI Profile and Install pyviyatools   In addition to scripts, in a job you can have tasks, such as the AzureKeyVault task.   Always use an Azure Key Vault instead of hardcoding passwords or secrets in your pipeline code! Remember that the pipeline code is stored in a Git repository. Even if the repository is private, it may still be consulted by many other users. If the Git repository is made public (by mistake, or forked and made public), well, your passwords will be visible, on the web for anyone to consult. Hence, the need for the key vault.   You should use it to retrieve secrets, such as the $(SASPASS) password from an Azure Key Vault of your choice.   After the task, the script clones the public GitHub repository that contains the pyviyatools.   Next, it creates a SAS_CLI_PROFILE to configure the SAS Viya CLI for a particular SAS Viya deployment. Notice how the SSL_CERT_FILE references the .pem file copied on the self-hosted agent in How to Make Your Virtual Machine Talk with SAS Viya on Azure. Without a valid certificate, any communication attempt will be blocked by SAS Viya. You also have to create certain Network Security Group rules to allow communication with your Azure Kubernetes Service cluster over the port 443 (HTTPS).   Even further, the script logs the user in the SAS Viya CLI, using the $(SASPASS) password and obtains a SAS Viya access token. Finally, it runs the setup for the pyviyatools. The setup uses the SAS Viya CLI profile created.   - job: pyviyatools dependsOn: installPackages displayName: Install the pyViyaTools steps: - checkout: self persistCredentials: true - task: AzureKeyVault@2 inputs: azureSubscription: 'GELDM(41baf198-cc93-4165-9882-6804e5fb95c0)' KeyVaultName: '${{ parameters.MYUSER }}-key-vault' SecretsFilter: '*' RunAsPreJob: true - script: | echo Clone the repo cd ~ git clone https://github.com/sassoftware/pyviyatools.git echo go to the folder ls ~/pyviyatools ls echo Set Variables echo Initialize needed variables export SAS_CLI_PROFILE=gelazure echo $SAS_CLI_PROFILE export current_namespace=gelazure export CURL_CA_BUNDLE=~/.certs/${current_namespace}_trustedcerts.pem echo $CURL_CA_BUNDLE export SSL_CERT_FILE=~/.certs/${current_namespace}_trustedcerts.pem echo ${SSL_CERT_FILE} export REQUESTS_CA_BUNDLE=${SSL_CERT_FILE} echo ${REQUESTS_CA_BUNDLE} echo Running with user: ${{ parameters.MYUSER }} export INGRESS_URL=https://${{ parameters.MYUSER }}-gel.eastus.cloudapp.azure.com echo Viya URL: ${INGRESS_URL} clidir=/opt/sas/viya/home/bin echo SAS-Viya CLI directory is: $clidir echo Create SAS-Viya CLI Profile $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile set-endpoint ${INGRESS_URL} $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile toggle-color on $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile set-output fulljson echo Show profile $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile show echo Login in $clidir/sas-viya --profile ${SAS_CLI_PROFILE} auth login -user sasadm -password $(SASPASS) echo Show profile after login $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile show echo Install pyviyatools python3 ~/pyviyatools/setup.py displayName: 'Install and setup the pyViyaTools'         Test the pyviyatools   A pipeline job running on the self-hosted agent functions like a self-contained code unit. That is why every job recreates the SAS_CLI_PROFILE . The pipeline will also work if you create it once, then you just reuse the profile in other jobs. Personally, I like to keep each job independent. It allows me to copy them in another pipeline or a new job.   The command  python3 ~/pyviyatools/loginviauthinfo.py -f ~/.authinfo obtains a SAS Viya access token using pyviyatools.   - job: testPyviyatools dependsOn: pyviyatools displayName: Test the pyViyaTools steps: - checkout: self persistCredentials: true - script: | echo Set Variables echo Create SAS-Viya CLI Profile echo Initialize needed variables export SAS_CLI_PROFILE=gelazure echo $SAS_CLI_PROFILE export current_namespace=gelazure export CURL_CA_BUNDLE=~/.certs/${current_namespace}_trustedcerts.pem echo $CURL_CA_BUNDLE export SSL_CERT_FILE=~/.certs/${current_namespace}_trustedcerts.pem echo ${SSL_CERT_FILE} export REQUESTS_CA_BUNDLE=${SSL_CERT_FILE} echo ${REQUESTS_CA_BUNDLE} echo Running with user: ${{ parameters.MYUSER }} export INGRESS_URL=https://${{ parameters.MYUSER }}-gel.eastus.cloudapp.azure.com echo Viya URL: ${INGRESS_URL} clidir=/opt/sas/viya/home/bin echo SAS-Viya CLI directory is: $clidir echo Get a SAS Viya Token with pyviyatools python3 ~/pyviyatools/loginviauthinfo.py -f ~/.authinfo echo Show setup python3 ~/pyviyatools/showsetup.py echo Listing CASLIBS with pyviyatools python3 ~/pyviyatools/listcaslibs.py displayName: 'Test Login and Setup with pyViyaTools' The login requires an .authinfo file, created on the self-hosted agent, before running the pipeline. cd ~ tee ~/.authinfo > /dev/null << EOF default user sasadm password ***your_password_here*** EOF sudo chmod 600 ~/.authinfo ls -la ~/.authinfo   You will see the global CASLIBS listed, as a proof that your self-hosted agent can now interact with SAS Viya.         Conclusion   You can pass SAS Viya Command Line Interface (CLI) commands from Azure Pipelines or make SAS Viya REST APIs calls from Python, via pyviyatools. The pipelines are using a self-hosted Linux Ubuntu agent from Microsoft Azure.   Additional Resources Create a virtual machine on Azure. Read How to Create a SAS Viya Azure Pipelines Agent. Configure communication with the SAS Viya Azure Kubernetes Service cluster. Read How to Make Your Virtual Machine Talk with SAS Viya on Azure. Create an Azure DevOps self-hosted agent. See Gerry’s posts: Introducing the GEL pyviyatools. The pyviyatools installation instructions are found here: install.md. A SAS Viya Model Release Pipeline with Azure DevOps A SAS Viya with Data Pipeline with Azure DevOps. A SAS Viya Decision Pipeline with Azure DevOps. SAS Viya CI/CD Pipelines with Azure DevOps.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

Read this post to understand how you can build an Azure DevOps self-hosted agent for SAS Viya using scripts. An agent is a virtual machine. This machine can be hosted in a cloud, in Azure, for example. It can also be hosted on-premises. Overall Picture   The steps to create your own self-hosted agent are:   Create a virtual machine on Azure. Read How to Create a SAS Viya Azure Pipelines Agent. Configure communication with the SAS Viya Azure Kubernetes Service cluster. Read How to Make Your Virtual Machine Talk with SAS Viya on Azure. Create an Azure DevOps self-hosted agent (this post). Install software and packages on the self-hosted agent: SAS Viya Command Line Interface (CLI), Python, pyviyatools, utilities, etc. (next post) Pre-Requisites Access to an Azure DevOps organization. Azure DevOps Organization policies allowing personal access tokens (PAT) use. Permissions to create an agent pool in the Azure DevOps organization or in a project. Azure DevOps Organization You need access to an Azure DevOps organization. Azure DevOps Organization Policies Your Azure DevOps organization policies must allow the usage of personal access tokens (PAT). The Azure DevOps user must have sufficient rights to create an agent pool and a self-hosted agent. Managing an Azure DevOps organization may be discussed in a future post. Create an Azure DevOps Self Hosted Agent The steps are: Create an Azure DevOps Personal Access Token (PAT). Create an agent pool. Add an agent to that pool. Create a Personal Access Token In your Azure DevOps organization, from User settings > Personal access tokens (PAT), create a new PAT.   To “recruit” the machine as a self-hosted agent, the machine has to initiate the communication. To communicate securely with your Azure DevOps organization a personal access token is required. The token is only needed for the agent configuration.     Select the scopes: Custom defined: Agent Pools > Read & manage. Build > Read & execute. Create an Agent Pool   Agent pools can regroup several agents. For example, you can add all the development machines in a pool, the test machines in a second pool and so on. You can also have a pool for cloud machines, another for on-premises machines.   Add an Agent in a Pool In an existing agent pool, add the new machine (New agent). Choose x64 for Linux or CentOS.   Jump on the Virtual Machine The configuration has to be initiated from the VM. SSH to that machine: cd ~ # adapt the resource group name, the VM name and the SSH key to your environment export MYUSER=$USER && echo $MYUSER export JUMPBOXKEY=~/.ssh/gelazuredm-aks-key JUMPBOXIP=$(az vm list-ip-addresses -g $MYUSER-SCR -n $MYUSER-jump-vm --query "[].virtualMachine.network.publicIpAddresses[0].ipAddress" -o tsv) echo "jump box Public IP: $JUMPBOXIP" # SSH connect to your jump VM cd ~/.ssh ssh -i $JUMPBOXKEY jumpuser@$JUMPBOXIP   Set Azure DevOps Agent Variables Scripts are from now on executed on the jump box. To configure the agent, you will need your PAT: export AZP_TOKEN=paste_here_the_PAT_you_created_in_Az_DevOps # MYUSER is optional. Only used here as a prefix for other variables. export MYUSER=your_User_ID # your-6-digit-SAS-userid   Set the following environment variables, adapt them to your environment: export AZP_POOL=$MYUSER-sas-viya-jump-vm # name of your agent pool export AZP_AGENT_NAME=$MYUSER-azuredm-jump-vm # your agent's name in Azure DevOps agent pool export AZP_URL=https://dev.azure.com/geldmdevops # URL to your Azure DevOps organization. export AZP_WORK=_work # working folder for the agent on this VM export SUDO_USER=jumpuser # user to run the agent configuration   Install and Update Packages You might need to do that before you set your agent version. sudo apt update sudo apt -y upgrade sudo apt install jq -y   Set the Agent Version Set the AZP_AGENT_VERSION environment variable to specify the latest version of the agent. export AZP_AGENT_VERSION=$(curl -s https://api.github.com/repos/microsoft/azure-pipelines-agent/releases | jq -r '.[0].tag_name' | cut -d "v" -f 2) echo $AZP_AGENT_VERSION A YAML pipeline on a Linux machine must be using the latest version of the agent, even if it's pre-release. The agent software is constantly updating, so you can curl the version information from a Microsoft GitHub repo. The command uses jq to read the latest version from the JSON string that's returned.   Build the Agent Script Create a script to install the Azure DevOps agent on this VM: cd ~ tee ~/build-agent.sh > /dev/null << EOF #!/bin/bash set -e # Select a default agent version if one is not specified if [ -z "$AZP_AGENT_VERSION" ]; then AZP_AGENT_VERSION=2.200.2 fi # Verify Azure Pipelines token is set if [ -z "$AZP_TOKEN" ]; then echo 1>&2 "error: missing AZP_TOKEN environment variable" exit 1 fi # Verify Azure DevOps URL is set if [ -z "$AZP_URL" ]; then echo 1>&2 "error: missing AZP_URL environment variable" exit 1 fi # If a working directory was specified, create that directory if [ -n "$AZP_WORK" ]; then mkdir -p "$AZP_WORK" fi # Create the Downloads directory under the user's home directory if [ -n "$HOME/Downloads" ]; then mkdir -p "$HOME/Downloads" fi # Download the agent package curl https://vstsagentpackage.azureedge.net/agent/$AZP_AGENT_VERSION/vsts-agent-linux-x64-$AZP_AGENT_VERSION.tar.gz > $HOME/Downloads/vsts-agent-linux-x64-$AZP_AGENT_VERSION.tar.gz # Create a working directory to extract the agent package to mkdir -p $HOME/myagent # Move to the working directory cd $HOME/myagent # Extract the agent package to the working directory tar zxvf $HOME/Downloads/vsts-agent-linux-x64-$AZP_AGENT_VERSION.tar.gz # Install the agent software ./bin/installdependencies.sh # Configure the agent as the sudo (non-root) user sudo chown $SUDO_USER $HOME/myagent sudo -u $SUDO_USER ./config.sh --unattended \ --agent "${AZP_AGENT_NAME:-$(hostname)}" \ --url "$AZP_URL" \ --auth PAT \ --token "$AZP_TOKEN" \ --pool "${AZP_POOL:-Default}" \ --work "${AZP_WORK:-_work}" \ --replace \ --acceptTeeEula # Install and start the agent service sudo $HOME/myagent/svc.sh install $SUDO_USER sudo $HOME/myagent/svc.sh start sudo $HOME/myagent/svc.sh status EOF ls -la   Make the script executable, and then run it. chmod u+x build-agent.sh sudo -E ./build-agent.sh Sudo enables the script to run as the root user. The -E argument preserves the current environment variables, including the ones you set, so that they're available to the script.   As the script runs, you can see the VM being transformed in an agent. Test the Self-Hosted Agent with a Pipeline   In your Azure DevOps Organization, select your project, go to pipelines. Edit the pipeline. Copy the pipeline code below and paste it over in your new pipeline: # Starter pipeline # Start with a minimal pipeline that you can customize to build and deploy your code. # Add steps that build, run tests, deploy, and more: # https://aka.ms/yaml trigger: - main pool: name: myuser-sas-viya-jump-vm steps: - script: echo Hello, world! displayName: 'Run a one-line script' - script: | echo Add other tasks to build, test, and deploy your project. echo See https://aka.ms/yaml displayName: 'Run a multi-line script'   What "tells" Azure Pipelines to run on a self-hosted agent are these lines in the pipeline definition:   pool: name: myuser-sas-viya-jump-vm   The pipeline will pick a suitable, available agent from this agent pool.   Supposing your agent pool has more agents, you can constrain the pipeline to run only on a specific agent, with the “demands” clause. pool: name: myuser-sas-viya-jump-vm demands: - agent.name -equals myuser-jump-vm   Conclusion You configured an Azure virtual machine as a self-hosted agent using scripts. You ran a first Azure pipeline on the self-configured agent.   After the build, you can use this virtual machine as an Azure Pipelines self-hosted agent with SAS Viya. This will be the subject of the next post. Resources Used Microsoft Learn – Exercise – Create a build agent that runs on Azure.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content.   If you wish to get more information, please write me an email.   ... View more

This post shows you step by step how to configure a Virtual Machine hosted on Azure to communicate with SAS Viya deployed on Azure. After the configuration, you can use this virtual machine as an Azure Pipelines self-hosted agent. You can also use this machine in a standalone mode, to execute scripts that interact with SAS Viya. The interaction is possible via the SAS Viya Command Line Interface or via SAS Viya REST API requests.   Overall Picture   The post assumes that SAS Viya is deployed on Azure, in an Azure Kubernetes Service (AKS) cluster.   For SAS Viya to work with Azure DevOps and Azure Pipelines, a self-hosted agent is the key piece. A self-hosted agent is a Virtual Machine (VM). The steps to create your own self-hosted agent are:   Create a virtual machine on Azure. Read How to Create a SAS Viya Azure Pipelines Agent. Configure communication with the SAS Viya AKS cluster: create Network Security Groups rules; copy SAS Viya certificates (this post). Create an Azure DevOps self-hosted agent. Install software and packages on the self-hosted agent: SAS Viya Command Line Interface (CLI), Python, pyviyatools, utilities, etc.   Azure Network Security Groups and Rules   One of the key objectives, when working with a self-hosted Azure agent, which can be a VM, is to establish a communication line between:   The virtual network (VNET) where the VM is deployed (left in the figure below). The VNET where SAS Viya resources are deployed (right in the figure below).   The communication line is represented by the double yellow – orange arrow in the middle. I deliberately over-simplified the components on the SAS Viya side, to the right.     To understand a Network Security Groups (NSG), let us use an analogy.   A NSG is the security personnel guarding the doors of an event. To attend the event you must be on the attendees list. The event location is the VNET containing the resources you need to access.   Letting packages from our VM enter the location where SAS Viya “performs”, means adding NSG rules.   A rule has:   A sense: inbound or outbound. Is made of source and destination IPs and ports. A directive: Allow or Deny.   If your objective is to launch SAS Viya CLI commands, from the VM, to interact with SAS Viya, you should know that the CLI traffic transits typically over port 443.   The next step is to create a NSG rule.   Pre-Requisites: Login, set your subscription and location   You need to run this code from a machine which has the az cli and kubectl installed. You can use the Azure portal Cloud Shell or another Linux machine with cloud access.   # Variables - ADAPT them to your environment! export MYUSER=$USER RG=$MYUSER-SCR LOCATION=eastus VM=$MYUSER-jump-vm NSG=$MYUSER-nsg MYSUB=GELDM # Actions # az login # if not logged in already az account list -o table az account set --subscription $MYSUB az config set defaults.location=$LOCATION   SAS Viya Azure Kubernetes Service Network Security Group Rule (Right Side)     The next rule will allow traffic, from the public IP of your VM,  JUMPBOXIP, to the SAS Viya Load Balancer Public IP,  INBOUND_IP, over port 443. The rule is created in the aks-agentpool-yyyyyyyy-nsg. This NSG is associated to the Azure Kubernetes Service AKS virtual network​.   The inbound rule allows outside access, from the VM, through the ingress controller (load balancer), to the SAS Viya pods.   cd ~ # Get the public IP of the Linux Virtual Machine $MYUSER-jump-vm: JUMPBOXIP=$(az vm list-ip-addresses -g $RG -n $VM --query "[].virtualMachine.network.publicIpAddresses[0].ipAddress" -o tsv) echo "${MYUSER} Jump Box Vm's Public IP: $JUMPBOXIP" # find the internal AKS resource group RESOURCEGRP=${MYUSER}-azuredm-rg AKS_RG=$(az aks list -g $RESOURCEGRP --query [].nodeResourceGroup -o tsv) echo "AKS resource group: $AKS_RG" # find the public IP address of the AKS cluster INBOUND_IP=$(az network public-ip list --query "[].{tags: tags.type, address: ipAddress}" -o tsv -g $AKS_RG | grep None | cut -f2) echo "AKS inbound IP: $INBOUND_IP" # Get the NSG of the VM echo "VM NSG: $NSG" echo "Check: $AKS_NSG is filled!" echo "In the https://portal.azure.com navigate to $AKS_RG and search for the aks-agentpool-yyyyyyyy-nsg" # Create inbound NSG rule - port 443 az network nsg rule create -g $AKS_RG --nsg-name $AKS_NSG -n Allow443From-${MYUSER}-jump-vm \ --priority 496 \ --source-address-prefixes $JUMPBOXIP \ --source-port-ranges '*' \ --destination-address-prefixes $INBOUND_IP \ --destination-port-ranges '443' --access Allow \ --protocol Tcp --description "Allow access from New JumpBox on 443" echo "Rule is created in ${AKS_NSG}"   Physically, you will find the aks-agentpool-yyyyyyyy-nsg under the so-called nodeResourceGroup (usually prefixed by MC_ in SAS Viya deployments on Azure).   SAS Viya Certificates   To make your VM “talk” with SAS Viya, you will most likely pass SAS Viya CLI commands or REST API requests.   But before that, you will need to obtain and copy SAS Viya certificate files on the VM. Otherwise any communication attempt with SAS Viya will result in a certificate error.   The process to copy the certificates can be tricky and involves a few steps.   Get the Certificates   ### 0. Setup-Environment-Variables export MYUSER=$USER echo $MYUSER ### 1. Get the SAS Viya certificates from the AKS Cluster #### 1.1 Set the KUBECONFIG file. This is the Kubernetes .conf file allowing you to pass kubectl commands with your SAS Viya deployment. #### On this machine, the file sits in ~/.kube/$MYUSER-aks-kubeconfig.conf cd ~ export KUBECONFIG=~/.kube/$MYUSER-aks-kubeconfig.conf # adapt to your environment echo $KUBECONFIG #### 1.2 Set the current namespace, where SAS Viya is deployed and the context export current_namespace=gelazure # adapt to your environment kubectl config set-context --current --namespace ${current_namespace} kubectl config get-contexts #### 1.3 Get the kube .conf file az aks get-credentials --resource-group $MYUSER-azuredm-rg --name $MYUSER-azuredm-aks --admin -y ls -la ~/.kube #### 1.4 List the SAS Viya Certificates cd ~ ls -la ~/.certs #### You might not have the folder on your machine. Create the folder. mkdir -p ~/.certs #### 1.5 Get the SAS Viya Certificates .PEM File cd ~ kubectl cp $(kubectl get pod -l app=sas-logon-app -o=jsonpath='{.items[0].metadata.name}'):security/trustedcerts.pem ~/.certs/${current_namespace}_trustedcerts.pem ls -la ~/.certs #### 1.6 Backup the .PEM File before the copy cd ~/.certs echo Change permission before overwriting chmod 644 gelazure_trustedcerts.pem chmod 644 gelazure_trustedcerts.bak ls -la & pwd echo Backup the certificates before refresh cp gelazure_trustedcerts.pem gelazure_trustedcerts.bak ls -la #### 1.7 Get the SAS Viya Certificates / Refresh .PEM File cd ~ kubectl cp $(kubectl get pod -l app=sas-logon-app -o=jsonpath='{.items[0].metadata.name}'):security/trustedcerts.pem ~/.certs/${current_namespace}_trustedcerts.pem ls -la ~/.certs   Copy the SAS Viya Certificates to your Virtual Machine   ### 2. Copy SAS Viya Certificates to VM #### 2.1 Initialize variables. The VM and the resource group name. echo $MYUSER echo Assume VM name is $MYUSER-jump-vm # adapt to your environment echo Get the VM Public IP JUMPBOXIP=$(az vm list-ip-addresses -g $MYUSER-SCR -n $MYUSER-jump-vm --query "[].virtualMachine.network.publicIpAddresses[0].ipAddress" -o tsv) echo "JumpBox Public IP: $JUMPBOXIP" #### 2.2 Initialize variables: CURL CA Bundle and SSH Private Key. You will SSH the file to the VM. export CURL_CA_BUNDLE=~/.certs/${current_namespace}_trustedcerts.pem echo $CURL_CA_BUNDLE export private_key=~/.ssh/gelazuredm-aks-key cert=${current_namespace}_trustedcerts.pem echo $cert #### 2.3 Create the .certs folder on jump box VM. Assign permissions. Copy files. Verify copy: cd ~/.certs ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "sudo mkdir -p ~/.certs" wait echo Transfer folder ownership from root to jumpuser: ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "sudo chown jumpuser ~/.certs" wait echo Set folder permissions on the jump box, just in case: ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "sudo chmod -R 755 ~/.certs" wait echo Copy the .pem file: scp -o "StrictHostKeyChecking=no" -i ${private_key} ${cert} jumpuser@${JUMPBOXIP}:~/.certs wait echo Finally, list .pem file permissions: ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "ls -la ~/.certs" wait   Adjust the Permissions   On the machine you copied the certificates from, put the file permissions back to an acceptable range.   ### 3. Certs permissions - VM from where you are copying the certificates and Azure Jump Box VM #### 3.1 Change back file permissions, on your VM ~/.certs folder, after their copy on the jump box: chmod 400 ~/.certs/gelazure_trustedcerts.pem chmod 400 ~/.certs/gelazure_trustedcerts.bak ls -la ~/.certs #### if no longer needed on this machine, you can delete them from here. Do not delete them form the jump box. #### 3.2 On the Jump Box VM: ##### Change the certificate permissions ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "sudo chmod 400 ~/.certs/gelazure_trustedcerts.pem" wait ##### Set .certs folder permissions ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "sudo chmod 700 ~/.certs" wait echo List .pem file and .certs folder permissions ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "ls -la ~/.certs" wait ssh -o "StrictHostKeyChecking=no" -i ${private_key} jumpuser@${JUMPBOXIP} "ls -la ~" wait     Conclusions   This post shown how to configure a Virtual Machine hosted on Azure to communicate with SAS Viya deployed on Azure. The first step is to establish Network Security Group rules that allow the communication. The second step is to obtain the SAS Viya .pem file from the SAS Viya Azure Kubernetes Cluster and copy it to the Virtual Machine.   After the configuration, you can use this virtual machine as an Azure Pipelines self-hosted agent. This will be the subject of the next post: Create an Azure DevOps self-hosted agent. Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.   ... View more

With faceted search you can look at your information assets, from multiple lenses, called facets. You can search assets by name, asset type, description, by date or by user. You can explore data assets by library, file type, table type, number of columns or rows, size. You can search by column information such as name, label, keyword, or completeness percentage. More surprisingly you can search by language detected, semantic type calculated or data privacy status. Watch the video to understand how faceted search can help you find what you need or discover something new about your information assets.   What Is Faceted Search?   The faceted search in SAS Information Catalog is like a photographer’s lens kit. The facets are the lenses. Each has its own characteristics. They can capture different angles, colors, levels of detail. You switch them, until you can get the perfect shot. The perfect shot can be a unique perspective over your information assets in SAS Viya.     These facets are calculated when assets are automatically indexed or discovered with an agent.   The automatic index keeps track of models, reports, decisions, SAS Studio flows, data plans and CASLIB in-memory or SASHDAT tables available in your SAS Viya.   The discovery agent analyzes libraries: CASLIBs or SAS Compute. It produces tens of metrics for each table or file included in the discovery. These calculated metrics are then associated with facets, through which you can search.     Faceted Search Short Demonstration   Watch the following short video to understand what faceted search can help you discover in SAS Viya.   Asset Type Facets   Try a faceted search, AssetType, and it will list the asset types catalogued.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The results are the union of assets catalogued via: Automatic indexing, e.g. model will list all catalogued models. Discovery agents, e.g. inmemorytable will list all the CAS in-memory tables.   Semantic Types Facets   SAS Information Catalog is using a Quality Knowledge Base to calculate semantic types for columns in a table or file, based on their name or content. It also labels the columns from a data privacy perspective. For example: Column.informationPrivacy: "private" lists all tables that contain private data.     Column.semanticType: "individual" shows tables or files where columns have been classified as belonging to a private person.   Detected Language Facets   More recently, language detection is also performed in long text columns:     Languages: "en" will show you tables that have columns with English language content.   Languages: "zh" will show you which tables have Chinese language content.   Languages: "tl" will indicate assets where Tagalog was detected in a column.   To search all CAS tables that contain Korean language data try: +Languages: "ko" +AssetType: "cas" . The + indicates a MUST type of search, which means every facet criterion must be satisfied in the search results.     Combine Faceted and Free Text   SAS Information Catalog supports two types of searches, free text and faceted, which you can mix and match in a single query, for even better precision or more surprising findings.   Software Version   The above examples were produced using the SAS Viya Long Term Support Release 2022.1 (May 2022), although facet search has been around since at least October 2021. You will require a SAS Information Governance license for language, semantic type and information privacy features.   Additional Resources   For more information, see SAS® Information Catalog | Faceted Search. Alternatively, see SAS® Information Catalog | Free Text Search.   Conclusion   The faceted search allows you to look at your information assets, from multiple lenses, called facets. You can search assets by name, asset type, library, file type, table type, language detected, semantic type calculated or information privacy status, just to name a few.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

The key to working with SAS Viya and Azure Pipelines is a self-hosted agent. The post looks at what a self-hosted agent is, their advantages and disadvantages. The post then explains how a self-hosted agent can work with SAS Viya. Two examples are given: in the first, agents and SAS Viya are deployed on Azure; in the second, agents and SAS Viya are deployed on-premises. Understanding the agents architecture can help you move code and artifacts from one environment to another using pipelines. Read more to understand how this can be achieved.   Self-Hosted Agents   An agent is a system that performs build tasks. Build is the keyword you will find in the Microsoft documentation. Think of a build as of instructions, scripts or programs that must be executed.   The agent is a dedicated server that runs your build process. The self-hosted agent can be a Linux, Windows, macOS virtual machine or a Docker container.   Microsoft explains very well the build and the agents. "Imagine that you have an Azure Pipelines project that receives build requests many times per day. Or perhaps you have multiple projects that can each use the same type of build agent. You can organize build agents into agent pools to help ensure that there's a server ready to process each build request.   When a build is triggered, Azure Pipelines selects an available build agent from the pool. If all agents are busy, the process waits for one to become available."   For example, the next Azure Pipelines definition, captured in a YAML file, indicates that the pipeline must run on an agent, from the “sbxbot-sas-viya-jump-vm” agent pool.   # starter pipeline # Start with a minimal pipeline that you can customize to build and deploy your code. # Add steps that build, run tests, deploy, and more: # https://aka.ms/yaml trigger: - main pool: name: sbxbot-sas-viya-jump-vm steps: - script: echo Hello, world! displayName: 'Run a one-line script' - script: | echo Add other tasks to build, test, and deploy your project. echo See https://aka.ms/yaml displayName: 'Run a multi-line script'   ​Watch the following starter pipeline running on a self-hosted agent.     Self-Hosted Agents and SAS Viya   Let us now look at how a self-hosted agent interacts with SAS Viya. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Azure Pipelines acquires an available agent, from the named pool (Execute).   This agent clones the Git repository associated with the pipeline on the agent machine (Version). It will then run the instructions in the pipeline definition (Build). These instructions can be bash commands invoking the SAS Viya CLI commands or python programs calling SAS Viya REST requests.   When working with SAS Viya, self-hosted agents present advantages and disadvantages compared to Microsoft-Hosted Agents.   Advantages   You install the software you need only once. For example, a specific Python version or package, SAS Viya Command Line Interface, Azure Command Line Interface, etc. You therefore save time at every run. Each time you run a pipeline, it will be on the same VM.​ This helps with logging, debugging, you can connect to the VM using SSH, explore its folders, files, etc.​ The agent can run as a service.​ The service can start when you start the machine. The agent can accommodate larger disk space needed. There is no time limit for a pipeline to run.   Disadvantages   You manage a self-hosted agent yourself: the deployment, the software installation, the updates and the maintenance, etc.​ You must keep this agent up and running. That means additional costs.   When you put into a balance the advantages and disadvantages, you will prefer self-hosted agents when working with SAS Viya.   Agents and SAS Viya on Azure   Agents are great when you need to move code and artifacts from one environment to another.   The holy grail of CI/CD can be summarized as follows. Write your code once, in a SAS Viya environment, that we will call DEV. Next, deploy and run the code in any number of environments, with a minimum of intervention.   This is where agents can help.   To deploy the code and artifacts in another environment, which we will call TEST, you can simply change the agent pool name, in your pipeline code, from DEV to TEST. At least this is the theory.   This theory assumes you have identical SAS Viya deployments, with no configuration drift.   If your environments are slightly different, you need to parametrize your pipeline definition and your code base. You need to work with parameters, as the SAS Viya URLs will be different, files might have a different name or be in a different location in the two deployments.     One agent pool can contain several agents. For example the DEV agent pool can have two agents, both hosted on Azure:     Agents and SAS Viya on-Premises and on Azure   Azure DevOps supports a hybrid model, with agents running in a cloud or on-premises. You can configure an agent on a VM you are hosting on-prem. This machine must communicate with the Kubernetes cluster where SAS Viya is running.     For example, the DEV agent pool has just one agent, hosted on-premises:     In this example, you can run and test your code using the on-prem agent pool called DEV. When you are ready to promote the code to a SAS Viya on Azure instance, you will change the agent pool name to TEST.   You will not get identical SAS Viya deployments on-prem and on Azure. Therefore the deployment would require more work to map these parameters. The parameters can point to the SAS Viya URL, certificates, file names or their locations.   Azure Networking Considerations   On Azure, the arrow between an agent and the SAS Viya Azure Kubernetes Service cluster, can be depicted more accurately, as in the next figure.     One of the key objectives, when working with self-hosted Azure agents, is to establish a communication line between the agent network and the network where SAS Viya resources are deployed. This is the double orange arrow in the middle. I deliberately over-simplified the components on the right side, where SAS Viya resides.   Think of Network Security Groups (NSG) as the security guarding the doors of an event. To attend the event you must be on the list. The event is the virtual network where resources are deployed. The list is made of the inbound rules in the NSG.   Letting agents attend the SAS Viya “event”, means adding their IPs to the NSG's inbound rules. The agent traffic transits over port 443.   We might look closer at Azure networking for agents in a future post.    Conclusions   In this post, you read what an Azure Pipelines self-hosted agent is, its advantages and disadvantages. A self-hosted agent is probably the best option to work with a SAS Viya deployment. Lastly, the post mentioned how Azure Pipelines agents could help promote code from one environment to another. The next posts will look at agents configuration. First, at a self-hosted CentOS machine, running on-prem. Later, at an Ubuntu virtual machine hosted on Azure.   Azure DevOps and SAS Viya Resources SAS Viya Decision Pipeline with Azure DevOps. A SAS Viya Model Release Pipeline with Azure DevOps. A SAS Viya with Data Pipeline with Azure DevOps. SAS Viya CI/CD Pipelines with Azure DevOps. SAS Viya and Azure Pipelines Microsoft-Hosted Agents   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more