Configure a SAS Viya Self-Hosted Agent with Azure Pipelines

Read the post to understand how you can use Azure Pipelines to install packages, tools and then pass commands from the SAS Viya Command Line Interface (CLI) or pyviyatools. The goal is to interact with SAS Viya hosted in Microsoft Azure from an Azure hosted virtual machine. You can use for this purpose the self-hosted agent you created and registered.

Overall Picture

The steps to create your own self-hosted agent are:

Create a virtual machine on Azure. Read How to Create a SAS Viya Azure Pipelines Agent.
Configure communication with the SAS Viya Azure Kubernetes Service cluster. Read How to Make Your Virtual Machine Talk with SAS Viya on Azure.
Create an Azure DevOps self-hosted agent.
Install software and packages on the self-hosted agent: SAS Viya Command Line Interface (CLI), Python, pyviyatools, utilities, etc. (this post).

Azure Pipeline Code

The Azure Pipeline code provided further will focus on four things:

Install and update needed packages on the agent.
Download and install the SAS Viya CLI on the agent.
Clone and install the pyviyatools. The pyviyatools are a set of command-line tools that call the SAS Viya REST APIs from Python. These tools simplify SAS Viya administration tasks. The source code is hosted in GitHub.
Test the SAS Viya CLIs and pyviyatools.

The Azure Pipeline code is written in yaml.

Pipeline Header

Parameters

The parameters section contains:

MYUSER: a string used to prefix the SAS Viya URL and Azure resources.
sasviyaclitgz: a compressed archive containing the sas-viya (CLI) executable. This points to the file name sas-viya-cli-1.21.0-linux-amd64.tgz. You can download it from the SAS Downloads Page and upload this file in the Git repository accompanying your Azure pipeline. Your Git repository can be hosted in Azure Repos, GitHub, Bitbucket, Gitlab, just to name a few options.

bt_1_240_Configure_SAS_Viya_Self_Hosted_Agent_Pipeline_SAS_Viya_CLI_Downloads.png

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

Variable

The variable defines the self-hosted agent pool name created in How to Build an Azure Self-Hosted Agent for SAS Viya.

You can use parameters in variables. For example, ${{ parameters.MYUSER }}-sas-viya-jump-vm becomes sbxbot-sas-viya-jump-vm when MYUSER=sbxbot.

Azure Pipelines does not allow you to use agent pool names fully stored in a parameter.

# this pipeline configures the SAS Viya self-hosted agent
parameters:
- name: MYUSER
  displayName: SAS User ID
  type: string
  default: 
- name: sasviyaclitgz
  displayName: SAS Viya CLI TGZ Name
  type: string
  default: sas-viya-cli-1.21.0-linux-amd64.tgz # Nov 2022

variables:
- name: pool
  value: ${{ parameters.MYUSER }}-sas-viya-jump-vm

trigger:
- main

pool:
  name: $(pool)

The trigger tells Azure Pipelines to start a build, a new pipeline run, when somebody pushes a change to the main branch of the underlying Git repository.

Pool

The pool tells the pipeline to build on (use) a self-hosted agent from your agent pool.

Pipeline Stage

You can structure an Azure pipeline in a hierarchy of stages, jobs, scripts or tasks. The hierarchy helps you keep the pipeline code clear and allows you to build-in dependencies between jobs or stages.

Install Job

There is only one stage called install. The installPackages job clones the Git repository on the agent in the checkout step. You will notice the checkout step repeated in every job. Further, the next four bash scripts will run on the agent, which is a Linux Ubuntu VM. The scripts will:

Update packages.
Install Python.
Install jq (a command-line JSON processor).
List the versions of the components installed.

stages:
- stage: install
  displayName: Install
  jobs:
  - job: installPackages
    displayName: Install and Update Needed Packages
    steps:
    - checkout: self
      persistCredentials: true
    - script: |
        echo 'List directory where Git Repo was cloned: ' && pwd
        ls -la
        echo Get the folder where the Git repo was cloned
        export GITFOLDER=$(pwd)
        echo Update Advanced Package Tool, or APT
        sudo apt update
        sudo apt -y upgrade
      displayName: 'Install and Update APT'
    - script: |
        echo Install Python 3.x with pip3
        sudo apt install python3-pip -y
      displayName: 'Install Python 3.x'
    - script: |
        echo Install a Linux Json parser named jq
        sudo apt install jq -y
      displayName: 'Install JQ'
    - script: |
        echo Check python
        python3 --version
        echo Check pip3 is installed
        pip3 --version
        echo Check jq
        jq --version
      displayName: 'Check install'

bt_2_250_Configure_SAS_Viya_Self_Hosted_Agent_Pipeline_install_packages-1024x488.png

Install SAS Viya CLI Job

The next job has also four scripts. Their purpose is to:

Un-compress the archive containing the sas-viya CLI executable.
Move the executable in the $clidir folder.
Install the SAS Viya CLI in that folder.
Test the installation, by listing the plug-ins.

- job: sasviyacli
    dependsOn: installPackages
    displayName: Install the SAS Viya CLI
    steps:
    - checkout: self
      persistCredentials: true
    - script: |
        echo 'List directory where Git Repo was cloned: ' && pwd
        ls -la
        echo Get the folder where the Git repo was cloned
        export GITFOLDER=$(pwd)
        echo Unpack the tgz
        tar zxvf ${{ parameters.sasviyaclitgz }}
        # confirm you have the sas-viya executable in your folder
        ls -la
      displayName: 'Unpack the SAS Viya CLI'
    - script: |
        echo 'List directory where Git Repo was cloned: ' && pwd
        ls -la
        echo Get the folder where the Git repo was cloned
        export GITFOLDER=$(pwd)
        clidir=/opt/sas/viya/home/bin
        echo Create $clidir folder
        sudo mkdir -p $clidir
        echo Move the sas-viya program to its default folder
        sudo mv sas-viya $clidir
        echo Give exec permissions to sas-viya in $clidir folder
        sudo chmod +x $clidir/sas-viya
        echo List $clidir folder
        ls -la $clidir
      displayName: 'Move the SAS Viya CLI'
    - script: |
        echo Install the SAS-VIYA CLI plugins
        clidir=/opt/sas/viya/home/bin
        echo Set viya_cli_repository
        viya_cli_repository=https://support.sas.com/repos/viyacli/
        echo $viya_cli_repository
        echo INSTALLING ALL plugins
        for p in $($clidir/sas-viya plugins list-repo-plugins --repo SAS | awk 'NR > 3 { print $1 }' )
           do
              $clidir/sas-viya plugins install --repo SAS ${p#"*"}
           done
        echo Done installing SAS_VIYA CLI
        echo List plugins
        $clidir/sas-viya plugins list
      displayName: 'Install the SAS Viya CLI'
    - script: |
        clidir=/opt/sas/viya/home/bin
        ls -la $clidir
        echo Test the sas-viya cli
        $clidir/sas-viya --help
        echo List sas-viya cli plugins
        $clidir/sas-viya plugins list
      displayName: 'Test the SAS Viya CLI'

260_Configure_SAS_Viya_Self_Hosted_Agent_Pipeline_Install_CLI.png

Create a SAS Viya CLI Profile and Install pyviyatools

In addition to scripts, in a job you can have tasks, such as the AzureKeyVault task.

Always use an Azure Key Vault instead of hardcoding passwords or secrets in your pipeline code! Remember that the pipeline code is stored in a Git repository. Even if the repository is private, it may still be consulted by many other users. If the Git repository is made public (by mistake, or forked and made public), well, your passwords will be visible, on the web for anyone to consult. Hence, the need for the key vault.

You should use it to retrieve secrets, such as the $(SASPASS) password from an Azure Key Vault of your choice.

After the task, the script clones the public GitHub repository that contains the pyviyatools.

Next, it creates a SAS_CLI_PROFILE to configure the SAS Viya CLI for a particular SAS Viya deployment. Notice how the SSL_CERT_FILE references the .pem file copied on the self-hosted agent in How to Make Your Virtual Machine Talk with SAS Viya on Azure. Without a valid certificate, any communication attempt will be blocked by SAS Viya. You also have to create certain Network Security Group rules to allow communication with your Azure Kubernetes Service cluster over the port 443 (HTTPS).

Even further, the script logs the user in the SAS Viya CLI, using the $(SASPASS)password and obtains a SAS Viya access token. Finally, it runs the setup for the pyviyatools. The setup uses the SAS Viya CLI profile created.

- job: pyviyatools
    dependsOn: installPackages
    displayName: Install the pyViyaTools
    steps:
    - checkout: self
      persistCredentials: true

    - task: AzureKeyVault@2
      inputs:
        azureSubscription: 'GELDM(41baf198-cc93-4165-9882-6804e5fb95c0)'
        KeyVaultName: '${{ parameters.MYUSER }}-key-vault'
        SecretsFilter: '*'
        RunAsPreJob: true

    - script: |
        echo Clone the repo
        cd ~
        git clone https://github.com/sassoftware/pyviyatools.git
        echo go to the folder
        ls ~/pyviyatools
        ls
        echo Set Variables
        echo Initialize needed variables
        export SAS_CLI_PROFILE=gelazure
        echo $SAS_CLI_PROFILE
        export current_namespace=gelazure
        export CURL_CA_BUNDLE=~/.certs/${current_namespace}_trustedcerts.pem
        echo $CURL_CA_BUNDLE
        export SSL_CERT_FILE=~/.certs/${current_namespace}_trustedcerts.pem
        echo ${SSL_CERT_FILE}
        export REQUESTS_CA_BUNDLE=${SSL_CERT_FILE}
        echo ${REQUESTS_CA_BUNDLE}
        echo Running with user: ${{ parameters.MYUSER }}
        export INGRESS_URL=https://${{ parameters.MYUSER }}-gel.eastus.cloudapp.azure.com
        echo Viya URL: ${INGRESS_URL}
        clidir=/opt/sas/viya/home/bin
        echo SAS-Viya CLI directory is: $clidir
        echo Create SAS-Viya CLI Profile
        $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile set-endpoint ${INGRESS_URL}
        $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile toggle-color on
        $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile set-output fulljson
        echo Show profile
        $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile show
        echo Login in
        $clidir/sas-viya --profile ${SAS_CLI_PROFILE} auth login -user sasadm -password $(SASPASS)
        echo Show profile after login
        $clidir/sas-viya --profile ${SAS_CLI_PROFILE} profile show
        echo Install pyviyatools
        python3 ~/pyviyatools/setup.py
      displayName: 'Install and setup the pyViyaTools'

270_Configure_SAS_Viya_Self_Hosted_Agent_Pipeline_install_pyviyatools.png

Test the pyviyatools

A pipeline job running on the self-hosted agent functions like a self-contained code unit. That is why every job recreates the SAS_CLI_PROFILE. The pipeline will also work if you create it once, then you just reuse the profile in other jobs. Personally, I like to keep each job independent. It allows me to copy them in another pipeline or a new job.

The command python3 ~/pyviyatools/loginviauthinfo.py -f ~/.authinfo obtains a SAS Viya access token using pyviyatools.

- job: testPyviyatools
    dependsOn: pyviyatools
    displayName: Test the pyViyaTools
    steps:
    - checkout: self
      persistCredentials: true
    - script: |
        echo Set Variables
        echo Create SAS-Viya CLI Profile
        echo Initialize needed variables
        export SAS_CLI_PROFILE=gelazure
        echo $SAS_CLI_PROFILE
        export current_namespace=gelazure
        export CURL_CA_BUNDLE=~/.certs/${current_namespace}_trustedcerts.pem
        echo $CURL_CA_BUNDLE
        export SSL_CERT_FILE=~/.certs/${current_namespace}_trustedcerts.pem
        echo ${SSL_CERT_FILE}
        export REQUESTS_CA_BUNDLE=${SSL_CERT_FILE}
        echo ${REQUESTS_CA_BUNDLE}
        echo Running with user: ${{ parameters.MYUSER }}
        export INGRESS_URL=https://${{ parameters.MYUSER }}-gel.eastus.cloudapp.azure.com
        echo Viya URL: ${INGRESS_URL}
        clidir=/opt/sas/viya/home/bin
        echo SAS-Viya CLI directory is: $clidir
        echo Get a SAS Viya Token with pyviyatools
        python3 ~/pyviyatools/loginviauthinfo.py -f ~/.authinfo
        echo Show setup
        python3 ~/pyviyatools/showsetup.py
        echo Listing CASLIBS with pyviyatools
        python3 ~/pyviyatools/listcaslibs.py
      displayName: 'Test Login and Setup with pyViyaTools'

The login requires an .authinfo file, created on the self-hosted agent, before running the pipeline.
cd ~
tee ~/.authinfo > /dev/null << EOF
default user sasadm password ***your_password_here***
EOF
sudo chmod 600 ~/.authinfo
ls -la ~/.authinfo

You will see the global CASLIBS listed, as a proof that your self-hosted agent can now interact with SAS Viya.

280_Configure_SAS_Viya_Self_Hosted_Agent Pipeline_Test_CLI.png

Conclusion

You can pass SAS Viya Command Line Interface (CLI) commands from Azure Pipelines or make SAS Viya REST APIs calls from Python, via pyviyatools. The pipelines are using a self-hosted Linux Ubuntu agent from Microsoft Azure.

Additional Resources

Create a virtual machine on Azure. Read How to Create a SAS Viya Azure Pipelines Agent.
Configure communication with the SAS Viya Azure Kubernetes Service cluster. Read How to Make Your Virtual Machine Talk with SAS Viya on Azure.
Create an Azure DevOps self-hosted agent.
See Gerry’s posts:
- Introducing the GEL pyviyatools.
- The pyviyatools installation instructions are found here: install.md.
A SAS Viya Model Release Pipeline with Azure DevOps
A SAS Viya with Data Pipeline with Azure DevOps.
A SAS Viya Decision Pipeline with Azure DevOps.
SAS Viya CI/CD Pipelines with Azure DevOps.

Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.

Find more articles from SAS Global Enablement and Learning here.