DevOps Applied to SAS Viya 3.5: Run a SAS Program with a Jenkins Pipeline

Started ‎04-26-2020 by

Modified ‎09-22-2021 by

Automate the development, testing and execution of SAS (CAS) code in SAS Viya 3.5 with Git and Jenkins. Learn how to create a simple Jenkins pipeline and the basics of Jenkins pipeline syntax, with a simple example to load flat files in CAS.

Jenkins is an open source automation server. With a Jenkins file, also stored in GitLab (or GitHub), Jenkins can run code (SAS, shell, etc.) on hosts called agents. SAS Viya can be an agent.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

A continuous integration / continuous delivery (CI/CD) pipeline is a sequence of events or jobs that can be executed.

CI continuous integration = version control.

CD continuous delivery = get software from version control right through to your users and customers.

CI/CD Business Benefits: Accelerate time-to-value. Reduce costs. Experimentation = innovation. Data-driven decision making. Quality.

Jenkins

Jenkins is one of the best known open source automation server for CI/CD.

The Jenkins Agent: SAS Viya

By default, Jenkins will attempt to run builds (jobs, code) wherever it is installed.

Jenkins can be directed to run on other hosts, called agents in "Jenkins-speak". This capacity is one powerful feature. SAS Viya can be “recruited” as a Jenkins agent.

For Jenkins, the agent code name, the label viyahost matters. On this SAS Viya machine, Jenkins must use a working folder to do its thing (such as /opt/sas/devops/workspace/ ).

Jenkins will launch an agent. It will then connect to the host indicated, with the credentials given. That means Jenkins will run all commands (and SAS programs) on the Viya machine, as the user specified.

GitLab

GitLab is a web-based DevOps lifecycle tool that provides a Git-repository manager and version control.

The SAS program we want to automate with Jenkins, 080_load_CAS.sas is stored in a GitLab project.

The necessary git commands were discussed in a previous post.

The SAS (CAS) Program

The script initiates a CAS session and loads a few files. I found out that you must use casauto if you want to run the CAS program in batch (which you must with Jenkins).

cas casauto;
* source caslib;
%let DM_src=DM;
* target caslib;
%let DM_TRG=Public;
* Your UserID;
%let userid=Your course assigned user ID (e.g. gatedemoxxx);

* Load files into target caslib;
* Drop first;
proc casutil incaslib="DM_TRG";
droptable casdata="&userid._catcode" quiet;
droptable casdata="&userid._mailorder" quiet;
droptable casdata="&userid._products" quiet;
droptable casdata="&userid._customers" quiet;
quit;

* Load;
proc casutil incaslib="&;DM_SRC" outcaslib="&DM_TRG";
load casdata="catcode.csv" casout="&userid._catcode" copies=0 promote;
load casdata="mailorder.csv" casout="&userid._mailorder" copies=0 promote;
load casdata="products.csv" casout="&userid._products" copies=0 promote;
load casdata="customers.csv" casout="&userid._customers" copies=0 promote;
quit;
cas casauto terminate;

The Jenkins Pipeline

The Jenkins Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines "as code". To define a pipeline, go to Jenkins > New Item (top left).

Choose Pipeline.

Indicate where the Jenkins pipeline definition (Jenkins file) can be found. In this example, the Jenkins file is stored in GitLab.

Definition: Pipeline script from SCM (source code management)
SCM: GIT (when the Jenkins file is stored in GitHub or GitLab)
Repository URL: this is your GitLab project suffixed by .git.
Branch: master (could be a different one).
Credentials: your GitLab or GitHub user and pass.
Script path: the relative path to your Jenkins file from the GitLab project root

The Jenkins File

The definition of a Jenkins Pipeline is typically written in a file, called Jenkinsfile. The Jenkins pipeline syntax is written in Apache Groovy.

In the GitLab project create a New file, called Jenkinsfile (or create it locally and push it remotely):

pipeline {
    agent { label 'viyahost'}
    stages {
        stage('Clone GIT on SAS Viya') {
            steps {
                sh 'echo "Hello " `logname`'
            }
        }
        stage('Copy source files') {
            steps {
                sh 'cp /opt/sas/devops/workspace/SBXBOT-PSGEL250-devops-applied-to-sas-viya-3.5/Data-Management/source-data/* /gelcontent/demo/DM/data/'
            }
        }
        stage('Load in CAS') {
            steps {
                sh '/opt/sas/spre/home/SASFoundation/sas -autoexec "/opt/sas/viya/config/etc/workspaceserver/default/autoexec_deployment.sas" /opt/sas/devops/workspace/SBXBOT-PSGEL250-devops-applied-to-sas-viya-3.5/Data-Management/scripts/080_load_CAS.sas -log /tmp/080_load_CAS.log'
            }
        }
    }
    post { 
        always { 
            echo 'Job Done!'
        }
    }
}

A few words on the Jenkins file syntax:

The agent ‘viyahost’ (SAS Viya) executes the pipeline.
A pipeline contains stages.
The stages can contain any type of script, for example shell that performs certain operations and runs SAS (CAS) code in batch.
The pipeline can contain post operations, run stages in parallel, expose detailed logs, import SAS content such as SAS Data Studio plans or SAS Visual Analytics reports.
For more info, see the Jenkins pipeline syntax.

Driven by the Jenkinsfile above, Jenkins will build a pipeline to:

Clone the GitLab project on the SAS Viya machine. The clone is sent to the folder defined in the agent workspace folder /opt/sas/devops/workspace/ , under a subfolder corresponding to the Jenkins pipeline name SBXBOT-PSGEL250-devops-applied-to-sas-viya-3.5 .
Copy the files in the folder matching ‘DM_SRC’ caslib in The SAS Program:

        stage('Copy source files') {
            steps {
                sh 'cp /opt/sas/devops/workspace/SBXBOT-PSGEL250-devops-applied-to-sas-viya-3.5/Data-Management/source-data/* /gelcontent/demo/DM/data/'
            }
        }

Run a SAS script in batch to load the files in CAS and write the output to a log.

        stage('Load in CAS') {
            steps {
                sh '/opt/sas/spre/home/SASFoundation/sas -autoexec "/opt/sas/viya/config/etc/workspaceserver/default/autoexec_deployment.sas" /opt/sas/devops/workspace/SBXBOT-PSGEL250-devops-applied-to-sas-viya-3.5/Data-Management/scripts/080_load_CAS.sas -log /tmp/080_load_CAS.log'
            }
        }
    }

Print a message: ‘Job done’.

    post { 
        always { 
            echo 'Job Done!'
        }
    }

Run the Jenkins pipeline

I am using Blue Ocean, a Jenkins plug-in, to run and visualize pipelines.

Your pipeline will show off previous builds, if any.

Run the pipeline.

If all was set up correctly, you would see the result, in the latest run.

The Jenkins file has been converted in a pipeline, containing the stages described in the Jenkinsfile.

You can consult the individual steps, look at their status and the logs.

In SAS Viya, you will see the loaded tables in CAS:

Conclusions

We automated the loading of a CAS table with SAS Viya 3.5, Jenkins and GitLab. A Jenkins pipeline was defined, with the SAS Viya machine as the agent. The Jenkinsfile containing the Jenkins syntax was stored in GitLab, along with the SAS program to be executed. Finally, we ran the Jenkins pipeline, analyzed the results and confirmed visually the results in CAS.

More will follow: how to introduce tests, parallelize stages, surface detailed logs or import SAS content, such as SAS Data Studio Plans or SAS Visual Analytics reports.

Acknowledgements

Mark Thomas, Rob Collum, Stephen Foerster.

References

Want to Learn More about Viya 3.5?

Thank you for your time reading this post. Please comment and share your experience with Jenkins and SAS.

Article Labels

Article Tags