Automation is key in the DevOps philosophy. If you plan to cross the "Automation Valley" to Continuous Integration (CI) land with SAS Data Integration (DI), then the Git plug-in, a Git repository and Jenkins will help you get there faster. The SAS Data Integration Studio Git plug-in is managing DI jobs as archives (.spk). It is also handling the git commands (pull, stage, commit, push, etc.) to work with a remote repository.
But from an archived job to Continuous Integration, there is a valley to cross. Let's call it the "Automation Valley".
And to cross that valley, you need to build a bridge. In Continuous Integration land, the bridge must be built on the fly.
The bridge must be made of archived jobs, scripts to import archives, and scripts to create, deploy and run the jobs.
Jenkins builds the bridge with these materials. The building plan is the Jenkins pipeline script.
When you combine SAS DI with the Git plug-in, a Git repository, such as GitLab and an automation server, Jenkins, the Continuous Integration land can be reached.
The post is building on the foundations laid in DevOps Applied to SAS 9: SAS Code, GitLab and Jenkins and SAS DI Developers: Unite! The new GIT plug-in in Data Integration Studio.
The following example assumes a SAS 9.4M6 platform on Windows. You will need:
Please note, the Git plug-in is only available as of SAS Data Integration Studio version 4.904.
If you did not setup the Git Plug-in your environment, you might want to see the second part of the post SAS DI Developers: Unite! The new GIT plug-in in Data Integration Studio post.
From Pre-Requisites using the Git Plug-In, complete the steps 1, 2, 4 and 5. Skip step number 3. Git on Windows is needed and is part of step 4.
In the above post, in step 3, a GitHub repository was used. The next example assumes a GitLab repository.
GitLab is a web-based DevOps lifecycle tool, using Git technology. You need to set up (or have access to) a GitLab repository:
The GitLab repository has to be accessible from the SAS 9.4 M6 Windows machine. (In my example, it is hosted at https://<server>/gatedemo025/psgel250-devops-applied-to-sas-94m6.git .
Jenkins is the automation server. You would need to create a Jenkins pipeline, have a pipeline script, and configure the Windows machine where SAS 9.4M6 is running, as a Jenkins agent.
On the Windows machine where SAS 9.4 M6 is installed:
In SAS Data Integration, from Tools > Options, configure the Git Plug-in:
Git Repository URL, this is your GitLab project, e.g., https://<server>/gatedemo025/psgel250-devops-applied-to-sas-94m6.git.
Location of the Git repository, the folder you created above: C:\temp\DevOps\
Connection type, depends on how you can access your repository: https, http or ssh
Initialize the repository button clones locally the GitLab repository. By default, the Git plug-in clones into <folder>\gitPlugin\gitRepo\ .
Create a new SAS Data Integration job, called List_Job (in my example, it is stored in the /Data Governance/Jobs folder). Add a User Written Code transformation, called List SAS.
In the Code tab, add:
proc printto print="C:\temp\DevOps\batch\SAScodeoutput.txt" new;run;
proc print data=sashelp.birthwgt (obs=10); run; proc printto; run; quit;
Save the transformation and the job.
Right click on the job and Archive as a SAS package. Archive will transform the DI job in a SAS package (.spk), stage, commit the changes and push them to the remote repository.
The Git plug-in archives the job, by default, in the /root of the Git repository.
To run a SAS DI job in a pipeline, it needs to be deployed first.
Right click the job List_Job > Scheduling > Deploy. Create a New Deployment Directory. Call it JobsDeployed, map it with C:\temp|DevOps\batch folder.
When you are done, you can delete the List_Job from SAS Data Integration folder.
Optional, you could set a webhook with your GitLab repository. The advantage is that, when new code is pushed to the remote repository, Jenkins will rebuild a new pipeline.
Instruct your pipeline to use the pipeline script in the GitLab repository. You need the repository URL and credentials to access the remote repository.
Need more details in regards to a pipeline definition? Go to DevOps Applied to SAS Viya 3.5: Run a SAS Program with a Jenkins Pipeline (The Jenkins Pipeline paragraph).
You then need to allow Jenkins to drive the DevOps pipelines on the SAS 9.4 M6 Windows machine. You must set up a Jenkins node, or agent.
On the designated Windows machine, Jenkins will install a node service that allows communication back with the controller.
The agent setup can be tricky, especially for a first time. Search online for "setup a Jenkins Windows agent". There are some excellent videos out there.
The agent must therefore be online before Jenkins can run anything on the SAS Windows agent.
You can create the Jenkinsfile directly in the GitLab repository, in the “Data-Management” folder (you specified the location when you created the pipeline).
pipeline {
agent { label 'sasclient.race.sas.com'}
stages {
stage('List Environment Variables') {
steps {
echo "Running ${env.JOB_NAME} in ${env.WORKSPACE} on ${env.JENKINS_URL}"
}
}
stage('Import the SAS DI job from the .spk') {
environment {
UserCredentials = credentials('SAS9Administrator')
}
steps {
withCredentials([usernamePassword(credentialsId: 'SAS9Administrator', passwordVariable: 'SAS9AdminPass', usernameVariable: 'SAS9Admin')]) {
// some block
bat '''
echo "Importing job in SAS DI with Masked Credentials"
@echo on
echo "Passing credentials: user" %SAS9Admin%
echo "Passing credentials: pass" %SAS9AdminPass%
"C:\\Program Files\\SASHome\\SASPlatformObjectFramework\\9.4\\ImportPackage" -host "sasclient.race.sas.com" -port 8561 -user %SAS9Admin% -password %SAS9AdminPass% -package List_Job.spk -target "/Data Governance/Jobs" -log "C:\\temp\\DevOps\\gitPlugin\\gitRepo\\JobImport.log"
'''
}
}
}
stage('Deploy the SAS DI job') {
steps {
withCredentials([usernamePassword(credentialsId: 'SAS9Administrator', passwordVariable: 'SAS9AdminPass', usernameVariable: 'SAS9Admin')]) {
// some block
bat '''
echo "Deploying the SAS DI job as SAS 9.4 code"
@echo on
echo "Passing credentials: user" %SAS9Admin%
echo "Passing credentials: pass" %SAS9AdminPass%
"C:\\Program Files\\SASHome\\SASDataIntegrationStudioServerJARs\\4.8\\DeployJobs.exe" -host "sasclient.race.sas.com" -port 8561 -user %SAS9Admin% -password %SAS9AdminPass% -deploytype deploy -objects "/Data Governance/Jobs/List_Job" -sourcedir "C:\\temp\\DevOps\\batch" -deploymentdir "C:\\temp\\DevOps\\batch" -metarepository Foundation -metaserverid A5CTTMUV.AT000002 -servermachine "sasclient.race.sas.com" -serverport 8591 -serverusername "sasdemo" -serverpassword %SAS9AdminPass% -batchserver "SASApp - SAS DATA Step Batch Server" -folder "JobsDeployed" -log "C:\\temp\\DevOps\\gitPlugin\\gitRepo\\JobDeployment.log"
'''
}
}
}
stage('Run the deployed SAS DI job') {
steps {
bat '''
echo "Running SAS 9.4 code on Windows client"
"C:\\Program Files\\SASHome\\SASFoundation\\9.4\\Sas.exe" -sysin C:\\temp\\DevOps\\batch\\List_Job.sas -log C:\\temp\\DevOps\\batch\\List_Job.log -nosplash -nologo -noicon
type C:\\temp\\DevOps\\batch\\List_Job.log
type C:\\temp\\DevOps\\batch\\SAScodeoutput.txt
'''
}
}
}
post {
success {
echo 'Imported and ran a SAS DI job on a Windows client'
}
}
}
The pipeline syntax explained:
If you have setup a webhook, the pipeline will run after each commit in GitLab. Look at each stage result:
Environment variables: Jenkins pulls the code from GitLab in a folder on the Windows machine.
Import the SAS DI job from the .spk:
Deploy the SAS DI job:
Run the deployed SAS DI job:
There are no tests in the pipeline, just to keep the post short, but it is best practice to design and include them. With testing, the "bridge" is complete and fit for purpose.
In Continuous Integration land, everything is short-lived, and bridges are usually destroyed after they are built.
A last step, after testing, might have been to delete the deployed job, the job and the logs on the Windows machine, upon successful pipeline completion. Once you have your pipeline definition and script, you can rebuild anytime.
When you combine SAS DI with the Git plug-in, a Git repository, such as GitLab and Jenkins, Continuous Integration can be achieved.
The benefits of combining these tools:
Stephen Foerster, Mark Thomas, Rob Collum.
Thank you for your time reading this post. Please comment and share your experience with SAS and DevOps.
Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.
If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website.
Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning and boost your career prospects.