DevOps Applied to SAS Viya 3.5: Integrate a Data Studio Data Plan in a DevOps Pipeline

Have you ever been in a situation where users have developed their own jobs and plans on top of an existing process? Your challenge is to integrate a SAS Data Studio data plan in an existing DevOps pipeline, run it and manage any future changes.

Baseline

The client has a back of the napkin sketch. She/he wants to include in an existing process the work of several users. Only the users do not want to code, but feel comfortable with visual interfaces, such as SAS Data Studio. Therefore, your customer asks you to alter an existing DevOps pipeline to include a SAS Data Studio Data Plan.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

You think it might be feasible to accommodate the request. First, you take stock of the situation and you see two challenges:

The CHANGE challenge: Content is prone to often change. You do not want manual intervention when the “content” changes. You decide to write some code to store the data plan and the job definition into a GitLab repository.
The RUN challenge: the SAS Data Studio plan is “content”. In DevOps world, to run something, you need to script it. You can run-script SAS content by creating a job out of it. You can then execute the job using the sas-admin CLI (command line interface).

Steps

Firstly, you come with your version of "the back of the napkin" process:

810-SAS Data Studio Data Plan in DevOps pipeline process.png

Secondly, after some reflection, trial and error, and mails to your colleagues from the administration team, you develop the following process:

Develop the SAS Data Studio Data Plan. Recover the URI.
Create a job from the data plan. Recover the id.
Address the “Run”. Run the job with the id. Export the plan and the job request.
Address the “Change”. Push the plan, the job and Jenkins file definition in the GitLab repository.
Import the data plan.
Import the job and create a job request.
Execute the job request.
Test and analyze the results.

Steps 1-3 are the focus this post. The whole process can be summarized:

(view in My Videos)

Develop the SAS Data Studio Data Plan. Recover the URI.

The plan CatcodePlan reads the USER _CATCODE table loaded in CAS, performs corrections and writes the output in USER_CATCODE_STD. The plan is very simple in this version, but you are told the plan is likely to change often and more steps will be added.

The plan URI can be easily found in SAS Environment Manager: /dataPlans/plans/2e1e5b22-8519-4c81-be42-d1af3d346763.

Create a job from the data plan. Recover the id.

Recover the job id: f9f59d8e-2898-40c8-8c02-046a9e7c00df.

In SAS Environment Manager, logged in as an administrator, go to Jobs and Flows > Scheduling > CatcodePlanJob > Properties:

860-SAS-Environment-Manager-Job-properties-1024x392.png

At this stage, you have the data plan and the job. Ideally, you would want a pipeline that runs them automatically.

To include the plan in a DevOps process, you should ask your users, every time there is a change to the data plan, to:

Create a job from the plan
Export the data plan and the job request
Push both remotely in the GitLab repository linked to the Jenkins pipeline.

Address the “Run”. Export the plan and the job request.

You could use the sas-admin cli for these tasks. Please note you need administrative credentials (sasadm is used in the example below). On the sasviya01 machine, log in as sasadm:

## Set the Viya endpoint, data plan uri and job id 
cd /opt/sas/viya/home/bin/
export GITSERVER="http://sasviya01.race.sas.com"
echo $GITSERVER
export DPURI="/dataPlans/plans/2e1e5b22-8519-4c81-be42-d1af3d346763"
echo $DPURI
export JID="f9f59d8e-2898-40c8-8c02-046a9e7c00df"
echo $JID

# Refresh the Viya token – CHANGE USER AND PASSWORD
/opt/sas/viya/home/bin/sas-admin auth login -user sasadm -password *******

Log:

Enter credentials for http://sasviya01.race.sas.com: Login succeeded. Token saved.

Then:

# List the job request containing the data plan json - redirect to a .json file
/opt/sas/viya/home/bin/sas-admin -output json job requests show -id $JID > /tmp/CatcodePlanJob.json

See Gerry Nelson's post regarding the syntax of the cli.

Optional, but recommended, go to the .json file saved, change the description to something you can differentiate, such as "created from the cli".

If you look at the job definition .json file above, you will notice the file stores the data plan metadata inside.

Next: create a job request from the job definition .json file. The job request can be executed and not the job definition.

/opt/sas/viya/home/bin/sas-admin job requests create --file-in /tmp/CatcodePlanJob.json

The output:

POST /jobExecution/jobRequests from "/tmp/CatcodePlanJob.json completed." Created an object of type Job Request with id fb99832b-13ca-46c5-be1d-fdb4ebfb16aa.

You might want to automate the reading of the id from the output. Adjust the --limit parameter.

/opt/sas/viya/home/bin/sas-admin -output text job requests list --limit 3 --sort-by "``~Â´Â´name" > /tmp/DataPlan.txt
cat /tmp/DataPlan.txt

The output:

Id                                     Name                     Version   Description
a9d5b53d-6ed2-4b31-9605-5e0386992ff4   BINARY_BACKUP_SCHEDULE   3         This job is created by sas.deploymentBackup to run binary backup first Saturday at 5AM every month
05224291-c5b1-45f5-94ad-1000ff165fd6   Background               3         
d57fcaba-4106-4e52-aa03-d778af80dd33   CatcodePlanJob           3         created from the CLI

Lastly, you want to execute the job with the description 'created from the CLI' (the third in the list). You need its id for the execution.

# Read the id of the created job from the out file. Using a json parser Linux package called jq.
jid=$( /opt/sas/viya/home/bin/sas-admin --output json job requests list --filter 'eq(name,"CatcodePlanJob")' | jq -r '.items[]["id"]')

# Execute the job request
/opt/sas/viya/home/bin/sas-admin job requests execute $jid

The confirmation:

Executed the job request "fb99832b-13ca-46c5-be1d-fdb4ebfb16aa" fb99832b-13ca-46c5-be1d-fdb4ebfb16aa

At this stage you could check the creation of the job request in SAS Environment Manager:

And the job execution:

Lastly, export the data studio plan – you must know the URI.

/opt/sas/viya/home/bin/sas-admin transfer export --name "CatcodePlan" --resource-uri $DPURI/opt/sas/viya/home/bin/sas-admin transfer export --name "CatcodePlan" --resource-uri $DPURI

Log:

running .. completed { "items": [ { "contentType": "dataPlan", "message": "Resource exported successfully", "name": "CatcodePlan", "state": "completed" } ] } The package with the ID e9374aff-424b-4348-8dae-f878e1292172 was created.

Download the exported package with the generated ID.

/opt/sas/viya/home/bin/sas-admin transfer download --id e9374aff-424b-4348-8dae-f878e1292172 --file "/tmp/CatcodePlan.json"

The next steps to run the Data Studio Data Plan in a DevOps pipeline:

Push the plan, the job and Jenkins file definition in the GitLab repository.
Import the data plan.
Import the job and create a job request.
Execute the job request.
Test and analyze the results.

DevOps Applied to Viya 3.5 Posts

Want to Learn More about Viya 3.5?

Thank you for your time reading this post. Please comment and share your experience with Jenkins, git and SAS. If you wish to get more information, please write me an email.