How to Run SAS Programs in Batch in the New SAS Viya

6 Likes

Did you know there is a new command line interface in the new SAS Viya (2020.1 and later)?

The Essential

To run in batch a .sas program called 1.sas:

In the new SAS Viya (2020.1 and later) you could use the Command-Line Interface (CLI) batch plug-in:


/opt/sas/viya/home/bin/sas-viya --profile ${SAS_CLI_PROFILE} batch jobs submit-pgm --pgm-path /tmp/1.sas --context default  --watch-output --wait-log-list --results-dir /tmp

vs. SAS Viya 3.5 where you had to write:


/opt/sas/spre/home/SASFoundation/sas -autoexec "/opt/sas/viya/config/etc/workspaceserver/default/autoexec_deployment.sas" /tmp/1.sas -log /tmp/1.log

More Details

SAS Batch Command Line Utility is a command line interface that enables you to submit SAS programs or commands from a command line to a SAS Viya environment running in a Kubernetes cluster for batch processing. The principles are described Command-Line Interface: Overview.

The top-level command is sas-viya. It is used to initialize, authenticate, and execute the batch plug-in. For a complete plug-in list, see Command-Line Interface: Plug-Ins.

Example

There are some extra steps before you can submit your sas-viya batch jobs submit-pgm command.

Setup a profile

Find your Kubernetes namespace. In this example, SAS Viya is running in the gelcorp namespace. If you don't know in which namespace Viya is running, type:

kubectl get ns

The SAS Viya applications (e.g. SAS Environment Manager, SAS Drive, etc.) can be accessed, by default, at the location described by $INGRESS_URL.

current_namespace=gelcorp
INGRESS_SUFFIX=$(hostname -f)
INGRESS_URL=http://${current_namespace}.${INGRESS_SUFFIX}
echo ${INGRESS_URL}

TLS case

CAS supports encrypted connections between the server and the clients. You can enable TLS to secure communications between the server and clients. The certificate used for client server communication needs to be signed by a certificate authority (CA) that is trusted by all potential clients.

Therefore, if you have a SAS Viya deployment, where TLS is enabled, you need to execute these commands to get the certificates once. You need a kubectl admin role.

mkdir -p ~/.certs
kubectl cp $(kubectl get pod -l app=sas-logon-app -o=jsonpath='{.items[0].metadata.name}'):security/trustedcerts.pem ~/.certs/${current_namespace}_trustedcerts.pem

Authentification

You need to run the following commands every time, before the sas-viya profile and auth commands. These will point the CLI to the location of the certificates and certificate authority bundle (TLS).

export SSL_CERT_FILE=~/.certs/${current_namespace}_trustedcerts.pem
export REQUESTS_CA_BUNDLE=${SSL_CERT_FILE}

Switch to the folder where the SAS VIYA CLI is installed. I assume you installed it in the $clidir.

clidir=/opt/sas/viya/home/bin
cd $clidir

Create a functional profile called ${SAS_CLI_PROFILE}. This resolves to gelcorp.

echo ${current_namespace}
export SAS_CLI_PROFILE=${current_namespace}
./sas-viya --profile ${SAS_CLI_PROFILE} profile set-endpoint "${INGRESS_URL}"
./sas-viya --profile ${SAS_CLI_PROFILE} profile toggle-color off
./sas-viya --profile ${SAS_CLI_PROFILE} profile set-output fulljson

# login and create a token
./sas-viya --profile ${SAS_CLI_PROFILE} auth login -user sasadm -password *******

By default, your authentication remains active for 12 hours. You can use the sas-viya batch auth logout command to sign out.

Run a .sas program

Create a 1.sas in the /tmp folder, for example:

echo 'proc setinit; run; quit;' | tee /tmp/1.sas

Submit the program for execution:

cd /tmp/
/opt/sas/viya/home/bin/sas-admin --profile ${SAS_CLI_PROFILE} batch jobs submit-pgm --pgm-path /tmp/1.sas --context default  --watch-output --wait-log-list --results-dir /tmp

.

Run a cas statement

In SAS Environment Manager, find out your CAS controller host and port.

bt_1_Run-a-sas-program-in-SAS-Viya-4-EV-CAS-host-port-1024x172.png

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

Create a 2.sas program and submit it for execution:

echo 'cas casauto host="controller.sas-cas-server-default.gelcorp.svc.cluster.local" port=5570;' | tee /tmp/2.sas
echo 'cas casauto terminate;' | tee -a /tmp/2.sas
cat /tmp/2.sas

Submit it for execution:

cd /tmp/
/opt/sas/viya/home/bin/sas-viya --profile ${SAS_CLI_PROFILE} batch jobs submit-pgm --pgm-path /tmp/2.sas --context default  --watch-output --wait-log-list --results-dir /tmp

The log would look like this:

>>> File set created, ID="JOB_20201009_045619_929_1"
>>>   Uploading "2.sas"
>>> Job submitted, ID="64139ca3-84aa-4d6c-bed8-d7c25a3647fd".
>>> Waiting for job to complete...
>>> Job started.
….
1          cas casauto host="controller.sas-cas-server-default.gelcorp.svc.cluster.local" port=5570;
NOTE: The session CASAUTO connected successfully to Cloud Analytic Services
      controller.sas-cas-server-default.gelcorp.svc.cluster.local using port 5570. The UUID is
      fc406c5b-3ec4-914c-97b2-48c99121eb1c. The user is sasadm and the active caslib is CASUSER(sasadm).
NOTE: The SAS System used:
      real time           0.40 seconds
      cpu time            0.06 seconds
…
<<< Job ended.
<<< Downloading files in fileset to "/tmp/JOB_20201009_045619_929_1".
<<<   Downloading "2.log"
<<<   Downloading "2.sas"
<<< Copying log and list files from "/tmp/JOB_20201009_045619_929_1" to current directory
<<<   Copying "2.log"
<<< Deleting results directory.
<<< Deleting batch job.
<<< Deleting fileset.

The sas-viya batch CLI seems to function more like the RSUBMIT statement from SAS/Connect. Only the tasks executed are not in a "server" session, but in a Kubernetes cluster, while the results and the output are delivered to the "client" session.

Options used

--profile uses the token you obtained, and the preferences defined earlier.
--context specifies the batch context in which the program is run.
--pgm-path the path to the program on the Kubernetes node machine. The specified program is uploaded to the file set so that it can be downloaded to the job directory and accessed on the execution pod.
--watch-output prints the execution of the program.
--wait-log-list waits until the job has completed running (either successfully or with an error).
--results-dir specifies the directory to store the results of commands or SAS programs if you also specified a --wait option.

More details on how to Submit SAS Programs in Batch.

Pre-Requisites

SAS Viya CLI installed

To perform the batch commands, you must have the SAS Viya CLI installed on the machine where you will execute the commands. Gerry Nelson provided more context in his post Keeping the SAS Administration Command-Line interfaces up-to-date. His post talks about the sas-admin executable, which has been replaced by sas-viya for SAS Viya release 2020.1 and later. Replace sas-admin with sas-viya in his instructions and you should be fine.

pyviyatools installed

To authenticate with an .authinfo file you need the pyviyatools. The pyviatools installation instructions are found here: INSTALL.md .

Think security

A more secure way is to avoid encoding credentials. You can use an .authinfo file in conjunction with the pyviya tools. See Gerry's post: Introducing the GEL pyviyatools.

# Instead of
# ./sas-viya --profile ${SAS_CLI_PROFILE} auth login -user sasadm -password *****

# Create .authinfo - replace ***** with real credentials
tee ~/.authinfo > /dev/null << EOF
default user sasadm password *****
EOF
 
chmod 600 ~/.authinfo
 
# Browse to PyViya Tools folder and use the loginviauthinfo command for authentification
cd ~/admin/pyviyatools
./loginviauthinfo.py -f ~/.authinfo

The pyviatools installation instructions are found here: INSTALL.md .

Conclusions

To execute sas programs in batch in the new SAS Viya (2020.1 and later), consider using the sas-viya batch CLI. It is not the only option, there are other CLIs that allow you to manage, transfer, automate. What if you have parameters to pass to a SAS program running in batch? Read How to Run SAS Programs with Parameters in Batch in SAS Viya. What if you need to run a SAS Studio flow instead of a SAS program? Read How to Run a SAS Studio Flow in Batch.

Acknowledgements

Full credits and many thanks to: Ajmal Farzam, Gerry Nelson, Mark Thomas and Stephen Foerster.

References

Thank you for your time reading this article. If you think the article was useful, like it.

Please comment and share your experience with SAS Viya and help others.

Find more articles from SAS Global Enablement and Learning here.

SASKiwi · ‎04-08-2021

@Bogdan_Teleuca - This is very informative. Is there a GUI that automates the command line interface in the same way Schedule Manager in SAS Management Console automates the SAS 9 batch job command line interface? SAS Workflow Scheduler perhaps?

Bogdan_Teleuca · ‎04-13-2021

I don't think there is one.

SASKiwi · ‎04-13-2021

@Bogdan_Teleuca - Are there any plans to provide one?

MarkBodt_NZ · ‎04-14-2021

Interesting article. Only two weeks ago I had to come up with some solutions to batch submit jobs in Viya 3.5 from Control-M as part of the DI ETL processing and subsequent load into CAS.

I came up with two solutions.

- The first being a shell script to run a SPRE session in batch as described in the article above.

- The second more interesting one was to use use REST calls. I came up with a working proof of concept where by from a SAS9.4 DI Server using proc http, I made two calls:1) obtain an auth token : https://xxxx.xxxx.x.nz/SASLogon/oauth/token, then followed this with a call to the SAS Viya job execution service. I believe that Control-M can make REST calls (not tested), but otherwise Control-M could be used to call a SAS9.4 job and SAS9.4 can make the REST calls.

Part of the code for the second REST call to kick off the job:

proc http url="&viyahostname/jobExecution/jobs"

method="POST"

in=json_in

out=out;

Headers "Content-Type"= "application/vnd.sas.job.execution.job.request+json"

'Accept'='application/vnd.sas.job.execution.job+json'

'Authorization'="Bearer &access_token"

;

run;

Job information and code is passed in the json_in file and included code, as well as parameters that you could pass. These parameters would appear as macro variables in the SAS session for example (json fragment):

"testparm": "test456",

"testparmB": "test789"

in the SAS Session - shows the parameters appearing as macro variables:

18 +%put _all_;

GLOBAL DTSTAMP 2021_03_18T22_53_23

GLOBAL SYS_COMPUTE_DATA /opt/sas/viya/config/data/compsrv/default

GLOBAL SYS_COMPUTE_JOB_ID 89EF29E8-ECCD-6C48-802D-D2146A658913

GLOBAL SYS_COMPUTE_SESSION_ID 259db425-7750-4c90-8d25-f47e5cb8fcf8-ses0000

GLOBAL SYS_COMPUTE_SESSION_OWNER mark

GLOBAL SYS_JES_JOB_URI %nrstr(/jobExecution/jobs/be57e9e9-e7b1-4370-8e54-14297f3997af)

GLOBAL TESTPARM test456

GLOBAL TESTPARMB test789

GLOBAL _CASPORT_ 5570

GLOBAL _CONTEXTNAME %nrstr

I kept the SAS code in the job definition simple - being a couple of statements to do a %include. The included .sas file could then contain more extensive code - in this case CAS table load, save, promote etc. code.

The job would display in the Environment Manager - Jobs screen. Like this one (from our Viya 3.4 AWS playpen):

This REST call solution meant that the batch jobs could be called from SAS9 and therefore be made into a SAS9 DI Job and be scheduled using the usual SAS9 scheduling tools.

In this particular application, the existing SAS DI environment was an older - pre M5 release which meant that integration with Viya was limited. We couldn't use proc casutil or proc cas, for example to load data into CAS as part of the nightly ETL processing. Using proc http and REST calls meant that we could kick off a job on Viya that would load data into CAS after the nightly ETL batch had completed.

To develop this, I used simple CURL statements and Postman to test the REST calls

Mark

pritesh_sas · ‎06-09-2021

Thank you BogdanT and MarkBodt_NewZealand .Very informative.

Quentin · ‎06-09-2021

Sorry for a naive question; I don't have access to a Viya environment. The above seems like a good bit of work just to batch submit. For a casual Viya batch submit, is the SAS Studio background submit a reasonable option? Obviously it's not CLI, but for most of my use cases, I'd be happy to manually open a program and click a "batch submit" button, as long as I got a logical batch submit (i.e. starts a new clean session, executes the program, and writes a .log file and .lst file).

SASKiwi · ‎06-09-2021

I see on a Viya 4 presentation roadmap that job scheduling equivalent to Platform Services / LSF is due in Viya 2021.2. Is this likely to fix the missing graphical user interface issue?