Did my SAS batch job run successfully? Get status, return codes and output from SAS batch CLI

I was recently asked how to get the return code of a SAS batch job from the command line, to find out if it has succeeded or failed. The context was for something in a customer's existing workflow engine (a.k.a. a job flow scheduler), which in their case was not Argo Workflow or Apache Airflow, but rather something script-based. They wanted their flow scheduler to run the SAS batch job, and after that do one thing if the batch job succeeded, and depending on the particular job either do something else or stop the flow if the batch job failed.

For customers using Apache Airflow or Argo Workflow as their scheduler, this may be supported by existing, well described modules or methods of integrating them with SAS batch. There are links to Gerry Nelson's posts covering that in the references bit at the end. But for setting up SAS batch jobs in any other scheduler that uses a script-based wrapper for each step, what follows might save you some time.

In this post we'll walk through a set of short bash examples that submit SAS batch jobs, query their status and other properties and parse the JSON output produced by those queries to get the values you want. We also save the return code and download the results fileset containing the SAS log output, and other files.

You can use these example scripts as they are, or modify them to suit your particular needs. It is convenient and a good idea to wrap some of the repetitive parts of the examples that follow in a bash function, so we will see how to do that too.

For more on the SAS Viya CLI in general, and for how to get started using the batch plug-in, there are links to great posts and documentation about both in the references at the end. I won't duplicate their work in this post where I can avoid it. Let's get started.

Prerequisite environment set up and experience

In the examples which follow, I will assume you have already installed SAS Viya and the SAS Viya CLI., I will assume you have set up a default profile for the SAS Viya CLI, and have successfully authenticated through the CLI as a user who can submit batch jobs. If not, there are posts linked in the references section at the end of this post which will walk you through all of that, and of course the documentation covers this very well.

I also assume you know a bit about how to submit a SAS batch job from the SAS Viya CLI, and will not explain that in much detail - again, see the references section if that is new to you.

Since I will use the default profile, let's set up a bash environment variable to specify that option, and to provide a place where you could specify other options which you will always specify when you run the sas-viya batch submit-pgm command. Run this interactively in your bash shell, or add it to your own script:

 # Provide a place to set some sas batch jobs command options.
sas_batch_options="--context default"

The task of understanding the status and return code from a batch job is made much easier with a great little tool called jq (https://jqlang.github.io/jq/) which we always install in our administration clients, and which we use to parse JSON documents such as the output that the SAS Viya CLI can produce.

Sample SAS programs

To demonstrate we are getting the return codes from SAS batch jobs correctly, let's create three SAS programs that are variously supposed to run successfully, run with errors, and abort. They should finish with different return codes when we run them as SAS batch jobs. I'll create these in my home directory (sometimes aliased as ~), and will assume you have done the same.

good.sas:

data _null_;
run;

bad.sas:

data _null_;
  mistake;
run;

abort_abend.sas:

data _null_;
  abort abend;
run;

You can create these SAS programs any way you like. One way is to run this block of bash script, which creates the files in your home directory:

# Create a SAS program called good.sas which should run to completion with no errors
tee ~/good.sas > /dev/null << EOF
data _null_;
run;
EOF

# Create a SAS program called bad.sas which should fail with errors
tee ~/bad.sas > /dev/null << EOF
data _null_;
  mistake;
run;
EOF

# Create a SAS program called abort_abend.sas which should abort and abend
tee ~/abort_abend.sas > /dev/null << EOF
data _null_;
  abort abend;
run;
EOF

Let's also create one more program, borrowed from the SAS language documentation and from Bruno Mueller's post referenced below, which runs normally and produces some HTML output:

html.sas:

%let batchjobdir = %sysget(BATCHJOBDIR);
ods _all_ close;
ods html file="&batchjobdir/sashelp_weight_over_100.html";
title 'Student Weight';
proc print data=sashelp.class;
  where weight>100;
run;
ods html close;

Or, if you want to create it in a script or from the command line, run this:

# Create a SAS program called html.sas which should run to completion with no
# errors and create HTML output, to a file in the batch job directory, where
# it will be included in the fileset for the job as output.
tee ~/html.sas > /dev/null << EOF
%let batchjobdir = %sysget(BATCHJOBDIR);
ods _all_ close;
ods html file="&batchjobdir/sashelp_weight_over_100.html";
title 'Student Weight';
proc print data=sashelp.class; 
  where weight>100;
run;
ods html close;
EOF

Run a SAS program in batch

This SAS Viya CLI command will run the program good.sas as an asynchronous SAS batch job, meaning without waiting for the program to start or to finish. The ${sas_batch_options} part includes any options we defined earlier, and the --pgm-path parameter passes in the path to the SAS program file good.sas in your home directory:

sas-viya batch jobs submit-pgm ${sas_batch_options} --pgm-path ~/good.sas

The expected output is something like this:

>>> The file set "JOB_20241028_135406_321_1" was created.
>>> Uploading "good.sas".
>>> The job was submitted. ID: "043a625c-c8e7-4018-944f-85d787be0965" Workload Orchestrator job ID: "19"

When the command prompt returns, that doesn't necessarily mean the program has run, though in a SAS Viya deployment which is not busy, it might run within a few seconds. To find out if it has run, you need to query the batch job's status, and the command to do that takes the job ID as a parameter, like this:

batch_job_id="043a625c-c8e7-4018-944f-85d787be0965"
sas-viya batch jobs list --job-id ${batch_job_id} --details

If everything is working and you run that command after the batch job has finished, the output should look something like this:

{
    "items": [
        {
            "contextId": "e3c2bcde-49e7-4ed4-957c-b1b470e6ad32",
            "createdBy": "geladm",
            "creationTimeStamp": "2024-10-28T13:54:07.88399Z",
            "endedTimeStamp": "2024-10-28T13:54:15Z",
            "executionHost": "10.42.0.67",
            "fileSetId": "JOB_20241028_135406_321_1",
            "id": "043a625c-c8e7-4018-944f-85d787be0965",
            "modifiedBy": "anonymous",
            "modifiedTimeStamp": "2024-10-28T13:54:15.848773Z",
            "name": "good",
            "processId": "397b12ca-a280-4b73-af74-077cad34bf03",
            "returnCode": 0,
            "startedTimeStamp": "2024-10-28T13:54:07Z",
            "state": "completed",
            "submittedTimeStamp": "2024-10-28T13:54:07Z",
            "workloadJobId": "19"
        }
    ]
}

You can perhaps see the state of the job (“completed”), and its return code (0) in that JSON output. So this is a good time for us to use jq to extract those values, and some others, and store them in variables for subsequent use in messages or decision logic.

If you pass a JSON document like the one above to the jq command which follows, it will find the first entry in the [items] array, and return the workloadJobID from that entry:

jq -r '.items[0]["workloadJobId"]'

The -r parameter (alternatively, --raw-output) writes the output string directly rather than as a JSON string with quotes.

We are ready to put this all together.

Store the status details of a SAS program in variables

The series of commands which follow will run the same sas-viya batch jobs list command as above, and redirect the output to a temporary file. The next few commands will each pass that file in to jq to have it select the values we are interested in from the JSON document.

In this example I've added a few more values to the set we retrieve from those details, in addition to the state and return code:

sas-viya batch jobs list --job-id ${batch_job_id} --details > /tmp/batch_job_details.txt

batch_job_workload_job_id=$(cat /tmp/batch_job_details.txt | jq -r '.items[0]["workloadJobId"]')
batch_job_workload_job_name=$(cat /tmp/batch_job_details.txt | jq -r '.items[0]["name"]')
batch_job_state=$(cat /tmp/batch_job_details.txt | jq -r '.items[0]["state"]')
batch_job_return_code=$(cat /tmp/batch_job_details.txt | jq -r '.items[0]["returnCode"]')
batch_job_fileset_id=$(cat /tmp/batch_job_details.txt | jq -r '.items[0]["fileSetId"]')

Now we have those values about the state of the batch job, we can do whatever we wish with them. For example, we can print a message to the console (stdout) displaying the values to show it worked so far:

echo "Workload Job ID: \"${batch_job_workload_job_id}\" Name: \"${batch_job_workload_job_name}\" State: \"${batch_job_state}\" returnCode: \"${batch_job_return_code}\" fileSetId: \"${batch_job_fileset_id}\" "

Expected output is something like this:

Workload Job ID: "19" Name: "good" State: "completed" returnCode: "0" fileSetId: "JOB_20241028_135406_321_1"

Define functions to get the state of SAS batch jobs

So far so good, but running those commands interactively every time is tedious, so we will define some bash functions that will help us reduce repetition, and streamline the process of reporting on batch job state:

# Define a bash function to echo (print to console) the status details of a batch job
# Usage: echo_batch_job_state ${batch_job_id}
function echo_batch_job_state ()
{
    # sas-viya batch jobs list --job-id ${batch_job_id} --details | tee /tmp/batch_job_details.txt
    sas-viya batch jobs list --job-id ${batch_job_id} --details > /tmp/batch_job_details.txt
    batch_job_workload_job_id=$(cat /tmp/batch_job_details.txt | jq -r '.items[0]["workloadJobId"]')
    batch_job_workload_job_name=$(cat /tmp/batch_job_details.txt | jq -r '.items[0]["name"]')
    batch_job_state=$(cat /tmp/batch_job_details.txt | jq -r '.items[0]["state"]')
    batch_job_return_code=$(cat /tmp/batch_job_details.txt | jq -r '.items[0]["returnCode"]')
    batch_job_fileset_id=$(cat /tmp/batch_job_details.txt | jq -r '.items[0]["fileSetId"]')
    echo "Workload Job ID: \"${batch_job_workload_job_id}\" Name: \"${batch_job_workload_job_name}\" State: \"${batch_job_state}\" returnCode: \"${batch_job_return_code}\" fileSetId: \"${batch_job_fileset_id}\" "
}

# Define a bash function to echo the return code of a batch job
function get_sasbatch_rc ()
{
    sas-viya batch jobs list --job-id ${batch_job_id} --details > /tmp/batch_job_details.txt
    batch_job_return_code=$(cat /tmp/batch_job_details.txt | jq -r '.items[0]["returnCode"]')
    echo ${batch_job_return_code}
}

# Define a bash function to echo the fileset ID associated with a batch job
function get_sasbatch_fileset_id ()
{
    sas-viya batch jobs list --job-id ${batch_job_id} --details > /tmp/batch_job_details.txt
    batch_job_fileset_id=$(cat /tmp/batch_job_details.txt | jq -r '.items[0]["fileSetId"]')
    echo ${batch_job_fileset_id}
}

Submit a SAS program asynchronously, and watch its progress

Now you can run this to submit a new batch job to run the SAS program good.sas again:

# Submit a good program as an asynchonous SAS batch job, and get the job ID
pgm_name=good
sas-viya batch jobs submit-pgm ${sas_batch_options} --pgm-path ~/${pgm_name}.sas | tee /tmp/batch_job_submission_response.txt
batch_job_id=$(cat /tmp/batch_job_submission_response.txt | grep "ID:" | awk '{print $7}' | tr -d \")

As soon as you have submitted the batch job, run this repeatedly, until the job has completed:

echo_batch_job_state ${batch_job_id}

If you are quick, you might be able to see in the output it produces that the job is first pending, then running, and then completed. Expected output from our custom bash function echo_batch_job_state() during the successive stages of the SAS batch job running might look like these examples:

Workload Job ID: "21" Name: "good" State: "pending" returnCode: "0" fileSetId: "JOB_20241028_144729_489_1"
Workload Job ID: "21" Name: "good" State: "running" returnCode: "0" fileSetId: "JOB_20241028_144729_489_1"
Workload Job ID: "21" Name: "good" State: "completed" returnCode: "0" fileSetId: "JOB_20241028_144729_489_1"

The returnCode value remains 0 when the program has completed, indicating it ran successfully.

Examples with non-zero return codes

Let's run the next two programs in a similar way and see how the output of echo_batch_job_state() differs.

bad.sas

Here is how it looks with the intentionally 'bad' program:

# Submit a good program as an asynchonous SAS batch job, and get the job ID
pgm_name=bad
sas-viya batch jobs submit-pgm ${sas_batch_options} --pgm-path ~/${pgm_name}.sas | tee /tmp/batch_job_submission_response.txt
batch_job_id=$(cat /tmp/batch_job_submission_response.txt | grep "ID:" | awk '{print $7}' | tr -d \")

As soon as you have submitted the batch job, run this repeatedly, until the job has completed:

echo_batch_job_state ${batch_job_id}

Output might look like this for bad.sas:

Workload Job ID: "23" Name: "bad" State: "pending" returnCode: "0" fileSetId: "JOB_20241028_150052_511_1"
Workload Job ID: "23" Name: "bad" State: "running" returnCode: "0" fileSetId: "JOB_20241028_150052_511_1"
Workload Job ID: "23" Name: "bad" State: "failed" returnCode: "2" fileSetId: "JOB_20241028_150052_511_1"

The return code of 2 indicates the job failed with errors. I think a return code of 1 is seen (or used to be, in earlier versions of SAS) after some kinds of SAS warning. But the handful of programs I tried which output warnings in their SAS logs, all had an exit code of 0 despite the warnings, which I recall meaning the warning is more for information and is not too serious. If you know better, or if you have an example of a SAS program which ends with an error code of 1, please let me know and I'll update this post to include it.

abort_abend.sas

Here is how it looks with the program that aborts and ends abruptly:

# Submit program which aborts and abends as an asynchonous SAS batch job, and get the job ID
pgm_name=abort_abend
sas-viya batch jobs submit-pgm ${sas_batch_options} --pgm-path ~/${pgm_name}.sas | tee /tmp/batch_job_submission_response.txt
batch_job_id=$(cat /tmp/batch_job_submission_response.txt | grep "ID:" | awk '{print $7}' | tr -d \")

As soon as you have submitted the batch job, run this repeatedly, until the job has completed:

echo_batch_job_state ${batch_job_id}

Output might look like this for abort_abend.sas:

Workload Job ID: "24" Name: "abort_abend" State: "pending" returnCode: "0" fileSetId: "JOB_20241028_150719_824_1"
Workload Job ID: "24" Name: "abort_abend" State: "running" returnCode: "0" fileSetId: "JOB_20241028_150719_824_1"
Workload Job ID: "24" Name: "abort_abend" State: "failed" returnCode: "5" fileSetId: "JOB_20241028_150719_824_1"

The return code of 5 indicates a more serious problem than just a syntax error, caused by the abort abend; statement in this SAS program.

Save the return code and output files produced by our SAS program

Seeing the status and return code on the console output is nice, but we can do more. First, we should store the return code of that batch job in a variable, like this:

batch_rc=$(get_sasbatch_rc ${batch_job_id})

That allows us to use the return code in decision logic. We can then download the fileset belonging to our batch job.

IMPORTANT: Downloading the fileset causes SAS Viya to stop listing the job and delete its fileset, so you can no longer get its return code (at least not as easily) after you download the fileset. Therefore, it is important to save the return code in a variable, e.g. using our get_sasbatch_rc() convenience function as shown above, BEFORE downloading the fileset.

The batch job fileset includes the .sas program file you submitted, any additional input files we passed in (there are none in these examples), the usual SAS program .log file and .lst output, and even additional files written to the batch job's fileset directory, as illustrated with the sample program html.sas above.

In this example, we download the fileset for the good.sas sample program to a directory below /tmp, named for the batch job fileset ID, and then we list the files in that directory to the console using ls:

# Download the results, i.e. the fileset - to a specific local directory.
sas-viya batch jobs get-results --job-id ${batch_job_id} --results-dir /tmp
# List the results files in that directory
ls -l /tmp/${batch_job_fileset_id}

Here's an example directory listing using the statements above, after running good.sas:

total 12
-rwxr--r-- 1 cloud-user cloud-user 1513 Oct 28 11:17 good.log
-rwxr--r-- 1 cloud-user cloud-user 18 Oct 28 11:17 good.sas
-rw------- 1 cloud-user cloud-user 363 Oct 28 11:17 job.info

You can then view the program log file using your preferred text editor. To show it worked, this command will show a few lines of the good.log file:

head /tmp/${batch_job_fileset_id}/good.log

Example output:

1 The SAS System Monday, October 28, 2024 03:17:00 PM

NOTE: Copyright (c) 2016-2023 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software V.04.00 (TS M0 MBCS3170)
Licensed to (SIMPLE) THIS ORDER IS FOR SAS INTERNAL USE ONLY, Site 70180938.
NOTE: This session is executing on the Linux 3.10.0-1062.12.1.el7.x86_64 (LIN X64) platform.



NOTE: Additional host information:

html.sas

Now, let's run html.sas in the same way as before:

pgm_name=html
sas-viya batch jobs submit-pgm ${sas_batch_options} --pgm-path ~/${pgm_name}.sas | tee /tmp/batch_job_submission_response.txt
batch_job_id=$(cat /tmp/batch_job_submission_response.txt | grep "ID:" | awk '{print $7}' | tr -d \")

If you like, run this repeatedly, to watch the batch job progress through the pending and running states until it reaches the completed state:

echo_batch_job_state ${batch_job_id}

When it is completed, save the return code (remember, you can't do this after you download the fileset), and then download the fileset:

# Save the return code of the last batch job - this will no longer be
# available after we download the results of that job
batch_rc=$(get_sasbatch_rc ${batch_job_id})

# Download the results, i.e. the fileset - to a specific local directory.
sas-viya batch jobs get-results --job-id ${batch_job_id} --results-dir /tmp
# List the results files in that directory
ls -l /tmp/${batch_job_fileset_id}

Expected output - notice we have the log file and also the HTML output file that our html.sas program creates, called sashelp_weight_over_100.html:

-rwxr--r-- 1 cloud-user cloud-user 2012 Oct 28 11:41 html.log
-rwxr--r-- 1 cloud-user cloud-user 214 Oct 28 11:41 html.sas
-rw------- 1 cloud-user cloud-user 363 Oct 28 11:41 job.info
-rwxr--r-- 1 cloud-user cloud-user 37370 Oct 28 11:41 sashelp_weight_over_100.html

We can then view the first few lines of the HTML file as text:

head /tmp/${batch_job_fileset_id}/sashelp_weight_over_100.html

Expected output – this certainly looks like HTML generated by SAS code:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta name="Generator" content="SAS Software Version V.04.00, see www.sas.com">
<meta http-equiv="Content-type" content="text/html; charset=utf-8">
<title>SAS Output</title>
<style type="text/css">
<!--
.accessiblecaption
{

Here's a screenshot of the (full) HTML file opened in a web browser:

HTML output produced by example program html.sas.

Select any image to see a larger version.

Mobile users: To view the images, select the "Full" version at the bottom of the page.

Use the return code for decision logic

You could use the return code to drive a decision step in your workflow, about whether to continue with the next step, or which step to run next. Here is an example using more bash shell statements - though again, I haven't managed to get an example SAS program which actually does result in a return code of 1, but I think this illustrates the idea. You will want to write something similar that better suits your own requirements:

# If the return code is 0, list the first lines of the output html file
if [ "${batch_rc}" == "0" ]; then
    echo "The batch job succeeded."
    echo
    echo "Here are the first few lines of the output html file:"
    head /tmp/${batch_job_fileset_id}/sashelp_weight_over_100.html
elif [ "${batch_rc}" == "1" ]; then
    echo "The batch job finished with important warnings."
elif [ "${batch_rc}" == "2" ]; then
    echo "The batch job finished with errors."
else
    echo "The batch job finished with return code of ${batch_rc}."
fi

Get the return code from synchronous batch jobs (that specify --wait)

When you submit a batch job with the --wait parameter, then instead of returning control to the command line or continuing to the next step in your script when the batch job is submitted, the sas-viya batch jobs submit-pgm command waits for the batch job to finish running.

The method for getting the return code from a SAS batch job run synchronously with --wait is similar to the method getting it from an asynchronous batch job. Here's an example, which needs little explanation if you have read this far:

# Submit a program which produces HTML output as a SAS batch job, wait for it to complete, and get the status and return code
pgm_name=html
sas-viya batch jobs submit-pgm ${sas_batch_options} --pgm-path ~/${pgm_name}.sas --wait | tee /tmp/batch_job_output.txt
batch_job_id=$(cat /tmp/batch_job_output.txt | grep "ID:" | awk '{print $7}' | tr -d \")
echo_batch_job_state ${batch_job_id}

##########################################
## Download the batch job logs (once only)
##########################################

# Get details of the job's output fileset
batch_job_fileset_id=$(get_sasbatch_fileset_id ${batch_job_id})
echo $batch_job_fileset_id
# List files in this job's fileset
sas-viya --output text batch filesets list-files --id ${batch_job_fileset_id}

# Save the return code of the last batch job - this will no longer be
# available after we download the results of that job
batch_rc=$(get_sasbatch_rc ${batch_job_id})

# Download the results, i.e. the fileset - to a specific local directory.
sas-viya batch jobs get-results --job-id ${batch_job_id} --results-dir /tmp
# List the results files in that directory
ls -l /tmp/${batch_job_fileset_id}

# If the return code is 0, list the first lines of the output html file
if [ "${batch_rc}" == "0" ]; then
    echo "The batch job succeeded."
    echo
    echo "Here are the first few lines of the output html file:"
    head /tmp/${batch_job_fileset_id}/sashelp_weight_over_100.html
elif [ "${batch_rc}" == "1" ]; then
    echo "The batch job finished with important warnings."
elif [ "${batch_rc}" == "2" ]; then
    echo "The batch job finished with errors."
else
    echo "The batch job finished with return code of ${batch_rc}."
fi

We have demonstrated how to run SAS programs as batch jobs, asynchronously and synchronously, how to monitor their progress, and how to get both their returnCode indicating success or failure, and their log files, all from the command line using shell script.

This post would have been much more difficult to write without the great SAS documentation, and the posts on closely related topics written by Gerry Nelson, Bogdan Teluca and Bruno Mueller. My thanks to all three for their work, referenced below.

References and resources

SAS Help Center documentation references:

SAS Viya Command-Line Interface documentation: Command-Line Interface: Overview, Command-Line Interface: Instructions When Running the CLI from a Container
Documentation for the SAS Batch Command-Line Interface specifically: SAS Batch CLI
Using the Output Delivery System (ODS) to create HTML and other formats of output in SAS code: Getting Started with the Output Delivery System

Gerry Nelson wrote a couple of posts on installing and configuring the SAS Viya Command-Line Interface natively or in a container, for any of its many use cases (not specifically just SAS batch jobs):

SAS Viya Command-Line Interface for Administration (update), in which Gerry explains the overall installation and initial configuration of the SAS Viya CLI and gives some simple examples of administrative tasks you can use it for in general.
Overview of installing and using the SAS Viya containerized CLI with SAS Viya LTI 2024.03 and later, Gerry explains how to download and use the SAS Viya CLI in a pre-built container image from SAS, which simplifies automation, and reduces the need for pre-requisite components. He also shows how to build your own custom docker image, using the SAS supplied image as a starting point and adding your own tools to it - he adds git and our pyviyatools project as an example.

Bogdan Teleuca has written an introductory post for readers who are new to the SAS batch command line:

In How to Run SAS Programs in Batch in the New SAS Viya Bodgan explains the basics of running SAS batch jobs from the SAS Viya Command-Line Interface (sas-viya CLI) in SAS Viya 2020.1 and later. He walks through configuring the SAS Viya CLI, setting up a profile for the CLI defining the REST endpoint that the CLI will connect to, where the TLS certificates are, preferred output options, and authenticating to SAS Viya as a specific user. He then shows how to run a .sas program synchronously through the SAS batch CLI, streaming the SAS log output to your console.

Bruno Mueller wrote this really nice post about sas-viya batch jobs, file sets and the SAS administrator, which is a great introduction to the topic, and from which I 'borrowed' some ideas.

And finally, if you are using Apache Airflow or Argo Workflow, I highly recommend these posts by Gerry Nelson:

See you next time!