BookmarkSubscribeRSS Feed

How to Run a SAS Studio Flow in Batch

Started ‎05-05-2021 by
Modified ‎06-05-2022 by
Views 9,562

If you ever wondered how to run a SAS Studio Flow in batch, this article provides you the approach and the scripts to:

  • Create the SAS Studio flow
  • Schedule the SAS Studio flow
  • Execute the job request scripts
  • Manage changes

Assumptions

Gerry Nelson provides more context in his post Keeping the SAS Administration Command-Line interfaces up-to-date

His post talks about the sas-admin executable, which has been replaced by sas-viya for SAS Viya release 2020.1 and later. Replace sas-admin with sas-viya in his  instructions and you should be fine.

 

To perform the batch commands, you must have the SAS Viya CLI installed on the machine where you will execute the commands.

The installation instructions can be found here: SAS® Viya®: Using the Command-Line Interface.

 

The top-level command sas-viya is used to initialize, authenticate the user and execute the job plug-in.

For a complete plug-in list, see Command-Line Interface: Plug-Ins.

 

 Back to the SAS Studio flows:

Create the SAS Studio flow

What is a flow? A flow is a sequence of operations on data. Flows are available in SAS Studio as of SAS Viya release 2020.1.

If you want to know more, please see SAS® Studio with SAS® and SAS® Viya® Programming Documentation / SAS® Studio Flows.

My colleagues will post more content about flows and how to use them in the coming weeks and months.

 

In the following example, the flow queries a data set, creates a result table and loads it in a CASLIB.

 

bt_1_SAS_Studio_Flow_example.png

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

 

 

 

Schedule the SAS Studio flow

You can schedule the flow from the interface. Pick a time in the future and choose just one occurrence. At this stage, the schedule transforms the flow in a job request.

 

bt_2_SAS_Studio_Flow_schedule_as_a_job-1024x436.png

 

 

After, you could check in SAS Environment Manager > Jobs and Flows > Scheduling the job request and the id. This id is key to running the flow in batch.

 

bt_3_SAS_Studio_Flow_scheduled_as_a_job_in_SAS_Environment_Manager-1024x419.png

 

 

While you can run and schedule the flow from the interface, I will focus on how to run a SAS Studio Flow using scripts and using the SAS Viya CLI mentioned earlier.

Execute the job request

The job request contains the SAS Studio flow code. Log-in the terminal as a Kubernetes administrator and:

 

Authenticate to SAS Viya and create a profile

 

# get namespace and host, adapt the value below to your environment
current_namespace=gelenv
INGRESS_SUFFIX=$(hostname -f)
INGRESS_URL=https://${current_namespace}.${INGRESS_SUFFIX}
echo ${INGRESS_URL}
 
# get certificates (TLS support)
## run just once
mkdir -p ~/.certs
kubectl cp $(kubectl get pod -l app=sas-logon-app -o=jsonpath='{.items[0].metadata.name}'):security/trustedcerts.pem ~/.certs/${current_namespace}_trustedcerts.pem
## run as many times as you authenticate export SSL_CERT_FILE=~/.certs/${current_namespace}_trustedcerts.pem export REQUESTS_CA_BUNDLE=${SSL_CERT_FILE} # go to sas-viya CLI folder clidir=/opt/sas/viya/home/bin cd $clidir # create a profile export SAS_CLI_PROFILE=${current_namespace} ./sas-viya --profile ${SAS_CLI_PROFILE} profile set-endpoint "${INGRESS_URL}" ./sas-viya --profile ${SAS_CLI_PROFILE} profile toggle-color off ./sas-viya --profile ${SAS_CLI_PROFILE} profile set-output fulljson # Login and get a token, change the password ./sas-viya --profile ${SAS_CLI_PROFILE} auth login -user sasadm -password *****

Output example:

Login successful. Token saved..png

 

There is a more elegant and secure approach to login and get a token, I have described it at the end.

 

List the existing job requests

The approach is to filter by name the jobs, then retrieve the id in a variable, jid.

 

# List the existing job requests
/opt/sas/viya/home/bin/sas-viya -output text job requests list -limit 25
 
# Read the id of the created job, filter by name
jid=$( /opt/sas/viya/home/bin/sas-viya --output json job requests list --filter 'eq(name,"CarMakeFlow.flw")' | jq -r '.items[]["id"]')
echo $jid

 

bt_4_List_job_requests_use_SAS_Viya_CLI.png

 

 

Execute the job request

 

/opt/sas/viya/home/bin/sas-viya job requests execute --id $jid

 

BINGO, the SAS Studio flow is executed. You can check the last execution update time in SAS Environment Manager > Jobs and Flows > Scheduling.

 

Managing change

What happens if you change something in the SAS Studio flow and save it? Will the job request, update itself with the new flow code? The answer is no. You will have to re-schedule the flow as a new job request. Now you will have two jobs with the same name, but different ids.

 

bt_5_List_job_requests__after_reschedule_use_SAS_Viya_CLI-1024x133.png

 

You have to delete the old job, supposing you no longer need that version and execute the new one.

 

# Delete previous job using old jid
/opt/sas/viya/home/bin/sas-viya -output text job requests delete --id $jid

## Relist the jobs
/opt/sas/viya/home/bin/sas-viya -output text job requests list -limit 25

# Read the id of the new job, filter by name
jid=$( /opt/sas/viya/home/bin/sas-viya --output json job requests list --filter 'eq(name,"CarMakeFlow.flw")' | jq -r '.items[]["id"]')
echo $jid

# Execute the job request based on the id of the job
/opt/sas/viya/home/bin/sas-viya job requests execute --id $jid

 

Think security

A more secure way is to avoid encoding credentials. You can use an .authinfo file in conjunction with the pyviya tools. See Gerry's post: Introducing the GEL pyviyatools.

 

# Instead of
# ./sas-viya --profile ${SAS_CLI_PROFILE} auth login -user sasadm -password *****

# Create .authinfo - replace ***** with real credentials
tee ~/.authinfo > /dev/null << EOF
default user sasadm password *****
EOF
 
chmod 600 ~/.authinfo
 
# Browse to PyViya Tools folder and use the loginviauthinfo command for authentification
cd ~/admin/pyviyatools
./loginviauthinfo.py -f ~/.authinfo

The pyviatools installation instructions are found here: INSTALL.md .

 

Conclusions

To execute SAS Studio Flows in SAS Viya 2020.1 or later, consider using the sas-viya batch CLI.

 

Acknowledgements

Many thanks to: Gerry Nelson and Mary Kathryn Queen.

 

Resources

For more information, see the following information:

Thank you for your time reading this article. If you liked the article, give it a thumbs up. Please comment and tell us what you think about running flows in batch.

 

Find more articles from SAS Global Enablement and Learning here.

Comments

Thanks. This seems like a lot, just to batch submit a flow. You mention “While you can run and schedule the flow from the interface, I will focus on how to run a SAS Studio Flow using scripts and using the SAS Viya CLI mentioned earlier.” What are the benefits to using the CLI to execute a flow rather than the Studio interface?  On a BI server, between SAS Management Console and LSF Flow Manager, I could do everything I wanted for scheduling and executing flows through reasonably friendly visual interfaces. Is the Studio interface more limited, thus requiring use of CLI?  I’m still on EG, which never had a batch submit feature. But I heard studio has better batch submit buttons built in??

Hi Quentin,

The architecture in SAS Viya 2020.1 and later is very different from SAS 9.4. The authentication, the security, etc.

 

If you need to execute SAS Studio flows in batch, in a CI/CD process, schedule the execution from outside SAS then SAS Viya CLI is facilitating your interaction with the Kubernetes cluster where SAS Viya is running.

You could also use REST APIs, but there's more code to write. The SAS Viya CLI is a wrapper on top of the REST APIs.

 

It may look like much code, but it can be scripted. If if can be scripted, it can be automated. There is nothing more annoying than a button that you need to keep on pushing.

If you ever need to integrate flows from SAS Viya in a broader enterprise flow, then the SAS Viya CLI is the way to do it.

Gotcha, thanks.  Hopefully I'll be playing in Viya before too long.  I do hope there is a batch submit button in Studio, as this was one of the top-voted EG enhancement ballot requests for years.  As I'm not worried about incorporating my flows in other non-SAS flows, as a SAS developer (not admin) I'm mostly focused on SAS giving me an easy way to batch submit (so that I can make sure my job runs in a clean session, and I get a log file), and a reasonable way to schedule jobs.  On non-BI linux server, batch submitting was of course easy, and scheduling was just crontab.  On BI server, both of these became a bit more of a headache, to schedule a SAS program to run I had to write the program, then put it in a DI job, then deploy the job for scheduling, blah blah.  I'm just hoping the Viya interfaces for batch submit / scheduling won't be much more complex than EBI.  From the "Jobs and Flows" screenshot in your article, looks similar.

What is the best way to diagnose/catch errors and review logs from the batch job and/or programs run as part of the flow?  Would the information be in SAS Environment Manager > Jobs and Flows?  What roles would be necessary to access?  I'm assuming the logs wouldn't be accessible on disk?

@kp21 from the linked article, https://communities.sas.com/t5/SAS-Communities-Library/How-to-Run-SAS-Programs-in-Batch-in-the-New-S... , it looks like you can retrieve the logs from a batch submit. If I'm reading correctly.  (If you couldn't get the logs, could you really call it a batch submit? : )

Version history
Last update:
‎06-05-2022 08:45 PM
Updated by:

SAS Innovate 2025: Call for Content

Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!

Submit your idea!

Free course: Data Literacy Essentials

Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning  and boost your career prospects.

Get Started

Article Tags