BookmarkSubscribeRSS Feed

Go with the Job Flow in SAS Viya 3.5

Started ‎12-19-2019 by
Modified ‎06-02-2022 by
Views 12,365

The Job Flows in SAS Viya 3.5 are useful when you want to create a chain of actions composed of: code (SAS or other languages), scripts (bash, shell, etc.), user interface actions (import files and tables with SAS Data Explorer), run data plans (SAS Data Studio). You can add logic gates (OR, AND), create time-based triggers or schedule a flow.

Job Flows in 60 seconds

In LEGO visual language, you want to go from left to right:

 

SAS-Viya-3.5-job-flow-vs-jobs-1024x502.png

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

The What

A job flow is a group of jobs, dependencies, and conditions that are organized in a sequence and can be scheduled or executed as a group.

 

Flows contain objects (such jobs, gates, and events) and connections (the order in which objects are evaluated in the flow). A few examples:

 

SAS-Viya-3.5-job-flow-dependencies.png

Jobs with dependencies

 

SAS-Viya-3.5-job-flow-logical-gates-AND-OR.png

Jobs with logical gates

 

SAS-Viya-3.5-job-flow-command-line-actions-bash-1024x288.png

Jobs with command line actions

 

Pre-requisites: You need a SAS Job Flow Scheduler license (SAS Viya 3.5).

Have more than 60 seconds?

Jobs: The Building Bricks

For job flows you need jobs. These are your building bricks, like in Lego. Suppose you have the following:

Job A: loads JPEG images in a CAS table

Fails if the target CAS table already exists.

 

SAS-Viya-3.5-job-flow-job-A-1024x417.png

Job B: loads a DICOM series (biomedical images) in a CAS table

DICOM series: a patient study consists of 1..n series. A series generally equates to a specific type (modality) of data, or the position of a patient on the acquisition device. The job replaces the target CAS table. Depends on A.

 

SAS-Viya-3.5-job-flow-job-B-1024x468.png

Job C: calls a SAS program to define an AWS S3 caslib

In the documentation this is called: create a job from a SAS DATA step program.

 

CAS mySession host="intcas01.race.sas.com" port=5570
SESSOPTS=(CASLIB=casuser TIMEOUT=999 LOCALE="en_US" metrics=true);
filename Job_C FILESRVC folderpath='/gelcontent/Demo/DM/Work in Progress' 
filename='Job_C_assign_AWSS3_caslib.sas';
%include Job_C / source;
cas mySession Terminate;

 

The job calls a program ‘Job_C_assign_AWSS3_caslib.sas’:

 

caslib AWSS3 datasource=(srctype="s3"
               accesskeyid= "AKIAY7OMEHNKPUVMH3I.."
               secretaccesskey= "1PWGBocdYPPSSFTB…" 
               region="AsiaPacificSydney"
               bucket="gelsas"
               usessl=true) global;

Job D loads a file from the AWSS3 caslib in a CAS table

Replaces the target CAS table if it already exists. Depends on C.

 

SAS-Viya-3.5-job-flow-job-D-1024x604.png

Job E: runs a data plan

The job calculates derived columns in a CAS table. Depends on D.

 

SAS-Viya-3.5-job-flow-job-E-1024x446.png

Job Recap

The building bricks in SAS Environment Manager > Scheduling will look like in the image below. By default, if you cannot see it here you cannot add it in a Job Flow. Exceptions: command line actions (read below).

 

SAS-Viya-3.5-job-flow-job-A-E-1024x537.png

Job Flow Examples

Simple flow

The simplest case is a sequence of chained jobs with simple dependencies. Note how several streams can run in parallel.

 

SAS-Viya-3.5-job-flow-simple-5-jobs-1024x223.png

 

Execute the job:

 

SAS-Viya-3.5-job-flow-simple-5-jobs-execution-600x220.png

 

Job A starts with C, B starts with D, E after D and so on. Job chains can start and run in parallel.

Job dependencies

Job A will fail as the import will be cancelled if the target CAS table exists. You can modify the job dependencies, the connections between the jobs. The available options:

  • Completes successfully
  • Ends with any exit code
  • Starts
  • Ends with exit code (you specify the exit code criteria)
  • Maximum run time (you specify the run time)
  • Minimum run time (you specify the run time)

To keep the flow running, edit the Job dependency and choose the event type: Ends with any exit code.

 

SAS-Viya-3.5-job-flow-dependencies.png

 

Execute the job:

 

SAS-Viya-3.5-job-flow-dependencies-execution-1024x371.png

 

The job flow execution is still successful (exit code 0). The flow will continue in case of job A failure (exit code 1).

 

The Monitoring tab can be used to view the execution history and you will find it useful to understand what the job flow does.

 

SAS-Viya-3.5-job-flow-monitoring-1024x690.png

Use logical gates

Logical gates are AND / OR conditions on the job execution status.

 

For example, execute job A or B, execute C, D. Execute job E only if D and (A or B) are successful. This is the moment when a photo is worth a hundred words:SAS-Viya-3.5-job-flow-logical-gates-AND-OR.png

 

Another example of a logical gate is the OR before an entire Job Flow is executed:

 

SAS-Viya-3.5-job-flow-logical-gates-OR-job-flow-1024x193.png

 

Yes, you can chain a job flow in another job flow.

 

You can also play with the job dependencies and logical gates to gain extra flexibility in execution.

Use command line actions

You can also use a command line action as a job in a flow:

 

For example, you can launch a SAS program in batch:

 

SAS-Viya-3.5-job-flow-command-line-actions-SAS-1024x254.png

 

/opt/sas/spre/home/SASFoundation/bin/sas_u8 
-sysin /tmp/jobflow_test/jobtest.sas -log /tmp/jobflow_test/jobtest.log

 

jobtest.sas is the program you want to execute.

 

Or, you can run a bash script:

 

SAS-Viya-3.5-job-flow-command-line-actions-bash-1024x288.png

 

/bin/bash -c '/tmp/jobflow_test/Job_G.sh'

The content of Job_G.sh can be something like:

/bin/bash -c 'ls -la /opt/sas/viya/home/bin/ >>
 /tmp/jobflow_test/Job_G.log'

 

Check the execution result (exit code = 0, successful):

 

SAS-Viya-3.5-job-flow-command-line-actions-execution-1024x319.png

 

Note: You cannot use the Monitoring tab for command line actions.

Time events

You can add time events, such as:

 

repeat a flow with a certain frequency:

 

SAS-Viya-3.5-job-flow-time-event-frequency-1024x495.png

 

start only at the time specified:

 

SAS-Viya-3.5-job-flow-time-event-start-1024x316.png

 

 

Saved Actions

You can save command-line action objects to the Saved Actions area and then use those objects in other flows. You cannot share between users the saved actions.

Scheduling job flows

Works in the exact same way as for jobs. See Mary Kathryn’s post: SAS Viya 3.4: Schedule SAS Studio 5.1 (Studio V) Program Files.

Future Developments

File based triggers

When I wrote this post, I received a message from Product Management: "file triggers are on the list of requirements for future enhancements and on the [product] roadmap".

Best practices: create a job from a SAS program

Spotted by my colleague Gilles.

 

If you create a job from a SAS program, for example J_A_CASLIB.sas:

 

SAS-Viya-3.5-job-flow-code-job-1-300x173.png

 

J_A_CASLIB.sas sample content: CASLIB IM_301 is created at job execution:

 

* define the PATH caslib on the images folder;
caslib IM_301 path="/gelcontent/demo/VSVDMML/images/Giraffe_Dolphin/Dolphin/"
type=path global;

Best Practices

Option 1: use a program, a "capsule", to call the code inside your program. Call this new program J_A_capsule.sas:

 

CAS mySession host="intcas01.race.sas.com" port=5570
SESSOPTS=(CASLIB=casuser TIMEOUT=999 LOCALE="en_US" metrics=true);
filename Job_A FILESRVC folderpath='/gelcontent/Demo/DM/Jobs'
filename='J_A_CASLIB.sas';
%include Job_A / source;
cas mySession Terminate;

 

Create Job_H from the code inside J_A_capsule.sas:

 

The advantage of this approach is that it will use the latest version of the program.

 

Option 2: use a command line action running a SAS code.

 

Conclusions

The Job Flow adds extra flexibility to the already existing Job scheduling and execution capabilities. The job dependencies, the logical gates and command line actions are a very useful addition to the job execution toolbox.

Resources

Want to Learn More about Viya 3.5?

Stay tuned for more stories. And please comment, share your experience with Job Flow and help others. Thank you for your time reading this post.

Comments

@Bogdan_Teleuca  - An excellent introduction to scheduling on Viya. I read somewhere that the SAS Job Flow Scheduler doesn't currently have the functionality to kill a running job. If so is that a feature being considered for a later release?

@SASKiwi Stop (kill) a running job is on Product Management features list, I am assured.

@Bogdan_Teleuca  - Good to know.

@Bogdan_Teleuca thanks for this really helpful article! If I have a job flow with 4 jobs, and 1-3 execute successfully, but job 4 fails, is there a way when the next time the flow runs to have it start with Job 4?

You can have Job Flow 4 running before Job 1 for example and play with the connection and event type to allow Job 1 to run if 4 fails.

Beautiful article!
Is it possible to manage the job frequency by considering the outcome of the job itself?
E.g. schedule a job every hour, but it should only start if the execution of the previous hour is completed.

Version history
Last update:
‎06-02-2022 12:18 AM
Updated by:
Contributors

sas-innovate-2024.png

 

Time is running out to save with the early bird rate. Register by Friday, March 1 for just $695 - $100 off the standard rate.

 

Check out the agenda and get ready for a jam-packed event featuring workshops, super demos, breakout sessions, roundtables, inspiring keynotes and incredible networking events. 

 

Register now!

Free course: Data Literacy Essentials

Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning  and boost your career prospects.

Get Started