Go with the Job Flow in SAS Viya 3.5

[#1]

The Job Flows in SAS Viya 3.5 are useful when you want to create a chain of actions composed of: code (SAS or other languages), scripts (bash, shell, etc.), user interface actions (import files and tables with SAS Data Explorer), run data plans (SAS Data Studio). You can add logic gates (OR, AND), create time-based triggers or schedule a flow.

Job Flows in 60 seconds

In LEGO visual language, you want to go from left to right:

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

The What

A job flow is a group of jobs, dependencies, and conditions that are organized in a sequence and can be scheduled or executed as a group.

Flows contain objects (such jobs, gates, and events) and connections (the order in which objects are evaluated in the flow). A few examples:

Jobs with dependencies

Jobs with logical gates

Jobs with command line actions

Pre-requisites: You need a SAS Job Flow Scheduler license (SAS Viya 3.5).

Have more than 60 seconds?

Jobs: The Building Bricks

For job flows you need jobs. These are your building bricks, like in Lego. Suppose you have the following:

Job A: loads JPEG images in a CAS table

Fails if the target CAS table already exists.

Job B: loads a DICOM series (biomedical images) in a CAS table

DICOM series: a patient study consists of 1..n series. A series generally equates to a specific type (modality) of data, or the position of a patient on the acquisition device. The job replaces the target CAS table. Depends on A.

Job C: calls a SAS program to define an AWS S3 caslib

In the documentation this is called: create a job from a SAS DATA step program.

CAS mySession host="intcas01.race.sas.com" port=5570
SESSOPTS=(CASLIB=casuser TIMEOUT=999 LOCALE="en_US" metrics=true);
filename Job_C FILESRVC folderpath='/gelcontent/Demo/DM/Work in Progress' 
filename='Job_C_assign_AWSS3_caslib.sas';
%include Job_C / source;
cas mySession Terminate;

The job calls a program ‘Job_C_assign_AWSS3_caslib.sas’:

caslib AWSS3 datasource=(srctype="s3"
               accesskeyid= "AKIAY7OMEHNKPUVMH3I.."
               secretaccesskey= "1PWGBocdYPPSSFTB…" 
               region="AsiaPacificSydney"
               bucket="gelsas"
               usessl=true) global;

Job D loads a file from the AWSS3 caslib in a CAS table

Replaces the target CAS table if it already exists. Depends on C.

Job E: runs a data plan

The job calculates derived columns in a CAS table. Depends on D.

Job Recap

The building bricks in SAS Environment Manager > Scheduling will look like in the image below. By default, if you cannot see it here you cannot add it in a Job Flow. Exceptions: command line actions (read below).

Job Flow Examples

Simple flow

The simplest case is a sequence of chained jobs with simple dependencies. Note how several streams can run in parallel.

Execute the job:

Job A starts with C, B starts with D, E after D and so on. Job chains can start and run in parallel.

Job dependencies

Job A will fail as the import will be cancelled if the target CAS table exists. You can modify the job dependencies, the connections between the jobs. The available options:

• Completes successfully
• Ends with any exit code
• Starts
• Ends with exit code (you specify the exit code criteria)
• Maximum run time (you specify the run time)
• Minimum run time (you specify the run time)

To keep the flow running, edit the Job dependency and choose the event type: Ends with any exit code.

Execute the job:

The job flow execution is still successful (exit code 0). The flow will continue in case of job A failure (exit code 1).

The Monitoring tab can be used to view the execution history and you will find it useful to understand what the job flow does.

Use logical gates

Logical gates are AND / OR conditions on the job execution status.

For example, execute job A or B, execute C, D. Execute job E only if D and (A or B) are successful. This is the moment when a photo is worth a hundred words:

Another example of a logical gate is the OR before an entire Job Flow is executed:

Yes, you can chain a job flow in another job flow.

You can also play with the job dependencies and logical gates to gain extra flexibility in execution.

Use command line actions

You can also use a command line action as a job in a flow:

For example, you can launch a SAS program in batch:

/opt/sas/spre/home/SASFoundation/bin/sas_u8 
-sysin /tmp/jobflow_test/jobtest.sas -log /tmp/jobflow_test/jobtest.log

jobtest.sas is the program you want to execute.

Or, you can run a bash script:

/bin/bash -c '/tmp/jobflow_test/Job_G.sh'

The content of Job_G.sh can be something like:

/bin/bash -c 'ls -la /opt/sas/viya/home/bin/ >>
 /tmp/jobflow_test/Job_G.log'

Check the execution result (exit code = 0, successful):

Note: You cannot use the Monitoring tab for command line actions.

Time events

You can add time events, such as:

repeat a flow with a certain frequency:

start only at the time specified:

Saved Actions

You can save command-line action objects to the Saved Actions area and then use those objects in other flows. You cannot share between users the saved actions.

Scheduling job flows

Works in the exact same way as for jobs. See Mary Kathryn’s post: SAS Viya 3.4: Schedule SAS Studio 5.1 (Studio V) Program Files.

Future Developments

File based triggers

When I wrote this post, I received a message from Product Management: "file triggers are on the list of requirements for future enhancements and on the [product] roadmap".

Best practices: create a job from a SAS program

Spotted by my colleague Gilles.

If you create a job from a SAS program, for example J_A_CASLIB.sas:

J_A_CASLIB.sas sample content: CASLIB IM_301 is created at job execution:

* define the PATH caslib on the images folder;
caslib IM_301 path="/gelcontent/demo/VSVDMML/images/Giraffe_Dolphin/Dolphin/"
type=path global;

Best Practices

Option 1: use a program, a "capsule", to call the code inside your program. Call this new program J_A_capsule.sas:

CAS mySession host="intcas01.race.sas.com" port=5570
SESSOPTS=(CASLIB=casuser TIMEOUT=999 LOCALE="en_US" metrics=true);
filename Job_A FILESRVC folderpath='/gelcontent/Demo/DM/Jobs'
filename='J_A_CASLIB.sas';
%include Job_A / source;
cas mySession Terminate;

Create Job_H from the code inside J_A_capsule.sas:

The advantage of this approach is that it will use the latest version of the program.

Option 2: use a command line action running a SAS code.

Conclusions

The Job Flow adds extra flexibility to the already existing Job scheduling and execution capabilities. The job dependencies, the logical gates and command line actions are a very useful addition to the job execution toolbox.

Resources

• SAS Viya 3.5 Administration / Jobs and Flows
• Sas Viya 3.5 Administration / Environment Manager
• SAS Viya 3.4: Schedule SAS Studio 5.1 (Studio V) Program Files
• Schedule SAS Studio Program Files

Want to Learn More about Viya 3.5?

• DevOps Applied to SAS Viya 3.5: Run a SAS Program with a Jenkins Pipeline
• DevOps Applied to SAS Viya 3.5: Top Git Commands with Examples
• Two Simple Ways to Import Local Files with Python in CAS (Viya 3.5)
• Six Ways to Import a Local File into CAS (SAS Viya 3.5)
• Query Performance? Use a CAS Star Schema in Viya 3.5
• Automatic Data Loading with a CAS Data Source Star Schema in Viya 3.5
• How to Load Images in SAS Viya 3.5
• Improve Your Relationships: With a REST API in SAS Viya 3.5

Stay tuned for more stories. And please comment, share your experience with Job Flow and help others. Thank you for your time reading this post.

[SASKiwi]

@Bogdan_Teleuca  - An excellent introduction to scheduling on Viya. I read somewhere that the SAS Job Flow Scheduler doesn't currently have the functionality to kill a running job. If so is that a feature being considered for a later release?

[Bogdan_Teleuca]

@SASKiwi Stop (kill) a running job is on Product Management features list, I am assured.

[SASKiwi]

@Bogdan_Teleuca  - Good to know.

[ATerado]

@Bogdan_Teleuca thanks for this really helpful article! If I have a job flow with 4 jobs, and 1-3 execute successfully, but job 4 fails, is there a way when the next time the flow runs to have it start with Job 4?

[Bogdan_Teleuca]

You can have Job Flow 4 running before Job 1 for example and play with the connection and event type to allow Job 1 to run if 4 fails.

[V_Altomonte]

Beautiful article!
Is it possible to manage the job frequency by considering the outcome of the job itself?
E.g. schedule a job every hour, but it should only start if the execution of the previous hour is completed.