Viya Jobs and Flows – File Event Triggers!

3 Likes

Previously I wrote about Jobs and Flows' time-based scheduling capabilities and I even wrote about making your own DIY file event triggering scheme. With the 2025.02 release, Jobs and Flows includes its own file event triggering functionality. Let's take a look.

Defining File Event Triggers

Triggering jobs to run using the system clock is great. You can make jobs run at specific times on specific dates; repeatedly every few minutes, hours, days, .... But! What people usually need is event-based scheduling. When everything the job needs - input data, system resources, permission from the stake holders, etc. - is in place, then run the job. No clock required! So how can we know that everything is in place? We need a signal. The trigger file is that signal. When the trigger file appears, we know everything is in place and we can run. We set an agreement with the system owners and/or the source data providers, "When everything is ready, put a file named X in location Y."

Adding Event Triggers in Jobs and Flows

To add a file event trigger, right click the job flow and select Schedule Flow. On the following screen, click New and select File Event.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

From there, name your file event, point to a file in SAS content and add a condition. The conditions are described in detail here but, for our scheme, we'll choose the condition, "Exists."

So, as shown, by the graphic below, our job flow will not execute until the file nightlyFlow.trigger is detected in SAS content.

Combining Event Triggers with Time Triggers

In many scenarios, you'll want to combine a file trigger with a time trigger to achieve something like "Run the flow every midnight when the trigger file is detected." This combination gives you a regular execution cadence but also adds the trigger component to ensure that source data and other resources are ready.

Getting Files into SAS Content

As discussed earlier, the 2025.02 implementation only detects trigger files in SAS content. Writing to SAS content is fairly straightforward. It can be done with base SAS code as well as the Viya API. Below are examples.

Base SAS - Write a SAS Content File

filename trgrFile filesrvc folderpath='/Public' filename='nightlyFlow.trigger';
data _null_;
   file trgFile;
   stop;
run;

Python - upload2sas.py, code to Upload a File to SAS Content

import argparse
import os
from sasctl import Session
from sasctl.services import files, folders

# Server and authentication details
server_name = "server.demo.sas.com"
user = "sasadm"
password = "lnxsas"

def upload_file(local_file, target_folder):
    """
    Upload a single file to the SAS Content Server.
    Parameters
    ----------
    local_file : str
        Path to the local file to be uploaded.
    target_folder : str
        Path to the target folder in SAS Content Server.
    Returns
    -------
    None
    """
    # Extract the file name from the local file path
    file_name = os.path.basename(local_file)

    # Ensure the target folder exists in SAS Content Server
    folders.create_path(target_folder)

    # Upload the file to the target folder
    files.create_file(local_file, target_folder, file_name)
    print(f"File '{file_name}' uploaded successfully to '{target_folder}'.")

if __name__ == "__main__":
    # Parse command-line arguments
    parser = argparse.ArgumentParser(description="Upload a file to the SAS Content Server.")
    parser.add_argument("--local-file", required=True, help="Path to the local file to be uploaded.")
    parser.add_argument("--target-folder", required=True, help="Target folder in SAS Content Server.")
    args = parser.parse_args()

    # Ensure the local file exists
    if not os.path.exists(args.local_file):
        print(f"Error: The file '{args.local_file}' does not exist.")
        exit(1)  # Exit with a non-zero status to indicate an error

    # Start a session and upload the file
    with Session(server_name, user, password):
        upload_file(args.local_file, args.target_folder)

Python - Commands to install the necessary Python libraries; Write a trigger file; Upload it to SAS content

/opt/conda/bin/pip install sasctl

touch /tmp/flow0.trigger

python upload2sas.py --local-file /tmp/flow0.trigger --target-folder /Public

Base SAS - Delete a SAS Content File

filename trgrMart filesrvc folderpath='/Public' filename='nightlyFlow.trigger';
data _null_;
 rc=fdelete('trgrMart');
run;

Find more articles from SAS Global Enablement and Learning here.