Run multiple Process Flows at once using SAS Code (on EG 7.11 HF3)

[DanielDor]

Greetings,

1. I have a project with ~40 Process Flows.

2. I would like to run 25 out of the 40 Process Flows every morning from within EG 7.11.

3. What would be the best way to do this? I saw few suggestions on how to do this, but all of them are not relevant to my need:

3.1. Using Ordered List (http://stackoverflow.com/questions/22388213/sas-enterprise-run-multiple-process-flows-at-once)

3.2. Picking the 25 process flows and then Right-Click and Run

3.3. Build the project as a stored process.

4. What can I do? I saw that there is an option to use DS2 and Run() method, but I'm not sure it is relevant for this specific use.

Thanks!

Daniel

[SASKiwi]

If you want to fully automate then export the 25 process flows as SAS code and run that as a scheduled SAS batch job, preferably on your SAS server as its a lot more reliable than your PC.

[DanielDor]

Hi SASKiwi,

Thanks a lot for your prompt response. Currently, I’m working on SAS EG Desktop version (we don’t have a Server on my organization).

How it can be done on SAS EG Desktop version? Is there any way to write SAS Code which run all the process flows one after the other?

Thanks!

D

[SASKiwi]

Where is your SAS code running then? On your desktop? If it is then you can still schedule a SAS batch job on your desktop with Windows task scheduler.

[rogerjdeangelis]

This is a related topic and probably does not directly answer the OPs question.

This is a bare minium systask macro driver, without checkpoint restart or programmable conditional processing.

I have posted solutions like this using a 'clickable' list of processes.
Unfortunately my SCL code will not run in EG.
Just to be safe programmers might consider either RShiny or Python Django with
'localhost', just in case SAS drops strored process support to get
users to migrate to the more web centric SAS Studio.

I have posted config file driven processing on SAS-L.


%macro sho_100(fro=Asia,typ=boot);
 data &fro;
 set sashelp.shoes(where=(product="&typ" and region="fro"));
 run;quit;
%mend sho_100;

%macro sho_200(fro=Asia,typ=boot,minstore=2);
 data &fro;
 set sashelp.shoes(where=(product="&typ" and region="fro" and stores>&minstore));
 run;quit;
%mend sho_200;

%let _r=/root;

* put in your autoexec;
%let _s=%sysfunc(compbl(/rjd/.../sas/sashome9.3/SASFoundation/9.4/sasexe/sas
 -sysin /rjd/.../sas/contract/SASEG_files/rde250/oto/dummy.sas
 -sasautos /rjd/.../sas/contract/SASEG_files/rde250/oto
 -autoexec /rjd/.../sas/contract/SASEG_files/rde250/oto/batch_autoexec.sas));

systask kill xeq1 xeq2;
systask command "&_s -memsize 2g -termstmt %nrstr(%sho_100(fro=Asia,typ=boot) -altlog &_r/log/xeq1.log" taskname=xeq1;
systask command "&_s -memsize 48g -termstmt %nrstr(%sho_100(fro=Canada,typ=boot) -altlog &_r/log/xeq2.log" taskname=xeq2;
waitfor xeq1 xeq2;
systask command "&_s -memsize 2g -termstmt %nrstr(% -altlog &_r/log/xeq33.log" taskname=xeq3;
systask command "perl sample.pl -log &_r/log/xeq4.log" taskname=xeq4;
systask command "R sample.R -log &_r/log/xeq5.log" taskname=xeq4;
systask command "python sample.py -log &_r/log/xeq5.log" taskname=xeq5;

[CaseySmith · Accepted solution]

Another option is to use EG's automation interface...

-With a process flow visible in EG, select Schedule-><processFlowName> on the contextual toolbar (above the process flow) to schedule a single process flow

-That will create a EGScript1.vbs file that runs a single process flow (and a Windows Scheduled Task that executes the VBScript file on a schedule you specify)

-You can edit the VBScript code (without too much trouble if you have some basic programming knowledge) to run the 25 desired process flows instead of a single one

Here is an example (just the section of the EGScript1.vbs I edited):
(I confirmed it works.)

    '-----
    'Get The Container Collection and Object
    '-----    
    Set containerColl = prjObject.ContainerCollection
    If Checkerror("Project.ContainerCollection") = True Then
        Exit Sub
    End If
    
    Dim i       ' As Long
    Dim count   ' As Long
    count = containerColl.count
    For i = 0 To count - 1
        Set containerObject = containerColl.Item(i)
        If Checkerror("ContainerCollection.Item") = True Then
            Exit Sub
        End If
        
        'Run the process flows with the desired names
        If (containerObject.Name = "Process Flow1" Or _
            containerObject.Name = "Process Flow3") Then
            '----
            ' Run the Container
            '----
            containerObject.Run
            If Checkerror("Container.Run") = True Then
                Exit Sub
            End If               
        End If
    Next 

[DanielDor]

Great! Thanks a lot for the info 🙂

I've solved the issue by doing the following process:

A. dividing my project to 4 different projects:

1. Extract Project

2. Transform Project

3. Load Project

4. Managament Project

B. Export the 3 uppers projects as SAS Code

C. Running the 3 SAS codes from Management Project

[luboxing]

Thanks for sharing the vbs code! I tested the code and it seem to work in serial, meaning "containerObject.Run" is blocking, preventing multiple processing. Until one object is done, the next one won't kick off.

Any clues?

Bo

[CaseySmith]

While you can run process flows asynchronously (at the same time) manually in the UI (multi-select all the process flows you want to run in the Project Tree, right-click and select Run), the EG scripting interface currently only supports running synchronously.

(Note: If the runnable items in your process flows are assigned to run on the same server, they will run synchronously by default.  If you want to spawn additional sessions on the same server to run items in parallel, you can turn on the File->Project Properties->Code Submission->"Allow parallel execution on the same server" option.)

Casey

[luboxing]

Thanks Casey! It really works!