Macros and Parallel processing

[SP_SAS]

Hi there,

I have a program that currently runs like this. I have simplified to ask question.

%let total_runs=30;

%let max_paralle_allowed = 10; /*you do not want to run more than 10 jobs at a time*/

%do i = 1 to 3;

%do j = %eval((&i*&max_paralle_allowed)-10+1) %to %eval(&i*&max_paralle_allowed);

options sascmd="!sascmld memsize=2g" autosignon;

signon remote=task&j;

rsubmit process=task&j wait=no log="" output="";

libname abc "";

%include "mymacro.sas";

%mymacro();

endrsubmit;

%end; /*end of j*/

waitfor _all_ task1 task2 task3.....task10;

signoff remote = task1 task2 task3... task10;

%end; /*end of i*/

Issue: Now what happens is currently it will run 10 parallel jobs. Once all 10 are done it will kick off another 10 and then another 10. What happens in this setting is when out of the first 10, 8 jobs are done and 2 are running for 10 more hours, the program waits for those 2 to complete before starting next 10.

What I want to do is : The program first kicks off 10 max allowed jobs. Then keeps checking if total running jobs are less then 10 if that is the case then it will kick off next one and so on.. i.e. as soon as some of the first 10 jobs are complete the new jobs keep getting submitted (max running at a time 10)..

Any help would be appreciated.

[Patrick]

@SP_SAS 

I guess what you would need to do is use a WAITFOR _ANY_.

If any of the tasks finishes you then need to check which tasks are still running and which one finished so that you know how many new tasks you can start - and eventually also get the correct names for the &task<n> macro variables so that you only reuse macro variable names for iterations which finished already.

If you've got SAS DI Studio: The Loop transformation would let you implement such logic easily and very robust.

[SP_SAS]

Thanks..

[ChrisBrooks · Accepted solution]

Hi @SP_SAS - it will involve some significant recoding but I think the article on this web page will do what you want https://direct-root.com/blog/dynamic-parallel-processing-in-sas-rsubmit/