DATA Step, Macro, Functions and more

Running multiple queries parallely

Reply
N/A
Posts: 0

Running multiple queries parallely

Hi,

I have 5 queries in 5 different sas files that takes 5 hrs app. while running sequentially using %include in main sas code. I'm planning to run all of them in parallel so that they will complete in 3 hrs.

I tried using PIPE and X command by calling the shell scripts which are calling these five jobs in each, but is there any way to run them in parallel in main code itself?


Thanks,
Guru
Respected Advisor
Posts: 4,173

Re: Running multiple queries parallely

Posted in reply to deleted_user
My personal preference would be to use systask() in the main program as it gives you a lot of control on how the child processes are executed (parallel or sequential).
Super User
Posts: 3,250

Re: Running multiple queries parallely

I have successfully used SAS/CONNECT for this purpose. If you have this licensed then this could be an option. The trick is to use the WAIT = NO option when submitting the child jobs as this then allows the controlling SAS job to continue without waiting for the submitted child SAS job/session to finish.

For example running two parallel jobs from a controlling SAS session:

signon session1;
signon session2;

rsubmit session1 wait = no;
%include "program1.sas";
endrsubmit;

rsubmit session2 wait = no;
%include "program2.sas";
endrsubmit;

Please note that session1 and session2 could be defined as additional SAS sessions on the same computer you are working on or they could refer to a remote computer. The choice is yours.

There is the problem of not knowing when the child sessions finish their work so you can continue on afterwards with more processing. This can be solved by getting the child sessions to write a special file or dataset when they end. Your controlling SAS session can periodically check for these special files and only continue after they are created.
Valued Guide
Posts: 2,177

Re: Running multiple queries parallely

to support asynchronos remote processing (like sas mpConnect) SAS/Connect provides features like WAITFOR
Lifted from the on-line help and documentation [pre]/* Wait for both tasks to complete. */
waitfor _all_ task1 task2;[/pre] which comes in SAS/Connect Using Compute Services Example 5: Using MP CONNECT and the WAITFOR Statement
That collection of examples can be found at http://support.sas.com/documentation/cdl/en/connref/61908/HTML/default/a000585202.htm .

enjoy the potential

peterC
Super User
Posts: 3,250

Re: Running multiple queries parallely

Thanks for the tip Peter.

WAITFOR is definitely the way to go to take back control after child parallel SAS processes have run.
Ask a Question
Discussion stats
  • 4 replies
  • 226 views
  • 0 likes
  • 4 in conversation