topic Running multiple queries parallely in SAS Programming

Running multiple queries parallely

deleted_user — Tue, 18 May 2010 06:11:33 GMT

Hi,

I have 5 queries in 5 different sas files that takes 5 hrs app. while running sequentially using %include in main sas code. I'm planning to run all of them in parallel so that they will complete in 3 hrs.

I tried using PIPE and X command by calling the shell scripts which are calling these five jobs in each, but is there any way to run them in parallel in main code itself?

Thanks,
Guru

Re: Running multiple queries parallely

Patrick — Tue, 18 May 2010 09:07:30 GMT

My personal preference would be to use systask() in the main program as it gives you a lot of control on how the child processes are executed (parallel or sequential).

Re: Running multiple queries parallely

SASKiwi — Thu, 20 May 2010 01:39:10 GMT

I have successfully used SAS/CONNECT for this purpose. If you have this licensed then this could be an option. The trick is to use the WAIT = NO option when submitting the child jobs as this then allows the controlling SAS job to continue without waiting for the submitted child SAS job/session to finish.

For example running two parallel jobs from a controlling SAS session:

signon session1;
signon session2;

rsubmit session1 wait = no;
%include "program1.sas";
endrsubmit;

rsubmit session2 wait = no;
%include "program2.sas";
endrsubmit;

Please note that session1 and session2 could be defined as additional SAS sessions on the same computer you are working on or they could refer to a remote computer. The choice is yours.

There is the problem of not knowing when the child sessions finish their work so you can continue on afterwards with more processing. This can be solved by getting the child sessions to write a special file or dataset when they end. Your controlling SAS session can periodically check for these special files and only continue after they are created.

Re: Running multiple queries parallely

Peter_C — Fri, 21 May 2010 11:43:35 GMT

to support asynchronos remote processing (like sas mpConnect) SAS/Connect provides features like WAITFOR
Lifted from the on-line help and documentation [pre]/* Wait for both tasks to complete. */
waitfor _all_ task1 task2;[/pre] which comes in SAS/Connect Using Compute Services Example 5: Using MP CONNECT and the WAITFOR Statement
That collection of examples can be found at http://support.sas.com/documentation/cdl/en/connref/61908/HTML/default/a000585202.htm .

enjoy the potential

peterC

Re: Running multiple queries parallely

SASKiwi — Mon, 24 May 2010 02:36:02 GMT

Thanks for the tip Peter.

WAITFOR is definitely the way to go to take back control after child parallel SAS processes have run.