Parallel processing with Loops

[prad001]

Hi All.,

I am trying to find out if there is a way to process the do loop in parallel.

Example:

data ds1;

  set sample(where=(a=1));

run;

%do i = 1 %to 100;

  data ds2_&i;

    set ds1(firstobs=&i obs=&i);

  run;

%end;

Instead of running one loop after another., I want the data step (ds2_&i) to run in parallel and process 100 datasteps at the same time.

Please let me the best possible way to run this in parallel. 

Thank you and appreciate your help. 

[SASKiwi]

To run a 100 DATA steps at the same time would require 100 separate SAS sessions. This is not a realistic possibility. It would help if you explained the processing problem you have rather than just describing the solution you want.

[prad001]

I meant not to create 100 SAS sessions but can at least divide it by 5 independent sessions (example: session1 - loops 1-20, session2 loops 21-40.... so on). 

We are trying to find a way to run our loops in parallel because of the efficiency purpose.  

[SASKiwi]

Do you have SAS/CONNECT available to you? Creating 5 parallel sessions is very easy if you have this product. You can check what products you have by running this:

proc product_status;
run;

proc setinit;
run;

[prad001]

Unfortunately, I do not have SAS/CONNECT.

[SASKiwi]

Assuming you are using a remote SAS server, do you have the ability to execute OS commands from your SAS sessions? You can check this by running this:

proc options option = xcmd;
run;

[prad001]

Yes, X command is enabled. And I am using SAS on Linux.

[SASKiwi]

Cool, that means you can "shell" a new SAS session to run the parallel processes like this:

x 'sas MySASProgram.sas';

I don't have the time to dig out an actual parallel example right now but you can search the Communities using the words "parallel processing" as there have been plenty of posts on this. 

[Kurt_Bremser]

On UNIX systems, you can run multiple programs in parallel from a shell script.

sas program1.sas&
sas program2.sas&
sas program3.sas&
sas program4.sas&
sas program5.sas&
wait

The & sends the execution to the background, and wait will only return when all background processes have finished. 

[prad001]

Thank you for the response Kurt.

Why do I have to split the program.

My intention was to run the loop in to 100 new SAS sessions. How can I do I create 100 sessions for a given example using SYSTASK ? 

And yes, I also read you another comment regarding increasing parallel processing may decrease in the performance. I am trying to figure out the optimal number of parallel processing. Not 100 ofcourse. 

[Kurt_Bremser]

Also note that parallel processing will only improve performance if you have enough CPU cores, and the storage to handle the I/O. As soon as you reach those limits, further parallelization will in fact lead to substantially worse overall performance.

[ChrisNZ]

What @Kurt_Bremser said: parallel means slower if the hardware is not up to it.

Also, I question your need to split the table like this. Why do you need to do this?

[mkeintz]

Why do you need to create 100 datasets single-observations datasets?  Why not make one dataset with 100 observations, and a new variable, say I_GROUP, corresponding to the DS2_&i dataset names?

data ds2_all;
  set ds1 (where=(a=1) obs=100);
  i_group=_n_;
run;

Note that the where filter is outsourced to the data set engine, prior to delivering observations to the data step.  Therefore the _N_ variable (i.e. the iteration number of the data step) is counting only the a=1 observations.  So i_group is equivalent to your DS2_&i dataset names.

And like @ChrisNZ,  I don't see any evident need to do parallel processing, especially for this particular task.

[s_lassen]

Take a look at the SYSTASK statement (for Linux here: SAS Help Center: SYSTASK Statement: UNIX ) which enables you to start several processes in parallel, wait for them all to finish, and get the results for each process.

[Patrick]

Given that you haven't got SAS/Connect but only xcmd available any parallelization will quickly add a lot of complexity to your program. 

I would first investigate if you can't performance optimize your current code without parallelization before going for such an approach.

If you share more detail what you really have then people here might be able to provide some guidance.