topic Re: Parallel processing with Loops in SAS Programming

Parallel processing with Loops

prad001 — Mon, 11 Oct 2021 18:47:06 GMT

Hi All.,

I am trying to find out if there is a way to process the do loop in parallel.

Example:

data ds1;

set sample(where=(a=1));

run;

%do i = 1 %to 100;

data ds2_&i;

set ds1(firstobs=&i obs=&i);

run;

%end;

Instead of running one loop after another., I want the data step (ds2_&i) to run in parallel and process 100 datasteps at the same time.

Please let me the best possible way to run this in parallel.

Thank you and appreciate your help.

Re: Parallel processing with Loops

SASKiwi — Mon, 11 Oct 2021 19:00:28 GMT

To run a 100 DATA steps at the same time would require 100 separate SAS sessions. This is not a realistic possibility. It would help if you explained the processing problem you have rather than just describing the solution you want.

Re: Parallel processing with Loops

prad001 — Mon, 11 Oct 2021 19:17:01 GMT

I meant not to create 100 SAS sessions but can at least divide it by 5 independent sessions (example: session1 - loops 1-20, session2 loops 21-40.... so on).

We are trying to find a way to run our loops in parallel because of the efficiency purpose.

Re: Parallel processing with Loops

SASKiwi — Mon, 11 Oct 2021 19:27:05 GMT

Do you have SAS/CONNECT available to you? Creating 5 parallel sessions is very easy if you have this product. You can check what products you have by running this:

proc product_status;
run;

proc setinit;
run;

Re: Parallel processing with Loops

prad001 — Mon, 11 Oct 2021 19:30:30 GMT

Unfortunately, I do not have SAS/CONNECT.

Re: Parallel processing with Loops

SASKiwi — Mon, 11 Oct 2021 20:16:21 GMT

Assuming you are using a remote SAS server, do you have the ability to execute OS commands from your SAS sessions? You can check this by running this:

proc options option = xcmd;
run;

Re: Parallel processing with Loops

prad001 — Mon, 11 Oct 2021 20:25:16 GMT

Yes, X command is enabled. And I am using SAS on Linux.

Re: Parallel processing with Loops

SASKiwi — Mon, 11 Oct 2021 20:34:01 GMT

Cool, that means you can "shell" a new SAS session to run the parallel processes like this:

x 'sas MySASProgram.sas';

I don't have the time to dig out an actual parallel example right now but you can search the Communities using the words "parallel processing" as there have been plenty of posts on this.

Re: Parallel processing with Loops

Kurt_Bremser — Mon, 11 Oct 2021 21:18:00 GMT

On UNIX systems, you can run multiple programs in parallel from a shell script.

sas program1.sas&
sas program2.sas&
sas program3.sas&
sas program4.sas&
sas program5.sas&
wait

The & sends the execution to the background, and wait will only return when all background processes have finished.

Re: Parallel processing with Loops

Kurt_Bremser — Mon, 11 Oct 2021 21:27:47 GMT

Also note that parallel processing will only improve performance if you have enough CPU cores, and the storage to handle the I/O. As soon as you reach those limits, further parallelization will in fact lead to substantially worse overall performance.

Re: Parallel processing with Loops

ChrisNZ — Mon, 11 Oct 2021 21:54:19 GMT

What @Kurt_Bremser said: parallel means slower if the hardware is not up to it.

Also, I question your need to split the table like this. Why do you need to do this?

Re: Parallel processing with Loops

mkeintz — Tue, 12 Oct 2021 01:51:36 GMT

Why do you need to create 100 datasets single-observations datasets? Why not make one dataset with 100 observations, and a new variable, say I_GROUP, corresponding to the DS2_&i dataset names?

data ds2_all;
  set ds1 (where=(a=1) obs=100);
  i_group=_n_;
run;

Note that the where filter is outsourced to the data set engine, prior to delivering observations to the data step. Therefore the _N_ variable (i.e. the iteration number of the data step) is counting only the a=1 observations. So i_group is equivalent to your DS2_&i dataset names.

And like @ChrisNZ, I don't see any evident need to do parallel processing, especially for this particular task.

Re: Parallel processing with Loops

s_lassen — Tue, 12 Oct 2021 11:32:02 GMT

Take a look at the SYSTASK statement (for Linux here: SAS Help Center: SYSTASK Statement: UNIX ) which enables you to start several processes in parallel, wait for them all to finish, and get the results for each process.

Re: Parallel processing with Loops

Patrick — Tue, 12 Oct 2021 11:41:47 GMT

Given that you haven't got SAS/Connect but only xcmd available any parallelization will quickly add a lot of complexity to your program.

I would first investigate if you can't performance optimize your current code without parallelization before going for such an approach.

If you share more detail what you really have then people here might be able to provide some guidance.

Re: Parallel processing with Loops

prad001 — Tue, 12 Oct 2021 20:49:53 GMT

Thank you for the response Kurt.

Why do I have to split the program.

My intention was to run the loop in to 100 new SAS sessions. How can I do I create 100 sessions for a given example using SYSTASK ?

And yes, I also read you another comment regarding increasing parallel processing may decrease in the performance. I am trying to figure out the optimal number of parallel processing. Not 100 ofcourse.

Re: Parallel processing with Loops

prad001 — Tue, 12 Oct 2021 20:58:59 GMT

HI Patrick,

Please find the below code.. I know this is not the best way to write SAS program. It is the legacy code, I want to see if parallel processing makes any difference. I wanna process the "J" loop in parallel.

%MACRO TEST;
LIBNAME test 'path/location/';

DATA DS1;
SET test.inp_1;
RUN;

DATA DS2;
SET tes.inp_2;
RUN;

PROC SORT DATA=DS1;
By var;
run;

PROC SORT DATA=DS2;
By var;
run;

DATA ds3;
MERGE ds1(in=a) ds2(in=b);
by var;

if a and b;
RUN;

PROC SORT DATA=ds3 nodupkey out=ds4;
By var2;
RUN;

DATA _null_;
SET ds3;
CALL SYMPUT('var3',compress(_n_));
RUN;

%DO j=1 %TO &var3;**********50 times;
DATA _null_;
set ds3(firstobs=&j obs=&j);
CALL SYMPUT('var4',compress(var4));
CALL SYMPUT('var5',compress(var5));
RUN;

DATA ds5;
SET ds3;
WHERE var4="var4" and var5="var5";
RUN;

PROC SORT DATA=ds5 nodupkey out=ds6;
BY var4 var5 var6;
RUN;

DATA _null_;
SET ds6;
CALL SYMPUT('var7',compress(_n_));
RUN;

%DO i=1 %TO &var7;***************100 times;
DATA _null_;
SET ds6;
CALL SYMPUT('var4',compress(var4));
CALL SYMPUT('var5',compress(var5));
CALL SYMPUT('var8',compress(var8));
RUN;

DATA ds7;
set ds5;
WHERE var4="&var4" and var5="&var5" and var8="&var5";
RUN;

DATA ds7;
set ds7;

do i=1 to count;
output;
end;

if a > . then
call symput ('a',compress(a));

if b > . then
call symput ('b',compress(b));
RUN;

PROC SUMMARY data=ds7;
var date;
output out=ds8 min=mindate max=maxdate;
RUN;

******
ODS FOR GRAPH
PROC CAPABILITY
PROC APPEND;

******;
%END;
%END;
%MEND;

%TEST;

Re: Parallel processing with Loops

Patrick — Tue, 12 Oct 2021 22:21:41 GMT

These are macro do loops that just generate SAS code. These loops won't take that long to run. It's the generated SAS code (50*100 times almost the same with many passes through the data) that will take up the time.

Looking into the code you've shared I'm rather certain that you could get rid of all macro processing and get this done via "normal" SAS only using by-group processing. This will then also perform much better. Fixing the code is where you should spend your time.

It's a bit hard to provide fixed code without representative sample data and desired result. I've mocked-up something below but it will likely not fully match what you need. It should show you the way to go.

data inp_1;
  infile datalines truncover dlm=',' dsd;
  input var (var4 var5) ($);
  datalines;
1,x,y
2,x,y
3,a,b
;
data inp_2;
  infile datalines truncover dlm=',' dsd;
  input var var2 $ date :date9.;
  format date date9.;
  datalines;
1,a,01jan2021
1,a,01feb2021
1,b,01mar2021
1,a,01apr2021
2,a,01jan2021
2,c,01feb2021
2,c,01mar2021
4,a,01jan2021
;

libname test "%sysfunc(pathname(work))";
data ds1;
  set test.inp_1;
run;

data ds2;
  set test.inp_2;
run;

proc sort data=ds1;
  by var;
run;

proc sort data=ds2;
  by var;
run;

data ds3;
  merge ds1(in=a) ds2(in=b);
  by var;
  if a and b;
run;

proc summary data=ds3 ;
  class var2;
  var date;
  ways 1;
  output out=ds8 min=mindate max=maxdate;
run;

Re: Parallel processing with Loops

prad001 — Wed, 13 Oct 2021 13:38:25 GMT

Hi Patrick., Thank you for the code. But if you see my code., ds3 and ds6 are made unique(nodupkey) and the both the loops are up until the last unique value on the full dataset. And if you do by processing, I am not sure of how the nested loops will work. Because this is not very complex code at all but I am not getting an approach of how to write it alternatively.

Re: Parallel processing with Loops

Kurt_Bremser — Wed, 13 Oct 2021 18:32:06 GMT

Nesting is done in BY by using multiple variables.

by a b;

will do a group change whenever b or a changes, and because of the preceding sort with the same BY, all b groups within the first a group will be dealt with first, then all b groups within the second a group, and so on.

You really need to get an understanding of BY first before you engage in such unwieldy and inefficient macro coding.