Run multiple simulations at the same time

[thewan]

I'd like to run multiple simulations at the same time. Say I have a dataset with 100 simulations. I wrote a do-while loop to go through these 100 simulations one by one. First I'll look at simulation 1, isolate those observations, process them, and then move onto simulation 2.

The problem is that this sequential approach is taking a really long time to run. I am wondering if there is a way to isolate 5 or 6 simulations at the same time and and run them simultaneously. The results of one simulation don't depend on the one before it so this approach makes more sense.

Thank you. I've included example code below:

Edit: I wanted to add: Is there a way to create rand1 and rand2 at the same time? Right now I'm setting &n=1, rand1 is created, &n is now 2, and rand2 is created. Going through a dataset sequentially is eating up a lot of time. My simulations don't depend on one another so I can run them at the same time. Is that possible? The z variable is something I came up with this morning. This example is to show how my do while loop works.

%let NumSimul = 2;

*Generating data from uniform distribution for 2 simulations with 50 observations;

data rand;
call streaminit(1726);
do SimulID = 1 to &NumSimul;
    do obs = 1 to 50;
	x = rand ('uniform');
	y = rand ('uniform');
	output;
    end;
end;
run;

%macro main(n);			*Creating new datasets;
    data rand&n;
	set rand;
	where SimulID=&n;
	z = x + y;
    run;
%mend;

%macro runmain;			*Macro where datasets are created sequentially;
	%LET n=1;
	%DO %WHILE(&n <=&NumSimul);
	%main(&n);
	%LET n=%EVAL(&n+1);
	%END;
%MEND runmain;
%runmain;

[sbxkoenk]

Hello,

What SAS version are you running? 

SAS 9.4 M7 or SAS VIYA 3.5 or ...?

You can find out by submitting:

%PUT &=sysvlong;

In VIYA you can leverage CAS.

In SAS 9.4 you can run asynchronous jobs with MP (Multi-Process) CONNECT. SAS/CONNECT should be licensed.

If you specify what SAS version you are running I can give more details (if still needed) and possibly an example.

Cheers,

Koen

[thewan]

Hello Koen,

This is what came up in the log file: SYSVLONG=9.04.01M6P110718

If I can make use of parallel computing that would be great!

[sbxkoenk]

Hello,

OK, no VIYA but SAS 9.4 M6 (Maintenance Level 6).

You need MP Connect (MPConnect, Multi-Process Connect) then for parallel processing. Unless you can solve your problem with by-groups (that would be easier).

Unfortunately I have absolutely no time today to prepare an example.

See here for some useful links.

SASGF Proceedings sidebar
Posted 01-21-2021 03:27 PM | by ChrisHemedinger
https://communities.sas.com/t5/Promo-and-Sidebar-TKB/SASGF-Proceedings-sidebar/ta-p/713212

You will find numerous papers on MP Connect (or MPConnect) by using the search engine on these links.

Good luck,

Koen

PS: let me know if you succeed. If not, I can post a small example on Monday (May 3)

[thewan]

Hello Koen,
I am still working on this problem. In the original code, I'm saving output to a permanent file /Temp/Mu. How can I replicate this in a remote session and pull the information back to the parent session?

Thank you.

[SASKiwi]

Have you confirmed you have SAS/CONNECT both installed and licensed?

proc product_status;
run;

proc setinit;
run;

If you don't have CONNECT then running multiple, simultaneous SAS batch jobs is an alternative.

[thewan]

Hello,
I checked the log file and it looks like I do have a license to SAS/CONNECT: ---SAS/CONNECT
14AUG2021 (CPU A)
I am referencing this paper (https://www.lexjansen.com/wuss/2018/15_Final_Paper_PDF.pdf) and using the SYSTASK command. It looks like the SYSTASK command is not running because my output files are not being saved to the location I specified.

[SASKiwi]

@thewan - Check out SAS/CONNECT example 5

SYSTASK isn't required for this approach.

[thewan]

I am hoping this is a better example. I'd like to call OpenBUGS into SAS and then run the Bayesian analysis. Right now each simulation is being completed sequentially. I would like to take advantage of parallel processing, generate the coda files simultaneously for each simulation and cut back on processing time. Please note that this is only an example. I've included comments so that each step is clear. Thank you.

dm 'log;clear;output;clear; '; 						/*Clear the log file, results window, working directory*/;
dm 'odsresults; clear';;
PROC DATASETS LIB=work NOlist MEMTYPE=data kill; 
options nosymbolgen nomprint nomlogic; 					/*system options set to default*/


%let path = /panfs/pfs.local/work/example2; 	                      /*Define path as a macro variable that can be referenced throughout program*/

%LET J=100;	                                                      *obs in each simulation;
%LET NumSimul=2;					              *number of simulations;

DATA Sim_Reg;								/*Generate data*/
  b0=1; b1=2; b2=3; sig_e=2; seed=20060118;
  DO NumSimul = 1 TO &NumSimul;
  	DO j = 1 to &J;
        x1=RANNOR(seed+&NumSimul);
		x2=RANNOR(seed+&NumSimul);
		e=RANNOR(seed+&NumSimul);
     	y = b0+b1*x1+b2*x2+sig_e*e;
   KEEP NumSimul J y x1 x2;
   OUTPUT;
   END;
   END;
RUN;


DATA model;								/*Define Model*/
INPUT model $80.;
CARDS;/*start the model*/
model{
#Model specification
   for (i in 1:100) {
      y[i]~dnorm(muy[i], Inv_sig2_e)
   muy[i]<-b0+b1*x1[i]+b2*x2[i]
   }
#priors
   b0~dnorm(0, 1.0E-6)
   b1~dnorm(0, 1.0E-6)
   b2~dnorm(0, 1.0E-6)
   Inv_sig2_e~dgamma(1.0E-3, 1.0E-3)
#parameter transformation
   Sig2_e<-1/Inv_sig2_e
}
;
RUN;
DATA _NULL_;							/*Store model as text file in this location*/
  SET model;
  FILE "&path/RegModel.txt";
  PUT model;
RUN;

DATA INIT;							/*Create the initial values*/
INPUT initials $80.;
CARDS;
list(b0=0, b1=0, b2=0, Inv_sig2_e=1)
;
RUN;

DATA _NULL_;							/*Export them*/
SET INIT;
FILE "&path/RegInit.txt";
PUT initials;
RUN;



%MACRO main(n);

data RegData(rename=(cx1=x1 cx2=x2 cy=y));			/*Prepare data to be exported as a text file*/
	set Sim_Reg;
	where NumSimul=&n;

	cx1=put(x1,7.3);					/*convert variables to character format*/
	cx2=put(x2,7.3);
	cy=put(y,7.3);

	drop x1 x2 y j NumSimul;
run;

data RegData;
	length x1 $7 x2 $7 y $7;
	if _n_=1 then do; 					/*insert variable names for OpenBUGS*/ 
		x1 = "x1[]";
		x2 = "x2[]";
		y = "y[]  ";
		output;  
	end;
set RegData;
output;
run;

proc sql;
	insert into RegData (x1) values ('END');			/*insert END statement at eof*/
run;

PROC EXPORT DATA=RegData						/*Export data as text file to this location*/
   OUTFILE="&path/RegData&n..txt"
      DBMS=TAB REPLACE;
   PUTNAMES=NO;
RUN;






DATA SCRIPT;								/*Script file so OpenBUGS knows what to do*/
LENGTH script $80.;
SCRIPT = "modelOutput('log')" ;OUTPUT;
SCRIPT = "modelCheck('&path/RegModel.txt')" ;OUTPUT;
SCRIPT = "modelData('&path/RegData&n..txt')" ;OUTPUT;
SCRIPT = "modelCompile(1)" ;OUTPUT;
SCRIPT = "modelInits('&path/RegInit.txt',1)" ;OUTPUT;
SCRIPT = "modelGenInits()"; OUTPUT;
SCRIPT = "modelUpdate(3000)" ;OUTPUT;
SCRIPT = "samplesSet(b0)" ;OUTPUT;						
SCRIPT = "samplesSet(b1)" ;OUTPUT;
SCRIPT = "samplesSet(b2)" ;OUTPUT;
SCRIPT = "samplesSet(Sig2_e)" ;OUTPUT;
SCRIPT = "modelUpdate(5000)" ;OUTPUT;
SCRIPT = "samplesStats('*')" ;OUTPUT;
SCRIPT = "modelSaveLog('&path/bugslog.txt')" ;OUTPUT;
SCRIPT = "samplesCoda('*','&path/output&n')" ;OUTPUT;			
SCRIPT = "modelQuit('yes')" ;OUTPUT;
RUN;

DATA _NULL_;								/*Export script file*/
SET SCRIPT;
FILE "Batch.txt";
PUT script;
RUN;


DATA _NULL_;
	X "OpenBUGS /panfs/pfs.local/work/example2/Batch.txt";		/*Execute*/
RUN;
%mend;




%MACRO runmain;								/*Call multiple simulations sequentially*/
  %LET n=1;
  %DO %WHILE(&n <=&NumSimul);
  %main(&n);
  %LET n=%EVAL(&n+1);
  %END;
%MEND runmain;
%runmain;

[sbxkoenk · Accepted solution]

Hello,

Sorry for not having gone through your code yet. I have time constraints.

I wonder why you are not doing your Bayesian analysis in SAS. The wealth of Bayesian analyses in SAS has grown pretty big over the recent years. What SAS version, maintenance release within SAS 9.4 are you using (submit %PUT &=sysvlong; to find out)?

Concerning parallel processing with base SAS and MP Connect.

See this recent communities (library) article. It gives you an example close to what you want to do (simulations).

Base SAS + SAS/CONNECT - A simple method to generate load on any number of licensed cores
Posted 4 weeks ago | by SimonWilliams
https://communities.sas.com/t5/SAS-Communities-Library/Base-SAS-SAS-CONNECT-A-simple-method-to-gener...

Cheers,

Koen