Specified the number of cores to be used in SAS/CONNECT

[RBA]

folks, I am using sas / connect in asynchronous mode, creating several hundred jobs with a loop. My challenge is that sas is not limiting the number of jobs it process simultaneously. It starts all jobs the CPU can take, engaging all available cores and saturating the hard drive (each job needs to create a copy of a dataset). I would like to be able to specify the maximum number of processes that are executed at any point in time, but SAS is ignoring the cpucount option.

I am using SAS is 9.4, SAS/STAT 15.2 on a 12-core/Windows 10 desktop and the relevant code is as follows:

options threads cpucount=4;

%let rc=%sysfunc(grdsvc_enable(_all_, resource=SASApp));
options autosignon;

%macro mymacro;

	/* loop over zip3's */ 
	%do i=1 %to %sysfunc(countw(&zip3list)) ;

		%let thisZip30=%scan(&zip3list,&i);
		%let taskname = t&thisZip30;

		%SYSLPUT year=&year./REMOTE=&taskname;
		%SYSLPUT thisZip3=&thisZip30./REMOTE=&taskname;

 		rsubmit &taskname wait=no connectpersist=no inheritlib=(ineq4 onlinek);

 		%nrstr(%%)PUT ******* zip3=&thisZip3  **************;

		/* extract data for zip3 and upc */
		data thiZip3train&thisZip3;
			set ineq4.promediosWeeRetTrain&year;
			where store_zip3="&thisZip3"; 
		run;

		/* do stuff with the extracted data */

                PROC DATASETS lib=work;
                      delete thiZip3train&thisZip3;
                quit;
                RUN;
		endrsubmit;	
	%end;

	waitfor _all_;
 	signoff _all_ ; 

%mend mymacro;

%mymacro;

[LinusH · Accepted solution]

First a little clarification about CPUCOUNT. This option controls no of cores available for the current SAS session. And it's valid in an OPTIONS statement. But it does not control no of SAS processes that can execute simultaneously. SAS "submits" these processes for the OS to manage.

It sopunds liek you need some kind of OS process queuing facility. This is not part of Base SAS or SAS/CONNECT (it's part of the SAS Grid Manager offering though).

If you don't have access to such software, I think you need to build the corresponding logic in your macro, and submit some of your remote sessions synchronously.

[RBA]

I appreciate the concise explanation. It helped me understand the problem. I am afraid I do not have the license for the grid manager, but I am trying some ideas that may not require it. I am trying the solutions discussed here: https://www.lexjansen.com/sesug/2016/PA-265_Final_PDF.pdf

[SASKiwi]

Running 100's of SAS sessions simulaneously on a 12-core desktop just isn't realistic. Run some trials to find what is a realistic maximum number of SAS sessions without stressing your computer, then code for that in your program. I would guess around 10-15 sessions would be a reasonable limit, depending on what they are doing. The WAITFOR statement could be useful to pause starting new sessions until an existing session has completed.

[SASKiwi]

Followup - Once you've figured out how many simultaneous SAS sessions is your maximum (let's say 10 for example), re-architect your program to split the zip list into 10 lists, signon to 10 SAS sessions, then loop through each zip list, one per SAS session. 

[RBA]

Thanks, working on it.

[Patrick]

IF you have SAS DI Studio licensed then implement your process via DIS and use the loop transformation for parallel processing a defined number of jobs. 

The DIS loop transformation provides config options to control parallelism (like a number or one active job per available core).

[Patrick]

To add to my previous post: You could also set-up something similar than what DIS does using waitfor any. 

First start all your n-sessions, then whenever a session ends start a new one until you're done. May-be do a bit of Internet research. Very possible that someone wrote a whitepaper how to do this.

[RBA]

Thanks for the suggestion. Indeed, I found this: https://www.lexjansen.com/sesug/2016/PA-265_Final_PDF.pdf