BookmarkSubscribeRSS Feed
☑ This topic is solved. Need further help from the community? Please sign in and ask a new question.
RBA
Fluorite | Level 6 RBA
Fluorite | Level 6

folks, I am using sas / connect in asynchronous mode, creating several hundred jobs with a loop. My challenge is that sas is not limiting the number of jobs it process simultaneously. It starts all jobs the CPU can take, engaging all available cores and saturating the hard drive (each job needs to create a copy of a dataset). I would like to be able to specify the maximum number of processes that are executed at any point in time, but SAS is ignoring the cpucount option.

I am using SAS is 9.4, SAS/STAT 15.2 on a 12-core/Windows 10 desktop and the relevant code is as follows:

 

options threads cpucount=4;

%let rc=%sysfunc(grdsvc_enable(_all_, resource=SASApp)); options autosignon; %macro mymacro; /* loop over zip3's */ %do i=1 %to %sysfunc(countw(&zip3list)) ; %let thisZip30=%scan(&zip3list,&i); %let taskname = t&thisZip30; %SYSLPUT year=&year./REMOTE=&taskname; %SYSLPUT thisZip3=&thisZip30./REMOTE=&taskname; rsubmit &taskname wait=no connectpersist=no inheritlib=(ineq4 onlinek); %nrstr(%%)PUT ******* zip3=&thisZip3 **************; /* extract data for zip3 and upc */ data thiZip3train&thisZip3; set ineq4.promediosWeeRetTrain&year; where store_zip3="&thisZip3"; run; /* do stuff with the extracted data */

PROC DATASETS lib=work;
delete thiZip3train&thisZip3;
quit;
RUN; endrsubmit; %end; waitfor _all_; signoff _all_ ; %mend mymacro; %mymacro;
1 ACCEPTED SOLUTION

Accepted Solutions
LinusH
Tourmaline | Level 20

First a little clarification about CPUCOUNT. This option controls no of cores available for the current SAS session. And it's valid in an OPTIONS statement. But it does not control no of SAS processes that can execute simultaneously. SAS "submits" these processes for the OS to manage.

It sopunds liek you need some kind of OS process queuing facility. This is not part of Base SAS or SAS/CONNECT (it's part of the SAS Grid Manager offering though).

If you don't have access to such software, I think you need to build the corresponding logic in your macro, and submit some of your remote sessions synchronously.

Data never sleeps

View solution in original post

8 REPLIES 8
LinusH
Tourmaline | Level 20

First a little clarification about CPUCOUNT. This option controls no of cores available for the current SAS session. And it's valid in an OPTIONS statement. But it does not control no of SAS processes that can execute simultaneously. SAS "submits" these processes for the OS to manage.

It sopunds liek you need some kind of OS process queuing facility. This is not part of Base SAS or SAS/CONNECT (it's part of the SAS Grid Manager offering though).

If you don't have access to such software, I think you need to build the corresponding logic in your macro, and submit some of your remote sessions synchronously.

Data never sleeps
RBA
Fluorite | Level 6 RBA
Fluorite | Level 6

I appreciate the concise explanation. It helped me understand the problem. I am afraid I do not have the license for the grid manager, but I am trying some ideas that may not require it. I am trying the solutions discussed here: https://www.lexjansen.com/sesug/2016/PA-265_Final_PDF.pdf

 

SASKiwi
PROC Star

Running 100's of SAS sessions simulaneously on a 12-core desktop just isn't realistic. Run some trials to find what is a realistic maximum number of SAS sessions without stressing your computer, then code for that in your program. I would guess around 10-15 sessions would be a reasonable limit, depending on what they are doing. The WAITFOR statement could be useful to pause starting new sessions until an existing session has completed.

SASKiwi
PROC Star

Followup - Once you've figured out how many simultaneous SAS sessions is your maximum (let's say 10 for example), re-architect your program to split the zip list into 10 lists, signon to 10 SAS sessions, then loop through each zip list, one per SAS session. 

RBA
Fluorite | Level 6 RBA
Fluorite | Level 6
Thanks, working on it.
Patrick
Opal | Level 21

IF you have SAS DI Studio licensed then implement your process via DIS and use the loop transformation for parallel processing a defined number of jobs. 

The DIS loop transformation provides config options to control parallelism (like a number or one active job per available core).

Patrick
Opal | Level 21

To add to my previous post: You could also set-up something similar than what DIS does using waitfor any

First start all your n-sessions, then whenever a session ends start a new one until you're done. May-be do a bit of Internet research. Very possible that someone wrote a whitepaper how to do this.

RBA
Fluorite | Level 6 RBA
Fluorite | Level 6

Thanks for the suggestion. Indeed, I found this: https://www.lexjansen.com/sesug/2016/PA-265_Final_PDF.pdf

SAS Innovate 2025: Call for Content

Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!

Submit your idea!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 8 replies
  • 1403 views
  • 5 likes
  • 4 in conversation