using rsubmit in a loop

[Ullsokk]

I am trying to create a macro loop that will loop through a table and run a rsubmit on a series of sas-programs. I am trying to emulate how the fork transformation works in DI-studio, but with a dynamic setup that allows me to run any number of sas programs in each remote submit\fork. My control table has one line for each sas program with a file path that is run using %incude.  This is my code in progress:

%macro runABT();
proc sql noprint;
select max(row) into :n from controltable;
quit;

%do i = 1 %to &n.;
	%put session = session&i.;

	proc sql noprint;
	select sascode into :sascode from controltable where row = &i.;
	quit;

	signon session&i. sascmd="!sascmd -nosyntaxcheck -noterminal"  ;
	%syslput sascode=&sascode.;
	rsubmit session&i. wait=no;
	%put sascode= &sascode;
		%include "&sascode.";
	endrsubmit; 
	signoff session&i.; 
%end;
%mend runABT;
%runABT;

But with this approach, the second session does not seems to start until the first is finished. Is there a way to have the loop start the second session before the first is finished?

[s_lassen]

Your can put a WAIT=NO option on the SIGNON or the RSUBMIT statements. That way, your local session will not wait for the remote sessions to end. If you want to have your macro wait until all your parallel processes are done after submitting them, you can put a WAITFOR _ALL_; statement at the end of the macro.

See the documentation here: WAITFOR , RSUBMIT

[Ullsokk]

I have wait=no on the rsubmit, I thought that would do the trick, however, it doesn't behave as I expected. Will try out the waitfor _all_. Will try out putting a wait=no on the signon as well.

[SASKiwi]

@Ullsokk - In my experience, having remote SAS sessions and RSUBMITs inside a macro can result in unexpected behaviour. The reason is the remote sessions are not aware they are running inside a macro in a parent SAS session.

As @Tom has suggested, get your processing working in open code first. Then perhaps consider a different approach for the looping. I've used CALL EXECUTE successfully for this. 

[Ullsokk]

I tried wait=no after the signon. But the macro doesn't end, even though both jobs are finished (checked the output tables). Maybe I can't signoff the session untill the waitfor = _all_ has resolved?

[s_lassen]

Sorry, didn't read the documentation carefully enough, thought the WAITFOR syntax was similar to the one used with SASTASK (submitting a SAS session on the same computer). It seems you have to specify the names of the sessions you want to wait for, maybe you can try something like this at the end of your macro:

waitfor _ALL_ %do i = 1 %to &n; session&i %end;;

[Tom]

Try it without the loop first.  Then you will know how to code the loop.

Skip the dynamic definition of SASCODE also.

signon session1 ....;
%syslput sascode=row1 / remote=session1;
rsubmit session1 wait=no;
%include("&sascode");
endrsumbit;
signon session2 ....;
%syslput sascode=row2 / remote=session2;
rsubmit session2 wait=no;
%include("&sascode");
endrsubmit;
waitfor _all_ session1 session2;
signoff session1;
signoff session2;

[ChrisNZ · Accepted solution]

I'd signon before the main loop. Something like.

%macro runABT();
%local i n;
proc sql noprint;
  select max(ROW) into :n from CONTROLTABLE;
  select SASCODE into :saspgm1-:saspgm999 from CONTROLTABLE;
quit;
%do i = 1 %to &n.;
  signon session&i. sascmd="!sascmd -nosyntaxcheck -noterminal"  ;
  %syslput sascode=&&saspgm&i / remote=session&i.;
%end;
%do i = 1 %to &n.;
  rsubmit session&i. wait=no;
    %put pgm=&&saspgm&i;
    %include "&&saspgm&i";
  endrsubmit; 
%end;
%do i = 1 %to &n.;
  signoff session&i.; 
%end;
%mend runABT;
%runABT;

to ensure the signon/off steps don't spoil the waiting.

Add  waitfor _all_ next.

[ChrisNZ]

Oops, just saw I got my remote macro variable name confused. Please correct accordingly.

[ScottBass]

I've only skimmed your post, so I may misunderstand what you're trying to do.  And I can't tell if you're using rsubmit to submit code on a remote server, or the current one.

However, have a look at 

https://github.com/scottbass/SAS/blob/master/Macro/RunAll.sas and https://github.com/scottbass/SAS/blob/master/Macro/RunAll_ControlTable.sas 

Perhaps they will help with what you're trying to do?

[Ullsokk]

Thanks, that looks really interesting. I am running code om the current server. I basically want all programs to run at the same time, in different sessions.

[ScottBass]

---

@Ullsokk wrote:

Thanks, that looks really interesting. I am running code om the current server. I basically want all programs to run at the same time, in different sessions.

---

My code runs SAS in batch mode, i.e. sas.exe -sysin myprogram.sas.  It uses the first SAS session as the "scripting engine" to launch multiple SAS sessions, and abort if upstream dependencies fail.  When I used it I had SAS DMS on the laptop, so the scripting engine ran in interactive mode, but the remaining sessions ran in batch.  As much as you can get batch on windows - there were a bunch of icons in the taskbar.  They don't run in "true" background as you can get with *nix.

If you just want to run all programs at once, either modify the macro or simply set all programs to the same "level", i.e. all "1".

Note that more is not necessarily better; when I ran this on (an old) two core CPU laptop, running more than two programs at a time increased the total execution run time for all the programs.  YMMV.  So you may want to experiment with the max_threads setting.