Running multiple files concurrently instead of consecutively

[AlanC]

Here is more of a macro version:

%macro runMyJobs(start=,end=);
%do ;
%let options=options noxwait noxsync;
%let sasexe_options = -nosplash ;
&options ;
data _null_ ;
%do i = &start %to &end ;
file 'c:\temp\job_&i..cmd' ;
put "sas -sysin 'c:\temp\job_&i..sas' &sasexe_options " @ ;
put "-log 'c:\temp\job_&i.log'";
%end ;
run;
%do i = &start %to &end ;
systask kill sas&i ;
systask command "c:\temp\job_&i..cmd" taskname=sas&i;
%end ;
waitfor _all_
;
%end ;
%mend runMyJobs;

%runMyJobs(start=2003,end=2016);

[MeganE]

Thanks.  Just so i'm clear...  Does this replacement code run the original file?  Or does it assume it's already got been copied for a year specific run?  I'm not seeing any reference to k4.sas in there, so i'm not sure.

Suppose i run it for 2014 and 2015 only to start:

%macro runMyJobs(start=,end=);
%do ;
%let options=options noxwait noxsync;
%let sasexe_options = -nosplash ;
&options ;
data _null_ ;
%do i = &start %to &end ;
file 'c:\temp\job_&i..cmd' ;                                           <-- is the .cmd name anything i want it to be?  or does it need to match k4 as the file name?
put "sas -sysin 'c:\temp\job_&i..sas' &sasexe_options" @ ;     <-- what is job_*?  Is this K4?  Or does this line assume i've already created separate K4s per year to be run and named them job_2014, job_2015?
put "-log 'c:\temp\job_&i.log'";                                     <-- again, is this any name i want it to be?
%end ;
run;
%do i = &start %to &end ;
systask kill sas&i ;
systask command "c:\temp\job_&i..cmd" taskname=sas&i;
%end ;
waitfor _all_
;
%end ;
%mend runMyJobs;

%runMyJobs(start=2014,end=2015);

[MeganE]

How about this.  Using these two file names, assuming for the sake of the code, they already exist, how do i set up your macro to run both of them at the same time in different sas sessions?

c:\temp\k4_2014.sas

c:\temp\k4_2015.sas

[Shmuel]

All you need is just to replace the string 'job' into 'k' in the macro, in PUT & SYSTASK staements.

[MeganE]

Sorry, i've never seen this kind of code before.  I need it spelled out.  Stuff you guys are taking for granted, i have no clue about and will miss it.

In fact, i did miss it b/c it didn't work.

K4_test_2014.sas

K4_test_2015.sas

Both exist and are in the \copyruns\folder.

EDIT:  Two .cmd files were created in the right location with the right name that contain all of that sas -sysin -log stuff.

[Shmuel]

I have copied the code you posted (using @AlanC macro) and addapted it:

%macro runMyJobs(start=,end=);
  %do ;
     %let options=options noxwait noxsync;  /* those options enable parallel submition and execution */
     %let sasexe_options = -nosplash ;
     &options ;
	 
     data _null_ ;
      %do i = &start %to &end ;
         file 'c:\temp\k&i._test.cmd' ;    /* this is the .cmd name and you can addapt it */
		       /* next line submits job, a new sas session to excute your program */
			   /* the @ means a continuation on next line */
         put "sas -sysin 'c:\temp\k&i._test.sas' &sasexe_options" @ ;    
         put "-log 'c:\temp\k&i._test.log'";  /* this is the log file to be created, can be addapted */
      %end ;
    run;
   %do i = &start %to &end ;
      systask kill sas&i ;
      systask command "c:\temp\k&i._test.cmd" taskname=sas&i;
    %end ;
     waitfor _all_
     ;
  %end ;
%mend runMyJobs;
%runMyJobs(start=2014,end=2015);

I hope this will clarify somewhat the macro to you.

In case it do not work as you want then:

- post the code you used using the {i} icon

- post the log you got - either using {i} icon or as attached text file.

- describe your issue shortly

- and please do not attach screenshots or pictures as we don't like to type text from them.

[AlanC]

Megan,

Let me explain what is happening here so it is clear.

1. The %lets are merely macro statements that are used later to set the environment up for parallel execution and to keep SAS from splattering your screen with loads of windows saying a new job is running. You don't need them done like that if you want to just hard-code the values:

Delete the %lets and use this:

...

options noxwait noxsync;

data _null_ ;
%do i = &start %to &end ;
file 'c:\temp\job_&i..cmd' ;  

file name?
put "sas -sysin 'c:\temp\job_&i..sas' -nosplash"

...

Your job is simple. imagine doing this for hundreds of jobs at the same time with each one opening up its own window. That is why we have the nosplash.

2. We create a null dataset that does nothing except write out some files. Think of this as an easy to use way of doing code substitution w/o using macros. It will create a bunch of cmd files. If you want to know what is created, open up one of the cmd files and that is what will be executed on the command line when that code is called. So, why use this vs macros? This allows for easier debugging.

3. the final part (systask commands) Launches all of the cmd files at the same time but tells the main program to wait until they all complete. Now, if your code truly runs 100% standalone and doesn't need to wait, drop the waitfor. The waitfor is included so that additional code can be run after it if the parallel jobs' output is needed for later steps. 

This is complex SAS code so I probably should explain it more in my document. 

Some notes:

1. Don't overload the CPUs. The rule of thumb I use is 2x the number of cores is the maximum number of jobs to submit. 

2. Don't overwhelm the I/O channels. If every jobs reads from 1 disk and you saturate the I/O, it will decrease the wall time. Stagger the I/O channels on both the input and output. Read from different disks, write to different disks. This is a general rule so test.