DATA Step, Macro, Functions and more

Running multiple files concurrently instead of consecutively

Reply
Frequent Contributor
Posts: 128

Running multiple files concurrently instead of consecutively

[ Edited ]

Hey,

 

I have no code to share b/c this is just brainstorming.  I'm not even sure how to google this, there is no test code written, i have nothing to share.

 

So i currently have a bunch of files that run once per year.  Ie, it runs for 2003, then runs for 2004, then runs for 2005, etc.  I usually just set it up to run 2003 to 2015 or whatever needs to be run and then just walk away and let it run.  But the other day, in a hurry, i did a save as with a different name and ran one for 2010 and one for 2011 concurrently instead of waiting for them to run consecutively.  And of course, it cut the processing time in half.

 

It should be noted that the source data for the files aren't shared across years.  When run for 2003, it only uses 2003 files, so running two different years at one time doesn't encounter a lock issue.


So i was wondering if running files concurrently is a thing in sas.  Can this be done?  Can i call the same file more than once at the same time with different input data?  Btw, we run our stuff in dos command batch mode because they can take hours, so these files aren't open interactively in sas if that makes a difference.

 

I know i don't want 15 copies of these files b/c there are ocassional updates and that would suck having to maintain multiple versions. I know that theoretically you can copy a file in sas, but i haven't really done it.  Maybe have a driver file that copies the main file, with a new name, passes in the macro values, runs it, then deletes the file after? 

 

Thanks
Megan

 

EDIT: So i was just thinking maybe this would have to be a specific option setting or something?  Even if i could figure out how to copy and save a new file and then have two to pass criteria on to, sas won't run the second one until the first one is done....  ??

 

 

Super Contributor
Posts: 259

Re: Running multiple files concurrently instead of consecutively

How do you tell the "files" to run for 2003?

What do you mean with "files"?

Frequent Contributor
Posts: 128

Re: Running multiple files concurrently instead of consecutively

files=.sas files.

 

We open ultra edit, go to advanced, and select a server that IT has written batch files for.

Trusted Advisor
Posts: 1,378

Re: Running multiple files concurrently instead of consecutively

Relating to your question:

Can i call the same file more than once at the same time with different input data?

the answer is YES as long no one is trying to update the dataset, at that time,  and lock it.

 

Super User
Posts: 5,083

Re: Running multiple files concurrently instead of consecutively

What  you are describing sounds very much like a macro, with parameters defined for the pieces that might change such as the year.  You store one version of the macro, but have 15 short programs that each call the macro (while supplying different values for the parameters).

 

If there is maintenance, just change the macro definition.  The 15 programs don't need to change at all (unless the changes to the macro require changes to the parameters being used when calling the macro).

Frequent Contributor
Posts: 128

Re: Running multiple files concurrently instead of consecutively

Needing 15 different files is what i'm trying to avoid.  Because i have 4 of them technically, each which can be run for 2003-2015, so i really don't want 60+ files floating around.  I want to try and do something where i can generate those files as needed, on the fly, run them, then delete them again.

 

Or something.


I just don't want to manually have to open, save as, run, close, open, save as, run, close, open, save as, run, close, repeat x times.  i just want to have a file where i can enter the start and end year, then have the code generate a new file for each year in the range, and then have something else run each file, but not wait until one is done before the next one starts.

 

I'm hoping someone will have an idea on how to run one, don't wait for it to finish, run the second one right away next, don't wait for it finish, run the third one right away next, etc.

 

I'm just stuck on the concurrently v consecutive issue.  I can find my way around creating what i need.  It's the running order i need help with.

 

Super User
Posts: 5,083

Re: Running multiple files concurrently instead of consecutively

Depending on your tolerance for managing and storing the files, you could group them.  This would be the small file that runs 12 reports consecutively:

 

%report1 (year=2003)

%report2 (year=2003)

%report3 (year=2003)

%report4 (year=2003)

%report1 (year=2004)

%report2 (year=2004)

%report3 (year=2004)

%report4 (year=2004)

%report1 (year=2005)

%report2 (year=2005)

%report3 (year=2005)

%report4 (year=2005)

 

This set of reports would run consecutively, with no intervention after submitting one job.  Concurrently, you could submit a job that would run all four reports for 2006 to 2008, one for 2009 to 2011, etc.

Frequent Contributor
Posts: 128

Re: Running multiple files concurrently instead of consecutively

Everything already runs consecutively.  I'm hoping for concurrently.  More like

 

%report1 (year=2003)

%report1 (year=2004)

%report1 (year=2005)

%report1 (year=2006)

 

And have them run all at the same time, each in its own sas session.  But even like this, it won't run 2004 until 2003 is done.  That's not what i'm looking for.  That's what happens now.

Trusted Advisor
Posts: 1,378

Re: Running multiple files concurrently instead of consecutively

Are all years' data given in one dataset? and you want to deal with specific years?

Then no need to copy the wanted years to another library or name, but just create a view

like in next code:

data my_years / view=my_years;
  set libr.all_years;
       where year in (2011 2012 2013 ); /* adapt to your needs */
run;

then your code should relate to the view:

data results;
   set my_years;
    ... any code ...
run;
Frequent Contributor
Posts: 128

Re: Running multiple files concurrently instead of consecutively

Each year has its own set of datasets to use.  It's medicare claim files, they have hundreds of millions of records.  How they're currently stored and stuff and presented, etc, isn't the issue.

 

I'm just trying to figure out how to run files concurrently or in parallel to each other instead of waiting for one to finish before running a second one.

Contributor
Posts: 61

Re: Running multiple files concurrently instead of consecutively

I have done this for thousands of files: it is more than doable. 

 

 

Here is a snippet out of a paper I wrote on SAS tips/tricks:

 

Sample Code

 

We will be using the following code as a sample. Whether it is test1.sas or test2.sas, it is executing this code. Adjust to fit your needs:

 

%let loops=40000;

 

data test;

   set sashelp.shoes;

   do i=1 to &loops;

      output;

   end;

run;

 

proc summary nway data=test;

   class region;

   var sales;

   output out=test mean= ;

run;

 

Unix

 

data _null_ ;

   file "/users/sasuser/code/ParallelSample/test1.sh" ;

   put "'/sas913/SAS_9.1/sas' -sysin '/users/sasuser/code/ParallelSample/Test1.sas' -log '/users/sasuser /logs/Test1.log'";

 

   file "/users/sasuser /code/ParallelSample/test2.sh" ;

   put "'/sas913/SAS_9.1/sas' -sysin '/users/sasuser/code/ParallelSample/Test2.sas' -log '/home/achurc1/logs/Test2.log'";

run;

 

 

systask kill chmod1 chmod2;

systask command "chmod a+x /users/sasuser/code/ParallelSample/test1.sh &" taskname=chmod1;

systask command "chmod a+x /users/sasuser/code/ParallelSample/test2.sh &" taskname=chmod2;

waitfor _all_ chmod1 chmod2;

 

systask kill test1 test2;

systask command "sh /users/sasuser/code/ParallelSample/test1.sh" taskname=test1;

systask command "sh /users/sasuser/code/ParallelSample/test2.sh" taskname=test2;

waitfor _all_ test1 test2;

 

 

Windows

 

options noxwait noxsync;

%let sasexe_options = -nosplash ;

 

data _null_ ;

   file "c:\temp\test1.cmd" ;

   put "sas -sysin 'c:\temp\test1.sas' -nosplash -log 'c:\temp\test1.log'";

 

   file "c:\temp\test2.cmd" ;

   put "sas -sysin 'c:\temp\test2.sas' -nosplash -log 'c:\temp\test2.log'";

 

run;

 

 

systask kill test1 test2;

systask command "c:\temp\test1.cmd" taskname=test1;

systask command "c:\temp\test2.cmd" taskname=test2;

waitfor _all_ test1 test2;

 

Frequent Contributor
Posts: 128

Re: Running multiple files concurrently instead of consecutively

[ Edited ]

Ok, did i do this right?

/*Test file*/

options noxwait noxsync;
%let sasexe_options = -nosplash;

data _null_;
    %let start=2014; %let end=2014;
    file "K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_&start..cmd";
    put "sas -sysin 'K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_test.sas' -nosplash -log 'K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_test_&start..log'";

    %let start=2015; %let end=2015;
    file "K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_&start..cmd";
    put "sas -sysin 'K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_test.sas' -nosplash -log 'K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_test_&start..log'";
run;

systask kill k4;
systask command "K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_2014.cmd" taskname=k4_2014;
systask command "K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_2015.cmd" taskname=k4_2015;
waitfor _all_ k4_2014 k4_2015;




I want to run K4 for 2014 and 2015.  I'm getting an error in the log saying "Specified taskname is already in use."  I wasn't sure i had the setup right though.

 

 EDIT: Just closed sas and re-opened, ran again from scrach.  I'm no longer getting the error, but it now says NOTE: Task "k4_2014" produced no LOG/Output." (two notes, one for each year, saying the same thing).  But it didn't run.  I know it didn't run b/c one year usually takes 2 hours.  This all went in the log in 2 seconds.

 

EDIT2: K4_test.sas is the name of the sas file that i want to run for each year.  I was unsure how to make files unique so they were kept, so i tried to add the year onto them, but i dk if that impacts why it didn't run.

 

Contributor
Posts: 61

Re: Running multiple files concurrently instead of consecutively

[ Edited ]

Megan,

 

I don't have SAS for testing so let me try and wing this.

 

First of all, I use macros but I also am not a fan of them: they are merely code substitution and there are other ways to handle that.I do a bit of a blend here w/o complicating things, IMO.

 

options noxwait noxsync;
%let start = 2003;

%let end=2016;

 

data _null_ ;

     file 'c:\temp\jobsToRun.sas';

     * put 'systask kill _all_' ; //NOT NEEDED YET UNTIL WORKING

     do i=&start to &end;

         put "sas -sysin 'K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_" year +(-1)".sas' -nosplash";     

     end;

      put "waitfor _all_;";

run;

 

%include 'c:\temp\jobsToRun.sas' ;

 

I cannot test this since I wrote it in this HTML editor but that is directionally where you need to go. If the jobs hang, kill them in Task Manager. No need to restart SAS but it effectively accomplishes the same thing (you kill the parent thread).

Trusted Advisor
Posts: 1,378

Re: Running multiple files concurrently instead of consecutively

@AlanC, I don't see you use i in your loop code:

do i=&start to &end;
         put "sas -sysin 'K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_" year +(-1)".sas' -nosplash";     
     end;
Contributor
Posts: 61

Re: Running multiple files concurrently instead of consecutively

See the replacement code I posted. it is better.

Ask a Question
Discussion stats
  • 21 replies
  • 181 views
  • 0 likes
  • 5 in conversation