BookmarkSubscribeRSS Feed
MeganE
Pyrite | Level 9

Hey,

 

I have no code to share b/c this is just brainstorming.  I'm not even sure how to google this, there is no test code written, i have nothing to share.

 

So i currently have a bunch of files that run once per year.  Ie, it runs for 2003, then runs for 2004, then runs for 2005, etc.  I usually just set it up to run 2003 to 2015 or whatever needs to be run and then just walk away and let it run.  But the other day, in a hurry, i did a save as with a different name and ran one for 2010 and one for 2011 concurrently instead of waiting for them to run consecutively.  And of course, it cut the processing time in half.

 

It should be noted that the source data for the files aren't shared across years.  When run for 2003, it only uses 2003 files, so running two different years at one time doesn't encounter a lock issue.


So i was wondering if running files concurrently is a thing in sas.  Can this be done?  Can i call the same file more than once at the same time with different input data?  Btw, we run our stuff in dos command batch mode because they can take hours, so these files aren't open interactively in sas if that makes a difference.

 

I know i don't want 15 copies of these files b/c there are ocassional updates and that would suck having to maintain multiple versions. I know that theoretically you can copy a file in sas, but i haven't really done it.  Maybe have a driver file that copies the main file, with a new name, passes in the macro values, runs it, then deletes the file after? 

 

Thanks
Megan

 

EDIT: So i was just thinking maybe this would have to be a specific option setting or something?  Even if i could figure out how to copy and save a new file and then have two to pass criteria on to, sas won't run the second one until the first one is done....  ??

 

 

21 REPLIES 21
andreas_lds
Jade | Level 19

How do you tell the "files" to run for 2003?

What do you mean with "files"?

MeganE
Pyrite | Level 9

files=.sas files.

 

We open ultra edit, go to advanced, and select a server that IT has written batch files for.

Shmuel
Garnet | Level 18

Relating to your question:

Can i call the same file more than once at the same time with different input data?

the answer is YES as long no one is trying to update the dataset, at that time,  and lock it.

 

Astounding
PROC Star

What  you are describing sounds very much like a macro, with parameters defined for the pieces that might change such as the year.  You store one version of the macro, but have 15 short programs that each call the macro (while supplying different values for the parameters).

 

If there is maintenance, just change the macro definition.  The 15 programs don't need to change at all (unless the changes to the macro require changes to the parameters being used when calling the macro).

MeganE
Pyrite | Level 9

Needing 15 different files is what i'm trying to avoid.  Because i have 4 of them technically, each which can be run for 2003-2015, so i really don't want 60+ files floating around.  I want to try and do something where i can generate those files as needed, on the fly, run them, then delete them again.

 

Or something.


I just don't want to manually have to open, save as, run, close, open, save as, run, close, open, save as, run, close, repeat x times.  i just want to have a file where i can enter the start and end year, then have the code generate a new file for each year in the range, and then have something else run each file, but not wait until one is done before the next one starts.

 

I'm hoping someone will have an idea on how to run one, don't wait for it to finish, run the second one right away next, don't wait for it finish, run the third one right away next, etc.

 

I'm just stuck on the concurrently v consecutive issue.  I can find my way around creating what i need.  It's the running order i need help with.

 

Astounding
PROC Star

Depending on your tolerance for managing and storing the files, you could group them.  This would be the small file that runs 12 reports consecutively:

 

%report1 (year=2003)

%report2 (year=2003)

%report3 (year=2003)

%report4 (year=2003)

%report1 (year=2004)

%report2 (year=2004)

%report3 (year=2004)

%report4 (year=2004)

%report1 (year=2005)

%report2 (year=2005)

%report3 (year=2005)

%report4 (year=2005)

 

This set of reports would run consecutively, with no intervention after submitting one job.  Concurrently, you could submit a job that would run all four reports for 2006 to 2008, one for 2009 to 2011, etc.

MeganE
Pyrite | Level 9

Everything already runs consecutively.  I'm hoping for concurrently.  More like

 

%report1 (year=2003)

%report1 (year=2004)

%report1 (year=2005)

%report1 (year=2006)

 

And have them run all at the same time, each in its own sas session.  But even like this, it won't run 2004 until 2003 is done.  That's not what i'm looking for.  That's what happens now.

Shmuel
Garnet | Level 18

Are all years' data given in one dataset? and you want to deal with specific years?

Then no need to copy the wanted years to another library or name, but just create a view

like in next code:

data my_years / view=my_years;
  set libr.all_years;
       where year in (2011 2012 2013 ); /* adapt to your needs */
run;

then your code should relate to the view:

data results;
   set my_years;
    ... any code ...
run;
MeganE
Pyrite | Level 9

Each year has its own set of datasets to use.  It's medicare claim files, they have hundreds of millions of records.  How they're currently stored and stuff and presented, etc, isn't the issue.

 

I'm just trying to figure out how to run files concurrently or in parallel to each other instead of waiting for one to finish before running a second one.

AlanC
Barite | Level 11

I have done this for thousands of files: it is more than doable. 

 

 

Here is a snippet out of a paper I wrote on SAS tips/tricks:

 

Sample Code

 

We will be using the following code as a sample. Whether it is test1.sas or test2.sas, it is executing this code. Adjust to fit your needs:

 

%let loops=40000;

 

data test;

   set sashelp.shoes;

   do i=1 to &loops;

      output;

   end;

run;

 

proc summary nway data=test;

   class region;

   var sales;

   output out=test mean= ;

run;

 

Unix

 

data _null_ ;

   file "/users/sasuser/code/ParallelSample/test1.sh" ;

   put "'/sas913/SAS_9.1/sas' -sysin '/users/sasuser/code/ParallelSample/Test1.sas' -log '/users/sasuser /logs/Test1.log'";

 

   file "/users/sasuser /code/ParallelSample/test2.sh" ;

   put "'/sas913/SAS_9.1/sas' -sysin '/users/sasuser/code/ParallelSample/Test2.sas' -log '/home/achurc1/logs/Test2.log'";

run;

 

 

systask kill chmod1 chmod2;

systask command "chmod a+x /users/sasuser/code/ParallelSample/test1.sh &" taskname=chmod1;

systask command "chmod a+x /users/sasuser/code/ParallelSample/test2.sh &" taskname=chmod2;

waitfor _all_ chmod1 chmod2;

 

systask kill test1 test2;

systask command "sh /users/sasuser/code/ParallelSample/test1.sh" taskname=test1;

systask command "sh /users/sasuser/code/ParallelSample/test2.sh" taskname=test2;

waitfor _all_ test1 test2;

 

 

Windows

 

options noxwait noxsync;

%let sasexe_options = -nosplash ;

 

data _null_ ;

   file "c:\temp\test1.cmd" ;

   put "sas -sysin 'c:\temp\test1.sas' -nosplash -log 'c:\temp\test1.log'";

 

   file "c:\temp\test2.cmd" ;

   put "sas -sysin 'c:\temp\test2.sas' -nosplash -log 'c:\temp\test2.log'";

 

run;

 

 

systask kill test1 test2;

systask command "c:\temp\test1.cmd" taskname=test1;

systask command "c:\temp\test2.cmd" taskname=test2;

waitfor _all_ test1 test2;

 

https://github.com/savian-net
MeganE
Pyrite | Level 9

Ok, did i do this right?

/*Test file*/

options noxwait noxsync;
%let sasexe_options = -nosplash;

data _null_;
    %let start=2014; %let end=2014;
    file "K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_&start..cmd";
    put "sas -sysin 'K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_test.sas' -nosplash -log 'K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_test_&start..log'";

    %let start=2015; %let end=2015;
    file "K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_&start..cmd";
    put "sas -sysin 'K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_test.sas' -nosplash -log 'K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_test_&start..log'";
run;

systask kill k4;
systask command "K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_2014.cmd" taskname=k4_2014;
systask command "K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_2015.cmd" taskname=k4_2015;
waitfor _all_ k4_2014 k4_2015;




I want to run K4 for 2014 and 2015.  I'm getting an error in the log saying "Specified taskname is already in use."  I wasn't sure i had the setup right though.

 

 EDIT: Just closed sas and re-opened, ran again from scrach.  I'm no longer getting the error, but it now says NOTE: Task "k4_2014" produced no LOG/Output." (two notes, one for each year, saying the same thing).  But it didn't run.  I know it didn't run b/c one year usually takes 2 hours.  This all went in the log in 2 seconds.

 

EDIT2: K4_test.sas is the name of the sas file that i want to run for each year.  I was unsure how to make files unique so they were kept, so i tried to add the year onto them, but i dk if that impacts why it didn't run.

 

AlanC
Barite | Level 11

Megan,

 

I don't have SAS for testing so let me try and wing this.

 

First of all, I use macros but I also am not a fan of them: they are merely code substitution and there are other ways to handle that.I do a bit of a blend here w/o complicating things, IMO.

 

options noxwait noxsync;
%let start = 2003;

%let end=2016;

 

data _null_ ;

     file 'c:\temp\jobsToRun.sas';

     * put 'systask kill _all_' ; //NOT NEEDED YET UNTIL WORKING

     do i=&start to &end;

         put "sas -sysin 'K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_" year +(-1)".sas' -nosplash";     

     end;

      put "waitfor _all_;";

run;

 

%include 'c:\temp\jobsToRun.sas' ;

 

I cannot test this since I wrote it in this HTML editor but that is directionally where you need to go. If the jobs hang, kill them in Task Manager. No need to restart SAS but it effectively accomplishes the same thing (you kill the parent thread).

https://github.com/savian-net
Shmuel
Garnet | Level 18

@AlanC, I don't see you use i in your loop code:

do i=&start to &end;
         put "sas -sysin 'K:\Projects\USRDS\Analysis\ADR\2017\ReferenceTable\K_econ\SASCode\SheetCode\k4_" year +(-1)".sas' -nosplash";     
     end;
AlanC
Barite | Level 11

See the replacement code I posted. it is better.

https://github.com/savian-net

SAS Innovate 2025: Call for Content

Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!

Submit your idea!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 21 replies
  • 5607 views
  • 1 like
  • 5 in conversation