topic Re: How to run a script on several directories? in SAS Programming

How to run a script on several directories?

annapechenina — Thu, 25 Oct 2018 20:03:40 GMT

Hi!

I have several directories (50) with data files and I would like to run the same script on all of them without editing directory path for each directory. How could I do that? What would be the code?

Using SAS 9.4. (English)

Thanks,

Anna

Re: How to run a script on several directories?

Reeza — Thu, 25 Oct 2018 20:12:31 GMT

What kind of script? Is it a macro, stored process, a SAS program? How would you call it for a single folder? We need a lot more details. It could be as simple as using a macro here to list all files and then use CALL EXECUTE to run the macro for each file. If it's reading all files in for example, you could just create a list of the locations and import all at once (if same format) or import them each into their own data.

Re: How to run a script on several directories?

annapechenina — Thu, 25 Oct 2018 20:20:22 GMT

Thank you for your questions, Reeza!

It is a SAS program and it performs manipulations with data sets within a directory. How would I tell SAS to go to run this script on each directory I have? Now I have to manually change the path within the script. I would like to automate this process.

I am open to non SAS solutions like powershell or command line.

Re: How to run a script on several directories?

ballardw — Thu, 25 Oct 2018 20:22:57 GMT

Are the "files" SAS data sets?

A SAS library can reference multiple directories, though I would say 50 at a time might be problematic due to possible length and the response to the following question:

Do any of the files share the same name other than being in different directories?

Re: How to run a script on several directories?

annapechenina — Thu, 25 Oct 2018 20:26:01 GMT

ballardw,

all the files are "SAS" data sets and they have the same names, the only thing different is directory name.

Re: How to run a script on several directories?

SuryaKiran — Thu, 25 Oct 2018 20:32:18 GMT

Are you trying to find all the .sas7bdat in a directory and sub-directory and execute a program?

Below is the code that used to list all the files withing a directory. Once you get the list then you can execute them.

%macro drive(dir,ext);                                                                                                                  
  %local filrf rc did memcnt name i;                                                                                                    
                                                                                                                                        
  /* Assigns a fileref to the directory and opens the directory */                                                           
  %let rc=%sysfunc(filename(filrf,&dir));                                                                                               
  %let did=%sysfunc(dopen(&filrf));                                                                                                     
                                                                                                                                        
  /* Make sure directory can be open */                                                                                                 
  %if &did eq 0 %then %do;                                                                                                              
   %put Directory &dir cannot be open or does not exist;                                                                                
   %return;                                                                                                                             
  %end;                                                                                                                                 
                                                                                                                                        
   /* Loops through entire directory */                                                                                                 
   %do i = 1 %to %sysfunc(dnum(&did));                                                                                                  
                                                                                                                                        
     /* Retrieve name of each file */                                                                                                   
     %let name=%qsysfunc(dread(&did,&i));                                                                                               
                                                                                                                                        
     /* Checks to see if the extension matches the parameter value */                                                                   
     /* If condition is true print the full name to the log        */                                                                   
      %if %qupcase(%qscan(&name,-1,.)) = %upcase(&ext) %then %do;                                                                       
        %put &dir\&name;                                                                                                                
      %end;                                                                                                                             
     /* If directory name call macro again */                                                                                           
      %else %if %qscan(&name,2,.) = %then %do;                                                                                          
        %drive(&dir\%unquote(&name),&ext)                                                                                               
      %end;                                                                                                                             
                                                                                                                                        
   %end;                                                                                                                                
                                                                                                                                        
  /* Closes the directory and clear the fileref */                                                                                      
  %let rc=%sysfunc(dclose(&did));                                                                                                       
  %let rc=%sysfunc(filename(filrf));                                                                                                    
                                                                                                                                        
%mend drive;                                                                                                                            
                                                                                                                                        
/* First parameter is the directory of where your files are stored. */                                                                  
/* Second parameter is the extension you are looking for.           */                                                                  
%drive(c:\temp,sas7bdat)

Re: How to run a script on several directories?

Tom — Thu, 25 Oct 2018 21:57:56 GMT

I don't know what you mean by "script". Normally you call a file with SAS code in it a program. How easy it is to convert to working on different directories depends a lot on what exactly it does.

Let's just create a trivial example of a program. One that just runs a LIBNAME statement.

libname mydata 'C:\mydata';

To prepare to make it dynamic change the hard coded directory specification into a macro variable reference.

Remember that macro triggers ( & % ) are not processed inside single quotes.

%let mydir=C:\mydata;
libname mydata "&mydir.";

The next step it so convert the logic that processes one directory into a macro. You can change your macro variable into a parameter to the macro instead. And then pass in the actual directory path when you call the macro.

%macro mymacro(mydir);
libname mydata "&mydir.";
%mend mymacro;

%mymacro(C:\mydata)

Now you could just replicate that macro call 50 times and type in the other directory names.

Or if you have the list of directories in a dataset then use that to generate the macro calls.

So for example if you had a dataset named MYLIST with a variable named MYDIR you could use this data step to generate one macro call for each observation.

data _null_;
  set mylist;
  call execute(cats('%nrstr(mymacro)(',mydir,')'));
run;

The %NRSTR() will prevent SAS from trying the expand the macro when CALL EXECUTE pushes the code onto the stack to run after the data step. Instead the macro will be called when SAS pulls the stored lines of code back off of the stack generated by the CALL EXECUTE() statements running.

The next level of automation is to generate the list of directories automatically. Usually that is most easily done if you can access operating system commands. For example on a PC you might read the results of a DIR commmand. Or on Unix you might read the results of a ls or find command.

data mylist;
  infile 'dir /b/ad c:\mydata\*' pipe truncover;
  input mydir $256. ;
run;

Re: How to run a script on several directories?

Shmuel — Thu, 25 Oct 2018 20:53:49 GMT

You can list all data-sets having same name and use a loop to execute your script supplying the libname into call execute.

Just a hint:

proc sql;
  create table list_of_libraries
  as select libname  /* , memname  */
  where memname = <dataset_name>
  from dictionary.tables;
quit.

Re: How to run a script on several directories?

annapechenina — Thu, 25 Oct 2018 21:36:58 GMT

Yes!! This is it! Thank you very much! I will report back after I test it.