How can I scan for UNIX folders that contain 2 layers of dynamic folders with special filter?

StickyRoll — Sun, 05 Dec 2021 17:33:59 GMT

I have a UNIX server that contain the following folder structure:

Root: /sas/config/master.data/

1st Layer (Possible folders):

/sas/config/master.data/12312016
/sas/config/master.data/06302017
/sas/config/master.data/08312017
/sas/config/master.data/johncena
/sas/config/master.data/brock123
/sas/config/master.data/elonmusk

2nd Layer 

/sas/config/master.data/johncena/09302019
/sas/config/master.data/johncena/04302012
/sas/config/master.data/brock123/02282013
/sas/config/master.data/elonmusk/01312020
/sas/config/master.data/elonmusk/03312021

Do notice that only those english name subfolders (eg. johncena) are being pointed out here assuming that it contains a MMDDYYYY folder inside. I can achieve this by checking it with anydate function (eg. folderdate=input(fname,anydtdte.);). Also, the folder master.data contains a dot (may cause issue if i understand correctly).

By looking at the folder structure above, we can tell that only the root directory is fixed. After the root directory, there are 2 more levels that are dynamic which require further looping.

Ultimately, I will need to store this into a SAS Dataset so that I can assign it into macro for generating mv -f command to either remove it or move or copy to another directory within the UNIX server.

Now my trouble is, I have a script that I referred to SAS documentation but I couldn't find the right way and logic to scan unix directory dynamically with a check for 2 additional layers (refer to 2nd Layer explanation.) despite me knowing that there is a function to check if the folder contains a valid date format or not.

The UNIX directory scan script as below. I can execute it but it will only show in log, not storing into a table with the full directory for me to further process. Would this be the most efficient script for what I am aimed to do?

%macro drive(dir,ext);                                                                                                                  
  %local filrf rc did memcnt name i;                                                                                                    
                                                                                                                                        
  /* Assigns a fileref to the directory and opens the directory */                                                           
  %let rc=%sysfunc(filename(filrf,&dir));                                                                                               
  %let did=%sysfunc(dopen(&filrf));                                                                                                     
                                                                                                                                        
  /* Make sure directory can be open */                                                                                                 
  %if &did eq 0 %then %do;                                                                                                              
   %put Directory &dir cannot be open or does not exist;                                                                                
   %return;                                                                                                                             
  %end;                                                                                                                                 
                                                                                                                                        
   /* Loops through entire directory */                                                                                                 
   %do i = 1 %to %sysfunc(dnum(&did));                                                                                                  
                                                                                                                                        
     /* Retrieve name of each file */                                                                                                   
     %let name=%qsysfunc(dread(&did,&i));                                                                                               
                                                                                                                                        
     /* Checks to see if the extension matches the parameter value */                                                                   
     /* If condition is true print the full name to the log        */                                                                   
      %if %qupcase(%qscan(&name,-1,.)) = %upcase(&ext) %then %do;                                                                       
        %put &dir/&name;                                                                                                                
      %end;                                                                                                                             
     /* If directory name call macro again */                                                                                           
      %else %if %qscan(&name,2,.) = %then %do;                                                                                          
        %drive(&dir/%unquote(&name),&ext)                                                                                               
      %end;                                                                                                                             
                                                                                                                                        
   %end;                                                                                                                                
                                                                                                                                        
  /* Closes the directory and clear the fileref */                                                                                      
  %let rc=%sysfunc(dclose(&did));                                                                                                       
  %let rc=%sysfunc(filename(filrf));                                                                                                    
                                                                                                                                        
%mend drive;                                                                                                                            
                                                                                                                                        
/* First parameter is the directory of where your files are stored. */                                                                  
/* Second parameter is the extension you are looking for.           */                                                                  
%drive(/sas/config/master.data/,sas)

Re: How can I scan for UNIX folders that contain 2 layers of dynamic folders with special filter?

ChrisNZ — Mon, 06 Dec 2021 00:12:13 GMT

  /* If directory name call macro again */

The way to test this is to use the dopen() function

Re: How can I scan for UNIX folders that contain 2 layers of dynamic folders with special filter?

Tom — Mon, 06 Dec 2021 14:02:15 GMT

You can use the MODIFY statement functionality of a DATA step to scan through a directory tree without have to actually write a recursive function.

Use this macro.

https://github.com/sasutils/macros/blob/master/dirtree.sas

If you only want to go down two levels use the MAXDEPTH option.

%dirtree
/*---------------------------------------------------------------------------
Build dataset of files in directory tree(s)
----------------------------------------------------------------------------*/
(/sas/config/master.data    /* Pipe delimited directory list (default=.) */
,out=dirtree  /* Output dataset name */
,maxdepth=2 /* Maximum tree depth */
);

To get list of directories that are two levels below your starting path you could use:

data level2;
  set dirtree;
  where level=2 and type='D';
run;

Re: How can I scan for UNIX folders that contain 2 layers of dynamic folders with special filter?

Ksharp — Mon, 06 Dec 2021 12:44:36 GMT

If you can use OS command ,it was very easy .

topic How can I scan for UNIX folders that contain 2 layers of dynamic folders with special filter? in SAS Programming

How can I scan for UNIX folders that contain 2 layers of dynamic folders with special filter?

Re: How can I scan for UNIX folders that contain 2 layers of dynamic folders with special filter?

Re: How can I scan for UNIX folders that contain 2 layers of dynamic folders with special filter?

Re: How can I scan for UNIX folders that contain 2 layers of dynamic folders with special filter?