I have a macro that I saw posted by @AllanBowe with an mp_dirlist macro, which I will list below. The macro works beautifully as intended to cleanup files older than 90 days (I modified the original post's 5 day limit). However, I have a shared folder where we store SAS logs that has almost 300K files in it and it takes forever just to run the macro to get the list of file names. Is there a way we can limit the macro to only retrieve the files older than x days before we go to the next statement? That way I could say, delete all files greater than 2 years, then greater than 1 year, and work my way down, versus trying to list everything all at once. Here is what I have, courtesy of Allan, using the mp_dirlist macro located below: /**
@file
@brief Returns all files and subdirectories within a specified parent
@details Not OS specific (uses dopen / dread). It does not appear to be
possible to reliably identify unix directories, and so a recursive
option is not available.
usage:
%mp_dirlist(path=/some/location,outds=myTable);
@param path= for which to return contents
@param outds= the output dataset to create
@returns outds contains the following variables:
- file_or_folder (file / folder)
- filepath (path/to/file.name)
- filename (just the file name)
- ext (.extension)
- msg (system message if any issues)
@version 9.2
@author Allan Bowe
**/
%macro mp_dirlist(path=%sysfunc(pathname(work))
, outds=work.mp_dirlist
)/*/STORE SOURCE*/;
data &outds (compress=no keep=file_or_folder filepath filename ext msg);
length filepath $500 fref $8 file_or_folder $6 filename $80 ext $20 msg $200;
rc = filename(fref, "&path");
if rc = 0 then do;
did = dopen(fref);
if did=0 then do;
putlog "NOTE: This directory is empty - &path";
msg=sysmsg();
put _all_;
stop;
end;
rc = filename(fref);
end;
else do;
msg=sysmsg();
put _all_;
stop;
end;
dnum = dnum(did);
do i = 1 to dnum;
filename = dread(did, i);
fid = mopen(did, filename);
if fid > 0 then do;
file_or_folder='file ';
ext = prxchange('s/.*\.{1,1}(.*)/$1/', 1, filename);
if filename = ext then ext = ' ';
end;
else do;
ext='';
file_or_folder='folder';
end;
filepath="&path/"!!filename;
output;
end;
rc = dclose(did);
stop;
run;
%mend; I want to find a way to limit the results above before we get to the part below, especially for directories that have been saving log files since we migrated to Linux in 2020. data _null_;
set work.mp_dirlist;
drop rc fid close;
format modified_dttm datetime19.;
rc=filename("fref",filepath);
fid=fopen("fref");
if fid>0 then do;
modified=finfo(fid,"Last Modified");
modified_dttm=input(modified,anydtdtm24.);
end;
close=fclose(fid); /* 60secs * 60 mins * 24 hours * 90 days */
if modified_dttm>0 and datetime()-modified_dttm > (60*60*24*90) /* and ext='???'*/ then do;
putlog 'deleting' filename;
rc=fdelete("fref");
end;
run; Any thoughts would be greatly appreciated!
... View more
