Recursive find files

RichardDeVen — Sun, 05 Feb 2023 17:30:34 GMT

Here is a new twist on an old trope. Recursive find files will create a data set listing all files below a root path to a certain depth. Recursion in DATA Step made possible by maintaining state data (i.e. stack) in a hash object. The only downside is that sub-folder detection in SAS is done via MOPEN returning a 0, which means every file detected has an attempted open and it takes a lot longer to get the results than say a bash find command.

Sample code:

%let root = G:;
%let maxdepth = 2;

data find_files (keep=folder filename);
  length folder $2000 fref $8 filename $2000 index dnum 8 ;

  call missing (depth, did, folder, index, dnum);

  declare hash state ();
  state.defineKey ('depth');
  state.defineData ('fref', 'did', 'folder', 'index', 'dnum');
  state.defineDone();

  folder = "&root";

  depth = 0;
  maxdepth = &maxdepth;

  link push;
  goto iterate;

push:
  depth = depth + 1;

  link fref;
  did = dopen (fref);

  if did = 0 then stop;

  filename = '';
  output;

  dnum = dnum (did);
  index = 0;

  state.add();

  if depth > maxdepth then do;
    link pop;
    goto iterate;
  end;
return;

%*-------------------------------------------------------;
iterate:
%*-------------------------------------------------------;
  if index >= dnum then do;
    link pop;
    goto iterate;
  end;

  index + 1;
  state.replace();

  filename = dread(did, index);

  mid = mopen(did, filename);

  if mid = 0 then do;
    folder = catx ('/', folder, filename);

    link push;
    goto iterate;
  end;

* add FINFO() calls here to obtain file size and mod date;

  mid = fclose(mid);
  output;

  goto iterate;

%*-------------------------------------------------------;
pop:
%*-------------------------------------------------------;
  did = dclose(did);
  rc = filename (fref);

  state.remove();

  depth = depth - 1;
  if depth = 0 then stop;

  state.find();  

  return;

%*-------------------------------------------------------;
fref:
%*-------------------------------------------------------;
  call missing (fref);
  rc = filename (fref, folder);

  if rc ne 0 then do;
    msg = sysmsg();
    put msg;
    stop;
  end;
  return;

run;

Re: Recursive find files

Tom — Sun, 05 Feb 2023 17:41:42 GMT

If your subtree is too large to load into a hash object you can use the MODIFY statement to affect a recursive search.

https://github.com/sasutils/macros/blob/master/dirtree.sas

Re: Recursive find files

RichardDeVen — Sun, 05 Feb 2023 19:04:58 GMT

The only issue I see with the recursion is if the number of allowed open
filerefs is exceeded. There is 1 fileref open per depth reached. No way
the hash will run out of resources.

topic Re: Recursive find files in SAS Programming

Recursive find files

Re: Recursive find files

Re: Recursive find files