topic Re: Saving the files found using the list_files macro into a global macro variable in SAS Programming

Saving the files found using the list_files macro into a global macro variable

serrld113 — Fri, 04 Sep 2020 16:07:16 GMT

I need to find all the SAS files in all the sub-directories in a drive and save them into a macro variable that I can use outside the macro. I found this example below that prints the names of all the files of said directory into the log. I know that it saves them locally into &name, but I need to use name outside this macro. I tried changing %local to %global and the macro ran forever. I also tried saving name into another variable within the macro but also failed. I'd much rather save these file names into a table, but a macro variable will also work. Any help would be appreciated.

I'm using the following macro that I found here:

https://documentation.sas.com/?docsetId=mcrolref&docsetTarget=n0js70lrkxo6uvn1fl4a5aafnlgt.htm&docsetVersion=9.4&locale=en

This is it below:

%macro list_files(dir,ext);
%local filrf rc did memcnt name i;
%let rc=%sysfunc(filename(filrf,&dir));
%let did=%sysfunc(dopen(&filrf));

%if &did eq 0 %then %do; 
%put Directory &dir cannot be open or does not exist;
%return;
%end;

%do i = 1 %to %sysfunc(dnum(&did));

%let name=%qsysfunc(dread(&did,&i));

%if %qupcase(%qscan(&name,-1,.)) = %upcase(&ext) %then %do;
%put &dir\&name;
%end;
%else %if %qscan(&name,2,.) = %then %do; 
%list_files(&dir\&name,&ext)
%end;

%end;
%let rc=%sysfunc(dclose(&did));
%let rc=%sysfunc(filename(filrf));

%mend list_files;
%list_files(c:\temp,sas);

Re: Saving the files found using the list_files macro into a global macro variable

PaigeMiller — Fri, 04 Sep 2020 16:10:59 GMT

I have a somewhat different macro that saves the results in a SAS data set. From there, you can do whatever you want with the data set.

%macro list_files(dir=,outdsn=files,all_subdirs=N);
data _null_;
	if upcase("&all_subdirs")='N' then call symputx('str',cat("'dir ",'"',"%trim(&dir)"" /b'"));
	else if upcase("&all_subdirs")='Y' then call symputx('str',cat("'dir ",'"',"%trim(&dir)"" /b /s'"));
run;
filename tmp pipe &str;
data &outdsn;
  infile tmp dlm="¬" missover;
  input filenames :$200.;
run;
%mend;

(Works only on Windows)

Re: Saving the files found using the list_files macro into a global macro variable

ballardw — Fri, 04 Sep 2020 16:46:13 GMT

I don't know about your system but I have some project that if the paths are kept in the values with the filename then the length of single macro variable would be exceeded.

Plus are your paths and file names really clean, like no spaces or odd characters in the folder or file names?

If not, then you might have some fun using such an ugly variable.

Anything that can be done with macro variable and much more can be done with values in a data set. I would, for any purpose I can envision using such, go with @PaigeMiller's suggestion.

Re: Saving the files found using the list_files macro into a global macro variable

serrld113 — Fri, 04 Sep 2020 16:48:03 GMT

I added that I prefer a data set over a macro variable.

Re: Saving the files found using the list_files macro into a global macro variable

serrld113 — Fri, 04 Sep 2020 16:55:59 GMT

I really appreciate this, but it didn't output a dataset for me. I added my directory path and changed all_subdirs = "Y"

Re: Saving the files found using the list_files macro into a global macro variable

PaigeMiller — Fri, 04 Sep 2020 17:26:45 GMT

@serrld113 wrote:
I really appreciate this, but it didn't output a dataset for me. I added my directory path and changed all_subdirs = "Y"

Works for me.

Note: for your own benefit, as well as the benefit of others, it is never sufficient to say something didn't work and provide no additional information. If it didn't work, give us details about what you did and what happened. Otherwise, the conversation goes "didn't work" "works for me" "doesn't work for me" "works fine for me"

Re: Saving the files found using the list_files macro into a global macro variable

serrld113 — Fri, 04 Sep 2020 17:32:51 GMT

I apologize. In my reply I included that I've added my directory as well as changed the option for all_subdirs to Y. I've also tried my directory between quotation marks and without. Thanks again.

Re: Saving the files found using the list_files macro into a global macro variable

serrld113 — Fri, 04 Sep 2020 17:34:46 GMT

The one thing that kinda works is adding a data step after the %put statement in my original post. However, it will only output the last SAS file the loop finds. I was wondering if there was a dynamic way to keep adding to the variable as the loop runs.

      %if %qupcase(%qscan(&name,-1,.)) = %upcase(&ext) %then %do;
			%put &dir\&name;

/* this is the part I add */
				data files;
			  		filenames = "&dir\&name";
				run;

		%end;

Re: Saving the files found using the list_files macro into a global macro variable

PaigeMiller — Fri, 04 Sep 2020 17:38:46 GMT

We need to see your code, or the log, or both. That's the only way to figure out what happens when code doesn't work.

Re: Saving the files found using the list_files macro into a global macro variable

Tom — Sat, 05 Sep 2020 03:59:57 GMT

Don't put the file names into macro variables. Put them into a dataset. A macro variable has a maxium length of 64K. Even a moderate number of names could easily overflow that limit.

There is a great idea for doing the recursion into the subdirectories by using the MODIFY statement in a data step posted in this thread: https://communities.sas.com/t5/SAS-Programming/listing-all-files-within-a-directory-and-subdirectories/m-p/332616/highlight/true#M74887

Here is an example. First create a dataset with the structure you want and one observation for each top level directory you want to read.

data filelist;
  length root filename $256 dir level 8;
  input root;
  retain filename ' ' level 0 dir 1;
cards4;
c:\temp
;;;;

Now run this data step to scan all of the files in the directory(s).

data filelist;
  modify filelist;
  rc1=filename('tmp',catx('\',root,filename));
  rc2=dopen('tmp');
  dir = not not rc2;
  replace;
  if dir;
  level=level+1;
  filename0=filename;
  do i=1 to dnum(rc2);
    filename=catx('\',filename0,dread(rc2,i));
    output;
  end;
  rc3=dclose(rc2);
run;

If you want to limit to files with a particular extension it is probably easier after you have the dataset already.

proc print data=filelist;
  where filename like '%.sas7bdat';
run;

Results

Obs     root      filename               dir    level

 4     c:\temp    one.sas7bdat            0       1
 6     c:\temp    two.sas7bdat            0       1
 7     c:\temp    sub1\three.sas7bdat     0       2