Hi all:
Quick question for a Friday afternoon to warm up your thinking devices...
I have a directory with over 9,000 members in it - they are all small (about 25 lines of text) - and the member name contains the date and time stamp for that data.
Currently, I do a "X command" directory listing, and then read that list back in and process the data with a macro that loops through all the members. The date and time is added to each of the members read and then the whole thing is rolled up into a single SAS table.
I'm wondering if I change this to a "wild card" read using th INFILE statement, is there a way to still retain the date and time values from the member name? I seem to remember that the name of the member may be in the meta data somewhere, but I can't find any references.
Also - I'm thinking that reading the entire directory using the wild card approach will be much faster, but does anyone have proof? (I expect that I will have said proof if there is a way for me to use the wild card input....)
Thanks in advance.
Here is an example of datanull's suggestion. Try it, compare it with your method, and let us know which works faster:
%let path=c:\art\test\;
/*create some test data;*/
%macro makem;
%do i=1 %to 1000;
data _null_;
file "&path.f&i..txt";
do j=1 to 1000;
put j j;
end;
run;
%end;
%mend makem;
%makem
data want;
length inname $80;
infile "&path.f*.txt" filename=inname;
input x y;
filenm=substr(inname,13);
run;
Are you reading the data in all files into a single SAS data set?
For reading from wild card you will need to understand FILENAME= INFILE statement option. Also you should learn about INFILE statement output EOV=.
Is should be faster than macro looping as you will need only ONE data step to read all files.
Here is an example of datanull's suggestion. Try it, compare it with your method, and let us know which works faster:
%let path=c:\art\test\;
/*create some test data;*/
%macro makem;
%do i=1 %to 1000;
data _null_;
file "&path.f&i..txt";
do j=1 to 1000;
put j j;
end;
run;
%end;
%mend makem;
%makem
data want;
length inname $80;
infile "&path.f*.txt" filename=inname;
input x y;
filenm=substr(inname,13);
run;
Art:
Just what I was looking for.
I now have > 16,000 members in the directory that I have to read. It takes about 20 minutes to read using a macro loop to read each memner, and about 3 minutes using the wildcard method.
Wild card method is obviously faster (and easier to code!).
Thanks again.
But wildcard is not as flexiable as the PIPE + filevar=
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
Learn how use the CAT functions in SAS to join values from multiple variables into a single value.
Find more tutorials on the SAS Users YouTube channel.