topic Reading All Members in a Directory in SAS Programming

Reading All Members in a Directory

OS2Rules — Fri, 16 Mar 2012 17:55:31 GMT

Hi all:

Quick question for a Friday afternoon to warm up your thinking devices...

I have a directory with over 9,000 members in it - they are all small (about 25 lines of text) - and the member name contains the date and time stamp for that data.

Currently, I do a "X command" directory listing, and then read that list back in and process the data with a macro that loops through all the members. The date and time is added to each of the members read and then the whole thing is rolled up into a single SAS table.

I'm wondering if I change this to a "wild card" read using th INFILE statement, is there a way to still retain the date and time values from the member name? I seem to remember that the name of the member may be in the meta data somewhere, but I can't find any references.

Also - I'm thinking that reading the entire directory using the wild card approach will be much faster, but does anyone have proof? (I expect that I will have said proof if there is a way for me to use the wild card input....)

Thanks in advance.

Re: Reading All Members in a Directory

data_null__ — Fri, 16 Mar 2012 18:39:09 GMT

Are you reading the data in all files into a single SAS data set?

For reading from wild card you will need to understand FILENAME= INFILE statement option. Also you should learn about INFILE statement output EOV=.

Is should be faster than macro looping as you will need only ONE data step to read all files.

Re: Reading All Members in a Directory

art297 — Fri, 16 Mar 2012 19:23:54 GMT

Here is an example of datanull's suggestion. Try it, compare it with your method, and let us know which works faster:

%let path=c:\art\test\;

/*create some test data;*/

%macro makem;

%do i=1 %to 1000;

data _null_;

file "&path.f&i..txt";

do j=1 to 1000;

put j j;

end;

run;

%end;

%mend makem;

%makem

data want;

length inname $80;

infile "&path.f*.txt" filename=inname;

input x y;

filenm=substr(inname,13);

run;

Reading All Members in a Directory

OS2Rules — Mon, 19 Mar 2012 19:18:17 GMT

Art:

Just what I was looking for.

I now have > 16,000 members in the directory that I have to read. It takes about 20 minutes to read using a macro loop to read each memner, and about 3 minutes using the wildcard method.

Wild card method is obviously faster (and easier to code!).

Thanks again.

Reading All Members in a Directory

Ksharp — Tue, 20 Mar 2012 02:56:01 GMT

But wildcard is not as flexiable as the PIPE + filevar=