DATA Step, Macro, Functions and more

Reading All Members in a Directory

Accepted Solution Solved
Reply
Super Contributor
Posts: 358
Accepted Solution

Reading All Members in a Directory

Hi all:

Quick question for a Friday afternoon to warm up your thinking devices...

I have a directory with over 9,000 members in it - they are all small (about 25 lines of text) - and the member name contains the date and time stamp for that data.

Currently, I do a "X command" directory listing, and then read that list back in and process the data with a macro that loops through all the members.  The date and time is added to each of the members read and then the whole thing is rolled up into a single SAS table.

I'm wondering if I change this to a "wild card" read using th INFILE statement, is there a way to still retain the date and time values from the member name?  I seem to remember that the name of the member may be in the meta data somewhere, but I can't find any references.

Also - I'm thinking that reading the entire directory using the wild card approach will be much faster, but does anyone have proof?  (I expect that I will have said proof if there is a way for me to use the wild card input....)

Thanks in advance.


Accepted Solutions
Solution
‎03-16-2012 03:23 PM
PROC Star
Posts: 7,356

Re: Reading All Members in a Directory

Here is an example of datanull's suggestion.  Try it, compare it with your method, and let us know which works faster:

%let path=c:\art\test\;

/*create some test data;*/

%macro makem;

  %do i=1 %to 1000;

    data _null_;

      file "&path.f&i..txt";

      do j=1 to 1000;

        put j j;

      end;

    run;

  %end;

%mend makem;

%makem

data want;

  length inname $80;

  infile "&path.f*.txt" filename=inname;

  input x y;

  filenm=substr(inname,13);

run;

View solution in original post


All Replies
Respected Advisor
Posts: 3,777

Re: Reading All Members in a Directory

Are you reading the data in all files into a single SAS data set? 

For reading from wild card you will need to understand FILENAME= INFILE statement option.  Also you should learn about INFILE statement output EOV=.

Is should be faster than macro looping as you will need only ONE data step to read all files.

Solution
‎03-16-2012 03:23 PM
PROC Star
Posts: 7,356

Re: Reading All Members in a Directory

Here is an example of datanull's suggestion.  Try it, compare it with your method, and let us know which works faster:

%let path=c:\art\test\;

/*create some test data;*/

%macro makem;

  %do i=1 %to 1000;

    data _null_;

      file "&path.f&i..txt";

      do j=1 to 1000;

        put j j;

      end;

    run;

  %end;

%mend makem;

%makem

data want;

  length inname $80;

  infile "&path.f*.txt" filename=inname;

  input x y;

  filenm=substr(inname,13);

run;

Super Contributor
Posts: 358

Reading All Members in a Directory

Art:

Just what I was looking for. 

I now have > 16,000 members in the directory that I have to read.  It takes about 20 minutes to read using a macro loop to read each memner, and about 3 minutes using the wildcard method.  

Wild card method is obviously faster (and easier to code!).

Thanks again.

Super User
Posts: 9,662

Reading All Members in a Directory

But wildcard is not as flexiable as the PIPE + filevar=

☑ This topic is SOLVED.

Need further help from the community? Please ask a new question.

Discussion stats
  • 4 replies
  • 259 views
  • 0 likes
  • 4 in conversation