Looping through files in a directory

[jerry898969]

I'm tasked with writing a program count all the rows within the log files in a directory. There is a log file for each day of the year.

The pattern is 090101.log for the log files.

What is the best way to loop through each file and create one big file or a dataset per file to get the total number of rows across all files.

Thank you for your help

[sbb]

Explore the use of FILENAME with the "pipe" parameter and using a DATA step to parse the result for your own processing with a DATA step approach, using INFILE with appropriate INPUT logic.

Search the forum archives and also the SAS.COM support website for SAS-hosted DOC and also supplemental technical reference with sample code pieces.

Scott Barry
SBBWorks, Inc.

[Cynthia_sas]

Hi:
Because the filenames fall into a regular pattern, you could use SAS macro processing to read the files. At the end of a macro program to read 1 file, you could either capture the count of rows into a data set or create one big file by appending a single log to a dataset of all logs (but that seems overkill if what you want is just a count of rows).

You need to investigate SAS macro processing. There have been a lot of previous forum postings on how to read all the files in a directory, how to write a macro program to loop through all the files in a directory and many, many postings on macro processing in general.

In addition, there are many user group papers, beginner, intermediate and advanced level on the use of SAS macros with many good examples. Searching through the forum and using Google to search for user group papers should get you pointed in the right direction.

This is a good paper to get you started on Macro basics:
http://www2.sas.com/proceedings/sugi28/056-28.pdf

cynthia

[sbb]

I'm not so sure that macro processing is required, by assumption.

A DATA step approach using FILENAME with "pipe" and a series of "nested" SAS code paragraphs to loop through the file list and use INFILE with the FILEVAR= parameter substitution may be sufficient.

Scott Barry
SBBWorks, Inc.

[data_null__]

I agree that macro is not needed and in fact will just complicate things.

A wildcard in a filename statement and a few select INFILE statement options and it's all good.

[pre]
filename FT32F001 '*.log';
data logN;
length filename fname $128;
retain filename;
infile FT32F001 filename=fname eov=eov end=end;
input;
lines + 1;
if eov or end then do;
lines + end;
output;
end;
if _n_ eq 1 or eov then do;
filename=fname;
eov =0;
lines =0;
end;
run;
[/pre]

[Peter_C]

hi jerry898969

examples of reading multiple files in a data step can be found in the "SAS on-line help and Documentation" probably as examples for the filename and infile statements.
Your platform will probably support an infile statement with ? or * in the place where you would put a physical file name like
infile 'whatever path/*.log' eov= eov filename= filename ;
which you will find by testing or from documentation, creates a stream of all lines of all .log files from that path with information in variables :"filename" indicating from which file the current line has been read; and "eov" which indicates you have passed an end-of -file condition for one of the log files.

On the server that hosts this forum, in its "knowledge Base" can be found the very related article "Sample 24710: Read all the log files in a specified subdirectory and create a data set with their error messages".

For whom do we add value by posting more examples here?

I can "feel good" by posting an alternative to that offered by "data _null_;" which would be very little different - but my example is probably already in the archives of this Forum. (and so available for your research)

So I recommend, you jerry898969, should extend your learning of the languages of SAS, by reading these references and studying hard, through these archives and the KnowledgeBase. I think you'll find far more answers than we could provide.

peterC.

[jerry898969]

Thank you all for all your help.

This is the solution I came up with that worked. Thanks for all the suggestions

data log1 ;
length myinfile $ 2000 ;
infile 'c:\LogFiles\2010\ex*.log'
filename=myinfile
eov=first_this_file
end=done ;
input name $ ;
savefile=myinfile ;
run ;