topic Read 1 line From Multiple Files in SAS Programming

Read 1 line From Multiple Files

OS2Rules — Mon, 10 Mar 2014 14:05:04 GMT

Hi All:

I was hoping somebody would have a simple solution to this.

I have a directory that has a large number of files in it. The first record of each file has a date on it, with the data following.

I need to read all the records from the directory for a specific month (and year).

This would mean that I have to read all files in the directory, and keep the name of the file. Then I would read the first line

to determine the date - if it is not the month that I was, I reject the file and read the next one.

If the file is within my date rage, I would keep the file name. Once I have examined all files, I would then re-process just

the file that I retained.

This seems like a "brute force" approach and I was hoping for something a little more elegant - any ideas?

Thanks in advance.

Re: Read 1 line From Multiple Files

Tom — Mon, 10 Mar 2014 14:24:14 GMT

Brute force is the way to go, but doing via a program.

Get the list of file names. I usually find this is easier with PIPEd operating system commands. DIR /B in Windoze and ls in Unix.

You will need to supply input statement for reading the date out of the first line as the syntax will depend on how it is formatted.

%let path=mydir;

%let month=01JAN2014 ;

data files ;

infile "ls &mydir/*" pipe truncover ;

input filename $255. ;

filen = filename ;

infile dummy filevar=filen truncover end=eof ;

if not eof then input date date9. @ ;

if date = "&month"d then output;

run;

Re: Read 1 line From Multiple Files

OS2Rules — Wed, 12 Mar 2014 13:19:18 GMT

Tom:

Thanks for the help - it looks like brute force is the only way to go.

Another question - is there a way for me to read only a single record from the file in a data step without reading the rest of the data? Some of these files are somewhat large and I only need the header.

Currently I use something like:

data one_file1;

infile myfile lrecl=100 pad missover;

input @001 text $char51.;

if _n_ = 1 then output;

else delete;

I would rather not have to read the entire file if that is possible (option obs=1 possibly?)

Re: Read 1 line From Multiple Files

Tom — Wed, 12 Mar 2014 13:25:07 GMT

The code I posted before would only read one record per file because the main loop was the list of files.

If you are reading a single file then just add the option OBS=1 to the INFILE statement.

data one_file1;

infile myfile lrecl=100 obs=1 truncover ;

input text $char51.;

run;

Re: Read 1 line From Multiple Files

RW9 — Wed, 12 Mar 2014 14:19:31 GMT

Hi,

Assuming that your files have a particular date structure, then you can get a list of files you want to process quite simply with a DOS command.

findstr "..MAY2013" *.txt

So:

filename rob pipe 'findstr /m "..MAY2013" "s:\temp\rob\*.txt"';

data dir;
attrib buffer format=$2000.;
infile rob lrecl=30000 dsd;
input buffer;
run;

Will give me a dataset called dir which contains the path and filename of any files with .txt extension in s:\temp\rob which have a pattern --MAY2013. I can then read those files in. This means you wouldn't have to read each file, but does mean the data needs to be structured.

Re: Read 1 line From Multiple Files

OS2Rules — Wed, 12 Mar 2014 14:23:32 GMT

Hi:

Unfortunately, the date is in the header of the data, not in the file name. I also have to compare this to the date the file was created to determine if the file was rerun more than once, and keep the latest one only.

Thanks.

Re: Read 1 line From Multiple Files

OS2Rules — Wed, 12 Mar 2014 14:24:00 GMT

Tom:

The "obs=1" did the trick.

Thanks