Read 1 line From Multiple Files

Reply
Super Contributor
Posts: 358

Read 1 line From Multiple Files

Hi All:

I was hoping somebody would have a simple solution to this.

I have a directory that has a large number of files in it.  The first record of each file has a date on it, with the data following.

I need to read all the records from the directory for a specific month (and year). 

This would mean that I have to read all files in the directory, and keep the name of the file.  Then I would read the first line

to determine the date - if it is not the month that I was, I reject the file and read the next one.

If the file is within my date rage, I would keep the file name.  Once I have examined all files, I would then re-process just

the file that I retained.

This seems like a "brute force" approach and I was hoping for something a little more elegant - any ideas?

Thanks in advance.

Super User
Super User
Posts: 6,502

Re: Read 1 line From Multiple Files

Brute force is the way to go, but doing via a program.

Get the list of file names.  I usually find this is easier with PIPEd operating system commands.  DIR /B in Windoze and ls in Unix.

You will need to supply input statement for reading the date out of the first line as the syntax will depend on how it is formatted.

%let path=mydir;

%let month=01JAN2014 ;

data files ;

  infile "ls &mydir/*" pipe truncover ;

  input filename $255. ;

  filen = filename ;

  infile dummy filevar=filen truncover end=eof ;

  if not eof then input date date9. @ ;

  if  date = "&month"d then output;

run;

Super Contributor
Posts: 358

Re: Read 1 line From Multiple Files

Tom:

Thanks for the help - it looks like brute force is the only way to go.

Another question - is there a way for me to read only a single record from the file in a data step without reading the rest of the data?  Some of these files are somewhat large and I only need the header.

Currently I use something like:

data one_file1;

infile myfile lrecl=100 pad missover;

input @001 text $char51.;

if _n_ = 1 then output;

           else delete;

I would rather not have to read the entire file if that is possible (option obs=1 possibly?)

Super User
Super User
Posts: 6,502

Re: Read 1 line From Multiple Files

The code I posted before would only read one record per file because the main loop was the list of files.

If you are reading a single file then just add the option OBS=1 to the INFILE statement.

data one_file1;

  infile myfile lrecl=100 obs=1 truncover ;

  input text $char51.;

run;

Super User
Super User
Posts: 7,421

Re: Read 1 line From Multiple Files

Hi,

Assuming that your files have a particular date structure, then you can get a list of files you want to process quite simply with a DOS command.

findstr "..MAY2013" *.txt

So:

filename rob pipe 'findstr /m "..MAY2013" "s:\temp\rob\*.txt"';

data dir;
  attrib buffer format=$2000.;
  infile rob lrecl=30000 dsd;
  input buffer;
run;

Will give me a dataset called dir which contains the path and filename of any files with .txt extension in s:\temp\rob which have a pattern --MAY2013.  I can then read those files in.  This means you wouldn't have to read each file, but does mean the data needs to be structured.

Super Contributor
Posts: 358

Re: Read 1 line From Multiple Files

Hi:

Unfortunately, the date is in the header of the data, not in the file name.  I also have to compare this to the date the file was created to determine if the file was rerun more than once, and keep the latest one only.

Thanks.

Super Contributor
Posts: 358

Re: Read 1 line From Multiple Files

Tom:

The "obs=1" did the trick. 

Thanks

Ask a Question
Discussion stats
  • 6 replies
  • 355 views
  • 3 likes
  • 3 in conversation