Hi All:
I was hoping somebody would have a simple solution to this.
I have a directory that has a large number of files in it. The first record of each file has a date on it, with the data following.
I need to read all the records from the directory for a specific month (and year).
This would mean that I have to read all files in the directory, and keep the name of the file. Then I would read the first line
to determine the date - if it is not the month that I was, I reject the file and read the next one.
If the file is within my date rage, I would keep the file name. Once I have examined all files, I would then re-process just
the file that I retained.
This seems like a "brute force" approach and I was hoping for something a little more elegant - any ideas?
Thanks in advance.
Brute force is the way to go, but doing via a program.
Get the list of file names. I usually find this is easier with PIPEd operating system commands. DIR /B in Windoze and ls in Unix.
You will need to supply input statement for reading the date out of the first line as the syntax will depend on how it is formatted.
%let path=mydir;
%let month=01JAN2014 ;
data files ;
infile "ls &mydir/*" pipe truncover ;
input filename $255. ;
filen = filename ;
infile dummy filevar=filen truncover end=eof ;
if not eof then input date date9. @ ;
if date = "&month"d then output;
run;
Tom:
Thanks for the help - it looks like brute force is the only way to go.
Another question - is there a way for me to read only a single record from the file in a data step without reading the rest of the data? Some of these files are somewhat large and I only need the header.
Currently I use something like:
data one_file1;
infile myfile lrecl=100 pad missover;
input @001 text $char51.;
if _n_ = 1 then output;
else delete;
I would rather not have to read the entire file if that is possible (option obs=1 possibly?)
The code I posted before would only read one record per file because the main loop was the list of files.
If you are reading a single file then just add the option OBS=1 to the INFILE statement.
data one_file1;
infile myfile lrecl=100 obs=1 truncover ;
input text $char51.;
run;
Hi,
Assuming that your files have a particular date structure, then you can get a list of files you want to process quite simply with a DOS command.
findstr "..MAY2013" *.txt
So:
filename rob pipe 'findstr /m "..MAY2013" "s:\temp\rob\*.txt"';
data dir;
attrib buffer format=$2000.;
infile rob lrecl=30000 dsd;
input buffer;
run;
Will give me a dataset called dir which contains the path and filename of any files with .txt extension in s:\temp\rob which have a pattern --MAY2013. I can then read those files in. This means you wouldn't have to read each file, but does mean the data needs to be structured.
Hi:
Unfortunately, the date is in the header of the data, not in the file name. I also have to compare this to the date the file was created to determine if the file was rerun more than once, and keep the latest one only.
Thanks.
Tom:
The "obs=1" did the trick.
Thanks
Don't miss out on SAS Innovate - Register now for the FREE Livestream!
Can't make it to Vegas? No problem! Watch our general sessions LIVE or on-demand starting April 17th. Hear from SAS execs, best-selling author Adam Grant, Hot Ones host Sean Evans, top tech journalist Kara Swisher, AI expert Cassie Kozyrkov, and the mind-blowing dance crew iLuminate! Plus, get access to over 20 breakout sessions.
Learn how use the CAT functions in SAS to join values from multiple variables into a single value.
Find more tutorials on the SAS Users YouTube channel.