Thanks for catching the FILENAME problem in untested code. End of post has tested example with generated data.
You probably want to check the value returned by filename=, it's probably $8, and should be lengthened
length source_filename in_filename $200 ;
Why is filename= automatic variable default length of $8? Probably because standard file references (i.e. filename <fileref> "path") are identifiers (a reference name) with an 8 character limit.
Yes, lots of files makes for lots of NOTES. You can get one NOTE if you use the operating system to your advantage, however you would lose the information about which file a record came from (if that loss is OK, hey no problem)
Use OS to execute a command that concatenates all the files together, and read from that commands stdout
filename ALLCSV pipe "powershell -c ""cat &PATH.\*.csv""";
data want;
infile ALLCSV dsd missover;
...
Back to business....
The first 'landmark' identifies the header line of the data. I chose the test to be "a line starting with <> is the header line" based on the sample data in the question.
<>Date Time Point_1 Point_2 Point_3 Point_4 Point_5 Point_6 Point_7 Point_8
The next line after the header line should be data.
If your test for the header line landmark is incorrect the program will never reach the second INPUT statement, and you will get 0 observations.
I chose// to be the landmark for end of data, again based on the sample data. Regardless, the input_now_flag is reset whenever a new filename is encountered.
12/31/2019 7:25:11 72.25 0.05 26.81 73.62 -0.09 -0.06 No Data No Data
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
Tested example (900 data rows written and 900 read in) :
data _null_;
do i = 1 to 9;
out_filename = cats("C:\temp\",i,".csv");
file multiple filevar=out_filename;
do j = 1 to 15;
put 'header detail' j;
end;
put '<>a,b,c';
do z = 1 to 100;
a = i * 100 + z-1;
b = a**2;
c = sqrt(a);
put a ',' b ',' c;
end;
put '/////////';
put '999,999,999, i dare you to ignore me';
end;
run;
proc format ;
invalue nodata
'TIME (-3)'=.t
'No Data'=.d
other=_same_
;
run;
filename ALLCSV pipe "powershell -c ""cat c:\temp\?.csv""";
data want;
length source_filename $200 in_filename $200 ;
infile 'c:\temp\?.csv' filename=in_filename dsd missover ;
/* infile ALLCSV dsd missover;*/
retain input_now_flag 0 source_filename;
if lag (in_filename) ne in_filename then do; /* start of reading from next file */
source_filename = in_filename;
input_now_flag = 0;
end;
input @;
if _infile_ =: '<>' then do; /* landmark for subsequent rows are data */
input_now_flag = 1;
delete;
end;
if _infile_ =: '//' then do; /* landmark for end of data, ignore anything that might be after it */
input_now_flag = 0;
delete;
end;
if not input_now_flag then delete; /* have not reached the data landmark yet */
/* presume all *.csv files have the same column structure and order */
input @1 a b c; *date :mmddyy. time :time. (point1-point8) (:nodata.);
drop input_now_flag;
run;
Click image to register for webinar
