Thanks @Kurt_Bremser @Tom @Reeza I have large datasets so I have to cut and paste bits and pieces instead of copying and pasting everything. The code below is from Kurt I believe and the dataset is empty, only has the rows belonging to the filename (nro rows = nro files). Hoever, all variables except filename and are missing. 12 * GET THE FILENAMES FOR ALL FILES IN THE FOLDER NAMED BELOW WITH AN EXTENSION .DAT;
13 Filename filelist pipe "dir /b /s C:\2020\*.dat";
14 Data filelist;
15 *USE THE LINE BELOW IF WANT ONLY THE LIST OF FILES IN THE LOG
16 Data _null_;
17 Infile filelist truncover;
18 Input filename $256.;
19 Put filename=;
20 Run;
NOTE: The infile FILELIST is:
Unnamed Pipe Access Device,
PROCESS=dir /b /s C:\2020\*.dat,
RECFM=V,LRECL=32767
filename=C:\2020\01\20\20200120-064931-DataLog_User.dat
filename=C:\2020\01\20\20200122-031029-DataLog_User.dat
....
....
filename=C:\2020\08\22\20201124-140120-DataLog_User.dat
filename=C:\2020\08\22\20201124-150129-DataLog_User.dat
NOTE: 4198 records were read from the infile FILELIST.
The minimum record length was 97.
The maximum record length was 97.
NOTE: The data set WORK.FILELIST has 4198 observations and 1 variables.
NOTE: DATA statement used (Total process time):
real time 9.85 seconds
cpu time 2.65 seconds
23 data import_all2;
24 set filelist;
25 length txt_file_name $256;
26 retain txt_file_name;
27 infile my_file eov=eov filevar=filename truncover;
28 input@;
29 if _n_ eq 1 or eov then do;
30 txt_file_name = scan(filename, -1, "\");
31 eov=0;
32 delete;
33 end;
34 else input X Y Z;
35 run;
NOTE: The variable filename exists on an input data set, but was also specified in an I/O statement option. The variable will not be included on any output data set.
NOTE: The infile MY_FILE is:
Filename=C:\2020\01\20\20200120-064931-Dat
aLog_User.dat,
RECFM=V,LRECL=32767,File Size (bytes)=1560240,
Last Modified=20 January 2020 18:15:08,
Create Time=26 July 2020 22:32:22
NOTE: The infile MY_FILE is:
Filename=C:\2020\01\22\JFAADS2172-20200122-031029-Dat
aLog_User.dat,
RECFM=V,LRECL=32767,File Size (bytes)=175230,
Last Modified=22 January 2020 14:13:20,
Create Time=26 July 2020 22:32:22
NOTE: Invalid data for X in line 1 1-4.
NOTE: Invalid data for Y in line 1 27-30.
NOTE: Invalid data for Z in line 1 53-72.
NOTE: Invalid data errors for file MY_FILE occurred outside the printed range.
NOTE: Increase available buffer lines with the INFILE n= option.
filename=C:\2020\01\22\20200122-071450-DataLog_U
ser.dat txt_file_name=JFAADS2172-20200120-064931-DataLog_User.dat eov=0 X=. Y=. Z=.
_ERROR_=1 _N_=7
NOTE: The infile MY_FILE is:
...
...
WARNING: Limit set by ERRORS= option reached. Further errors of this type will not be printed.
NOTE: Invalid data errors for file MY_FILE occurred outside the printed range.
NOTE: Increase available buffer lines with the INFILE n= option.
filename=C:\2020\01\22\20200122-211701-DataLog_User.dat txt_file_name=20200120-064931-DataLog_User.dat eov=0 X=. Y=.
Z=.
_ERROR_=1 _N_=21
NOTE: The infile MY_FILE is:
...
...
NOTE: The infile MY_FILE is:
Filename=C:2020\11\24\20201124-150129-Dat
aLog_User.dat,
RECFM=V,LRECL=32767,File Size (bytes)=2852470,
Last Modified=24 November 2020 15:36:56,
Create Time=24 November 2020 15:36:56
NOTE: 1 record was read from the infile MY_FILE.
The minimum record length was 988.
The maximum record length was 988.
NOTE: 1 record was read from the infile MY_FILE.
The minimum record length was 988.
The maximum record length was 988.
NOTE: 1 record was read from the infile MY_FILE.
The minimum record length was 988.
The maximum record length was 988.
NOTE: 1 record was read from the infile MY_FILE.
The minimum record length was 988.
The maximum record length was 988.
NOTE: 1 record was read from the infile MY_FILE.
The minimum record length was 988.
The maximum record length was 988.
...
...
NOTE: There were 4198 observations read from the data set WORK.FILELIST.
NOTE: The data set WORK.IMPORT_ALL2 has 4197 observations and 39 variables.
NOTE: DATA statement used (Total process time):
real time 6:36.55
cpu time 50.64 seconds
The code below is from Tom I believe and it results in the correct number of rows being read (nro rows = nro files * nro rows/file). However, all rows have missing data except for the variable filename. 3 Filename filelist pipe "dir /b /s C:\2020\05\*.dat";
4 Data filelist;
5 *USE THE LINE BELOW IF WANT ONLY THE LIST OF FILES IN THE LOG
6 Data _null_;
7 Infile filelist truncover;
8 Input filename $256.;
9 Put filename=;
10 Run;
NOTE: The infile FILELIST is:
Unnamed Pipe Access Device,
PROCESS=dir /b /s C:\2020\05\*.dat,
RECFM=V,LRECL=32767
filename=C:\2020\05\01\20200501-105705-DataLog_U
ser.dat
filename=C:\2020\05\01\20200501-115714-DataLog_U
ser.dat
filename=C:\2020\05\01\20200501-125724-DataLog_U
ser.dat
...
...
filename=C:\2020\05\31\20200601-095358-DataLog_U
ser.dat
NOTE: 761 records were read from the infile FILELIST.
The minimum record length was 97.
The maximum record length was 97.
NOTE: The data set WORK.FILELIST has 761 observations and 1 variables.
NOTE: DATA statement used (Total process time):
real time 1.84 seconds
cpu time 0.62 seconds
17 data import_all;
18 set filelist;
19 length txt_file_name $256;
20 filevar=filename ;
21 txt_file_name = scan(filename, -1, "\");
22 infile my_file dsd firstobs=2 filevar=filevar truncover end=eof;
23 do while(not eof);
24 input X Y Z;
25 output;
26 end;
27 run;
NOTE: The infile MY_FILE is:
Filename=C:\2020\05\30\JFAADS2172-20200530-104633-Dat
aLog_User.dat,
RECFM=V,LRECL=32767,File Size (bytes)=4882680,
Last Modified=30 May 2020 11:46:44,
Create Time=26 July 2020 22:32:17
NOTE: Invalid data for X in line 2 1-978.
NOTE: Invalid data for X in line 3 1-978.
NOTE: Invalid data for X in line 4 1-978.
NOTE: Invalid data for X in line 5 1-978.
NOTE: Invalid data for X in line 6 1-978.
...
...
NOTE: The infile MY_FILE is:
Filename=C:\2020\05\31\JFAADS2172-20200601-075337-Dat
aLog_User.dat,
RECFM=V,LRECL=32767,File Size (bytes)=4868820,
Last Modified=01 June 2020 08:53:48,
Create Time=26 July 2020 22:32:31
NOTE: The infile MY_FILE is:
Filename=C:\2020\05\31\JFAADS2172-20200601-085348-Dat
aLog_User.dat,
RECFM=V,LRECL=32767,File Size (bytes)=4896540,
Last Modified=01 June 2020 09:53:58,
Create Time=26 July 2020 22:32:31
NOTE: The infile MY_FILE is:
Filename=C:\2020\05\31\JFAADS2172-20200601-095358-Dat
aLog_User.dat,
RECFM=V,LRECL=32767,File Size (bytes)=4890600,
Last Modified=01 June 2020 10:54:08,
Create Time=26 July 2020 22:32:31
NOTE: 4939 records were read from the infile MY_FILE.
The minimum record length was 988.
The maximum record length was 988.
...
...
NOTE: 4917 records were read from the infile MY_FILE.
The minimum record length was 988.
The maximum record length was 988.
NOTE: 4945 records were read from the infile MY_FILE.
The minimum record length was 988.
The maximum record length was 988.
NOTE: 4939 records were read from the infile MY_FILE.
The minimum record length was 988.
The maximum record length was 988.
NOTE: There were 761 observations read from the data set WORK.FILELIST.
NOTE: The data set WORK.IMPORT_ALL has 3661162 observations and 40 variables.
NOTE: DATA statement used (Total process time):
real time 14:08.26
cpu time 36.32 seconds
... View more