I don't have DIS available right now but I believe you could implement below using DIS with External File Metadata and File Readers. The suggested approach will read the source data only once.
/* create sample source file */
data _null_;
file '~/source.csv';
infile datalines truncover;
input;
put _infile_;
datalines;
1,2,3,4
5,6,7,8
trailer1, trailer2
;
/*
read sample source file
- write last source record to file trailer_&sysjobid..csv
*/
data information;
infile '~/source.csv' dsd truncover end=eof;
file "%sysfunc(pathname(work))/trailer_&sysjobid..csv";
input @;
if eof then
do;
call symputx('trailerFile',
"%sysfunc(pathname(work))/trailer_&sysjobid..csv",
'g'
);
put _infile_;
delete;
end;
input a b c d;
run;
/*
read trailer file created in previous step
*/
data trailer;
infile "&trailerFile" dsd truncover;
input (var1 var2) (:$50.);
run;
proc print data=information;
run;
proc print data=trailer;
run;
External File Metadata allows you to define your own infile statements and I believe nothing prohibits to also add additional statements there. So what you could do is add below (eventually DIS will add the INFILE on its own; check the generated code once connected to a FileReader; also check in the generated code if DIS generates a macro variable for the path/filename so you could use this one instead of something hard-coded). The input statement with the variable mapping should then still get generated based on the metadata definition and mapping.
infile '~/source.csv' dsd truncover end=eof;
file "%sysfunc(pathname(work))/trailer_&sysjobid..csv";
input @;
if eof then
do;
call symputx('trailerFile',
"%sysfunc(pathname(work))/trailer_&sysjobid..csv",
'g'
);
put _infile_;
delete;
end;
Above code will not read the last record into the target SAS table but write it to a file in SAS Work. The full path and name gets stored in macro variable &trailerFile.
So now you can implement a 2nd External File definition where you use the macro variable &trailerFile as name (in double quotes). Because the layout of the trailer is the same for all your source data you only need to define this once and then you can use it in all your jobs.
N.B: If you can't be 100% sure that the very last line in your source is always the trailer (= never an additional empty line after the trailer line) then instead of using end=eof implement some check like: if substrn(_infile_,1,5) ="some string" then...
... View more
