topic Re: PIPE Command Start of NEW File in SAS Programming

PIPE Command Start of NEW File

shl007 — Thu, 14 May 2020 21:32:16 GMT

Hi - I've used the PIPE command to import multiple files at once, which comes in quite handy. Is there any way, using PIPE, to tell whether a NEW file is encountered? Because what I have to do is take some action once a NEW file is imported. Example: using PIPE, I import 2 files. At the start and end of each file, I have to insert a value. My uncertainty is, how to detect when I reach a new file because the inserts have to be done each time a new file is encountered in the PIPE. Thanks for any tips.

Re: PIPE Command Start of NEW File

Reeza — Thu, 14 May 2020 21:46:32 GMT

FILENAME has EOF and EOV options to help with this now, if you can go that route instead of the PIPE option. Depends on how you define a 'new' file which I don't quite understand from your question.

Re: PIPE Command Start of NEW File

ballardw — Thu, 14 May 2020 22:45:28 GMT

It might help to show us how you are actually using the files, i.e. example code.

Re: PIPE Command Start of NEW File

ChrisNZ — Thu, 14 May 2020 22:46:35 GMT

> I've used the PIPE command to import multiple files at once

Pipe runs a system command, it does not import files.

Seeing your code would help understand what you are doing. Including the filename statement.

Re: PIPE Command Start of NEW File

mkeintz — Fri, 15 May 2020 02:45:42 GMT

Depending on the command you are PIPEing, it's not likely, in the case of multiple files, that file end/file start information is available to SAS. If so, you either have to modify the pipe to insert signals between the files, or else pipe the files one-at-a-time such that your program knows the source for each data record.

Re: PIPE Command Start of NEW File

Kurt_Bremser — Fri, 15 May 2020 05:12:52 GMT

The PIPE receives an amorphous stream of bytes from the external command, it can only detect when the stream ends, but not when there is a change of files on the input side of the external command. If the files that are read have header lines, you can use that to detect a file change; if not, you need to structure your external command in a way that inserts markers.

Alternatively, you could use a dynamic pipe, with a dataset of filenames and the filevar= option.

Re: PIPE Command Start of NEW File

shl007 — Fri, 15 May 2020 15:03:07 GMT

I am invoking PIPE this way with a macro variable embedded to look for certain key word(s). I knew about the "END" option but wasn't aware there were also EOF and EOV options, but from Kurt's response above, it sounds like those won't do me any good, which explains why my testing has not been successful to date.

Unfortunately, the files don't have any header line ... Kurt or anyone, can you elaborate on what you mean by dynamic pipe? I think I may already be using it because I am using filevar option to store the filename. So I can use the filevar option to see if the filename changes? Would I need a LAG function or something like it to compare previous to current value? I had tried LAG on the filevar variable, but SAS didn't like it.

filename output pipe "ls -a /myfolder/subfolder/&other.*";

Re: PIPE Command Start of NEW File

Kurt_Bremser — Fri, 15 May 2020 15:06:58 GMT

@shl007 wrote:

filename output pipe "ls -a /myfolder/subfolder/&other.*";

This pipe creates only one stream of filenames, not the contents of the files.

Re: PIPE Command Start of NEW File

Kurt_Bremser — Fri, 15 May 2020 15:11:21 GMT

The pipe you showed is good to create a dataset with filenames, but if you want to read those files without any non-SAS processing in between, you can use the same wildcard, and the filename= option of the INFILE statement to get the name of the file that is currently read; a change in that variable signals a switch between input files.

Re: PIPE Command Start of NEW File

Kurt_Bremser — Tue, 26 May 2020 10:04:07 GMT

Here is a simplified example of the "dynamic pipe". It combines the use of PIPE, FILEVAR= and END= options of the infile statement. To simulate running an external command on a file, I used "cat".

First, create some files to read:

data _null_;
set sashelp.class;
file '$HOME/class1.csv' dlm=',';
put name sex age;
run;

data _null_;
set sashelp.class;
file '$HOME/class2.csv' dlm=',';
put name sex age;
run;

Create a list of filenames to process; this dataset might be created by the "ls" pipe you posted:

data filenames;
input fname $80.;
datalines;
$HOME/class1.csv
$HOME/class2.csv
;

Now we use this dataset to control the creation of a command for the PIPE:

data want;
set filenames;
length fvar $200 name $8 sex $1;
fvar = "cat " !! trim(fname);
infile dummy pipe filevar=fvar end=done dlm=',';
/* since we have a "change of file" for every data stepiteration, here is the place to insert a special value */
name = "start";
output;
/* now read the result of the pipe in a loop */
do while (not done);
  input name sex age;
  output;
end;
drop fname;
run;

Let's see if this worked:

proc print data=want noobs;
run;

Result:

name       sex    age

start               .
Alfred      M      14
Alice       F      13
Barbara     F      13
Carol       F      14
Henry       M      14
James       M      12
Jane        F      12
Janet       F      15
Jeffrey     M      13
John        M      12
Joyce       F      11
Judy        F      14
Louise      F      12
Mary        F      15
Philip      M      16
Robert      M      12
Ronald      M      15
Thomas      M      11
William     M      15
start               .
Alfred      M      14
Alice       F      13
Barbara     F      13
Carol       F      14
Henry       M      14
James       M      12
Jane        F      12
Janet       F      15
Jeffrey     M      13
John        M      12
Joyce       F      11
Judy        F      14
Louise      F      12
Mary        F      15
Philip      M      16
Robert      M      12
Ronald      M      15
Thomas      M      11
William     M      15

Voila! We've successfully inserted a value at the start of processing each separate file.