topic Re: PIPE imports only first line in each file in SAS Programming

PIPE imports only first line in each file

shl007 — Mon, 15 Jun 2020 18:37:14 GMT

Hi - I am using a PIPE to import multiple text files. The PIPE does recognize all files in the directory but is only importing the first line in each file? For example, if files A & B are recognized by the pipe, only the first line in A and the first line in B are imported. The file is delimited by '^'. Any ideas why only the first line of each file is imported? Below is the code I am using, including a DO-WHILE loop. I am using this as part of a macro.

filename output pipe %sysfunc(quote(ls -a "&pipepath"*_*.txt));

DATA WORK.testupd;
    INFILE output TRUNCOVER ;
    INPUT name $250.;/*Pull pipe in*/
    fil2read=name;
	INFILE dummy FILEVAR=fil2read END=last FIRSTOBS=1  LENGTH=len truncover
		 /*LRECL=292*/
        ENCODING="LATIN1"
        TERMSTR=CRLF
        DLM='^' 
        MISSOVER
        DSD ;

    LENGTH
        F1               $ 28	/* field1*/
        F2               $ 1	/* field2 */
        F3               $ 8  /* field3 */
        F4               $ 10  /* field4 */
        F5               $ 60	/* field5 */
        F6               $ 50	/* field6 */
;
    FORMAT
        F1               $CHAR28.
        F2               $CHAR1.
        F3               $CHAR8.
        F4               $CHAR10.
        F5               $CHAR60.
        F6               $CHAR50.
;
    INFORMAT
        F1               $CHAR28.
        F2               $CHAR1.
        F3               $CHAR8.
        F4               $CHAR10.
        F5               $CHAR60.
        F6               $CHAR50.
 ;
	do while (not last);
	INPUT
        F1               : $CHAR28.
        F2               : $CHAR1.
        F3               : $CHAR8.
        F4               : $CHAR10.
        F5               : $CHAR60.
        F6               : $CHAR50.
;
		Rundt  = today();
		format Rundt date10.;

		filenm = fil2read;

		output;
	end;
RUN;

DATA WORK.testupd;
    INFILE output TRUNCOVER ;
    INPUT name $250.;/*Pull pipe in*/
    fil2read=name;
	INFILE dummy FILEVAR=fil2read END=last FIRSTOBS=1  LENGTH=len truncover
		 /*LRECL=292*/
        ENCODING="LATIN1"
        TERMSTR=CRLF
        DLM='^' 
        MISSOVER
        DSD ;

    LENGTH
        F1               $ 28	/* field1 */
        F2               $ 1	/* field2*/
        F3               $ 8  /* field3*/
        F4               $ 10  /* field4 */
        F5               $ 60	/* field5 */
;
    FORMAT
        F1               $CHAR28.
        F2               $CHAR1.
        F3               $CHAR8.
        F4               $CHAR10.
        F5               $CHAR60.
;
    INFORMAT
        F1               $CHAR28.
        F2               $CHAR1.
        F3               $CHAR8.
        F4               $CHAR10.
        F5               $CHAR60.
;
	do while (not last);
	INPUT
        F1               : $CHAR28.
        F2               : $CHAR1.
        F3               : $CHAR8.
        F4               : $CHAR10.
        F5               : $CHAR60.
;
		Rundt  = today();
		format Rundt date10.;

		filenm = fil2read;

		output;
	end;
RUN;

Re: PIPE imports only first line in each file

shl007 — Mon, 15 Jun 2020 18:38:04 GMT

I inadvertently copied my code above twice; just refer to one of the snippets.

Re: PIPE imports only first line in each file

Kurt_Bremser — Mon, 15 Jun 2020 19:00:21 GMT

You are working on UNIX. Are you sure the files use the Windows line separator CRLF, and not the UNIX-typical LF?

Re: PIPE imports only first line in each file

shl007 — Mon, 15 Jun 2020 20:31:19 GMT

Hi Kurt - not sure to your question. How can I tell which LF character is being used? If it is using a different-than-expected LF character, is there a way I can account for that in the code? Thanks for any tips!!

Re: PIPE imports only first line in each file

ballardw — Mon, 15 Jun 2020 20:36:15 GMT

@shl007 wrote:

Hi Kurt - not sure to your question. How can I tell which LF character is being used? If it is using a different-than-expected LF character, is there a way I can account for that in the code? Thanks for any tips!!

Your code shows the option TERMSTR=CRLF. Which tells SAS to read the file with the Windows standard CarriageReturn/LineFeed character combination as the end of line marker. This is an operating system default. If the file is actually a UNIX file which would use LF as the end of line marker then you have told the program with the CRLF that there is only one line. If the LF should be the end of line character than either use Termstr=LF or remove the Termstr option entirely which will then use the default for the operating system involved.

Re: PIPE imports only first line in each file

SASKiwi — Mon, 15 Jun 2020 20:38:21 GMT

@shl007 - What happens if you remove TERMSTR=CRLF?