Special Character in Log File Hindering Full Import

[djbateman]

My company utilizes PROC PRINTTO to send the log to a txt file.  We then import this txt file and search it for error/warning messages so we know if our program ran successfully.  We create another file that tells us if the log is clean or not so we know if it is reliable to proceed with reviewing the output.  I have noticed that the final log report occasionally tells me that there are no errors found, but the output seems off.  When I manually search the log for errors or warnings, I find some.

I did some digging, but I realized that SAS is not importing the full log file.  The problem is that a greater than sign is getting translated into an arrow.  For some reason, SAS hits this arrow and stops importing.  I am wondering if anyone can help me understand why this is?  1) Is there some kind of encoding option I can apply to allow SAS to recognize/bypass this character?  2) Is there a reason SAS is translating ">" to "->" in this one case but not anywhere else throughout the log?

Below is a snippet to import the log file of interest (note that I create a .log file normally, but I converted it to .txt so I could upload here).  Please note that there are 3287 lines in the file, but only 1969 are getting read in because that is where the arrow first appears.  Like I said, this arrow is replacing a greater than sign, but you can see in other places in the code that the symbol is displaying as expected (e.g., lines 1973, 1979, 1988).

filename logfile "C:\ErrorWarningLogExample.txt";
data log;
	infile logfile length=linelen;
	input logline $varying200. linelen;
	logline=upcase(logline);
run;

[Tom · Accepted solution]

That SAS program in not going to modify any of the characters being read.

The most likely character that would cause that data step to TRUNCATE the input stream would be the Microsoft DOS End of File character.  There is an INFILE option you can use to prevent SAS from giving that character any special meaning.

Windows INFILE statement 

IGNOREDOSEOF

is used in the context of I/O operations on variable record format files. When this option is specified, any occurrence of ^Z is interpreted as character data and not as an end-of-file marker.

Note also that your program is also going to truncate any lines that are longer than 200 bytes.

Try this program instead:

filename logfile "C:\ErrorWarningLogExample.txt";
data log;
  infile logfile ignoredoseof truncover ;
  input logline $upcase200.;
run;

PS I would not call what that program is doing IMPORT.  To me import implies converting data from some foreign format.  A text file is not some foreign format. This program is just READing in the file.

[djbateman]

Thank you so much! The IGNOREDOSEOF seems to have worked. I'll start applying this to all my other programs moving forward. It makes me wonder how often my programs had issues that were never detected.

[Tom]

I looked at your example file and @Quentin is correct that the end of file character is appearing in line(s) generated by the MLOGIC option.

data _null_;
  infile logfile truncover encoding='any'
%if "&SYSSCP"="WIN" %then %do; IGNOREDOSEOF %end;
  ;
  input;
  if verify(_infile_,collate(32,126)||'09'x||'0c'x) then list;
run;

Notice that in addition to the FormFeed characters that SAS inserts to start each new page in the SAS log I had to also tell it to ignore TAB characters.  I looks like your original program file had tab characters in it. 

You should probably change your editor settings so that the tabs are replaces by spaces.  The tabs will not only cause trouble for people editing the program files they will cause you trouble when trying to analyze the dataset you created by reading in the LOG file.

It also had another non-ASCII character being used in this statement. 

infile SASlist dlm="�";

You should probably replace that with a hex literal to avoid having issues when switching between character encodings.

infile SASlist dlm="AC"x;

Also I would not use PROC IMPORT to read in a CSV file, especially one where you expect it to have a specific format.  Just write the data step to read it yourself.

[Quentin]

I think Tom is correct.

If you view the file in Notepad++, you can see some of what look like control characters:

I see that you are using macro quoting:

202        	%createcheck(chk=11, whr=%str(bmdtc>today()), keepvars=siteid subjid questid visit lbnam

When you do that, SAS will temporarily convert the > sign into non-printable characters.  It looks like MLOGIC does not automatically unquote the value, so you are ending up with non-printable characters in your log file, and one of them gets interpreted as and end-of-file character.  That's quite a gotcha.