topic Re: scan multiple files and write list of file name in which contain error in SAS Programming

scan multiple files and write list of file name in which contain error

nnl3256 — Thu, 29 Mar 2018 16:31:14 GMT

I have several SAS generated log files in c:\temp i.e. test1.log test2.log... I need a program that reads each log and writes to a SINGLE text file if the log contains patter ERROR:. the code I have can give me a list of log name in log window without format. Is there better way to generate a text file and format better?

%let path=c:\temp;
filename dirlist pipe 'dir "c:\temp" /s';
data dirlist ;
     length logname outname $100;
	 retain cnt;
     infile dirlist length=reclen ;
	 input buffer $varying256. reclen ;
	 
	 if index(buffer, ".log") gt 0 then do;
	 	cnt+1;
		call symput('totallog',cnt);
	 	logname ="&path\"||strip(scan( buffer, -1, ' ' ));
		s1=scan(logname, -1, '\');
  		outname=scan(s1, 1, '.');
		output;
	end;
	keep logname outname;
run ;

%put &totallog;
%let cnt=&totallog;

proc sql noprint; select logname, outname into 	:log1-:log&cnt, :out1-:out&cnt from dirlist; quit;  


%macro read;
%do i=1 %to &totallog;
	data out_&&out&i; 
	 	 infile "&&log&i" lrecl=40 pad;
	 	 input @1 line $; 
		 if  upcase(substr(left(line),1,6))='ERROR:' then output;
	run;

	proc sql noprint; select line from out_&&out&i; quit; 
	%if &sqlobs gt 0 %then %do;
		%let all=&all. out_&&out&i;
	%end;
%end;
%mend;

%let all=;  %put &all;
%read;
%put ***********Error files: &all***********;

Re: scan multiple files and write list of file name in which contain error

ErikLund_Jensen — Thu, 29 Mar 2018 19:19:16 GMT

I will suggest a different approach with a loop over a data set with file names. I have made an example based on code I am using. For each log file, the erroneus records are appended to the output file using the MOD file option. The code will not find all errors, because some sas errors are reported in the format ERROR 3-20: or some other number. So using prxmatch with a suitable pattern will find all errors. I will be happy to ansver further questions.

%let path=e:\logs\batchjobs;
%let outfile = e:\work\errorlist.txt;
filename dirlist pipe "dir ""&path"" /s";

%let filesfound = 0;
data dirlist (drop=rec folder file filesfound);  	
	length fullfilename folder file $255;
	retain folder;
	infile dirlist truncover end=eof;
	input rec $char300.;
	if left(rec) =: 'Directory of ' then folder = substr(rec,15);
	else if substr(rec,1,1) ne '' then do;
		file = substr(rec,37);
			if file not in ('.','..') then do;
			fullfilename = trim(folder)||'\'||substr(rec,37);
			filesfound + 1;
			output;
		end;
	end;
	if eof then call symputx('filesfound',filesfound);
run;
%put &=filesfound;

%macro read;
	%if &filesfound > 0 %then %do;

		* Get max filename length for formatting output;
		proc sql noprint;
			select max(length(fullfilename))  into :maxlen from dirlist;
		quit;
		%put &=maxlen;

		* Initiate output file;
		data _null_;
			file "&outfile";
			datetime = datetime();
			put "* Error file list generated at " datetime datetime.;
			put "* Error File" @%eval(&maxlen+2) 'Recno' @+2 'Error Line';
		run;

		* loop over files;
		%do i=1 %to &filesfound;

			data _null_; set dirlist (firstobs=&i obs=&i);
				call symput('thisinfile',fullfilename);
			run;

			data _null_;
				 infile "&thisinfile" lrecl=255;
				 file "&outfile" mod;
			 	 input;
				 recno = _N_;
				 if _infile_ =: 'ERROR:' then put "&thisinfile" @%eval(&maxlen+2) recno 5. @+2 _infile_;
			run;
		%end;
	%end;
%end;
%mend;
%read;

Re: scan multiple files and write list of file name in which contain error

nnl3256 — Thu, 29 Mar 2018 19:59:46 GMT

Thank you for the quick reply. In my C:\temp directory, there are several log files and no sub-directories. I’ve change path to C:\temp and run your code to generate dataset dirlist. I got an empty dataset.

Re: scan multiple files and write list of file name in which contain error

ErikLund_Jensen — Sun, 01 Apr 2018 18:27:05 GMT

Hi.

Interesting - It Works fine for me.

Try to insert a put rec $char200.; statement after the Input statement, It lists all input lines in the log, so you can see the lines that goes into the rest of the code. If you don't get the files listed, then something is wrong with the DIR command, otherwise it should be possible to see where the problem occurs. Use similar put- statements to see whether the if-constructs become true and what you get there.

Re: scan multiple files and write list of file name in which contain error

Tom — Thu, 05 Apr 2018 13:43:37 GMT

There are options you can add to your DIR command to make that part easier. To get just the names of files add the /b and /a-d options.

dir /s/b/a-d

There are easier ways to read the filenames. Use the TRUNCOVER option on the INFILE statement.

infile "dir c:\temp /s/b/a-d" pipe truncover ;
input filename $256. ;

Also if you are looking for actual SAS error messages in the log then you should look for lines that start with 'ERROR' and also contain and ':'. Sometimes SAS will insert information between the 'ERROR' and the colon.

Making sure it starts in column 1 will eliminate many false positives from lines that are just echoing code that might be used by a programmer to generate an error message like

if age < 0 then put 'ERROR: Invalid age. ' age=;

So putting it together you want something like this:

%let path=c:\temp;

data logfiles ;
  infile %sysfunc(quote(dir "&path" /b/s/a-d)) pipe truncover ;
  input filen $256. ;
  filename=filen ;
  if index(filename,'.') and lowcase(scan(filename,-1,'.'))='log' ;
  infile saslog filevar=filen end=eof ;
  nerrors=0;
  do lineno=1 by 1 while (not eof);
    input;
    if _infile_=:'ERROR' and index(_infile_,':') then do;
      if not nerrors then put filename= ;
      putlog lineno= _infile_ ;
      nerrors+1;
    end;
  end;
  keep filename nerrors ;
run;

If you just want the filenames with errors then you can make it even simpler and stop reading the LOG file when you get a hit.

%let path=c:\temp;

data logfiles ;
  infile %sysfunc(quote(dir "&path" /b/s/a-d)) pipe truncover ;
  input filen $256. ;
  filename=filen ;
  if index(filename,'.') and lowcase(scan(filename,-1,'.'))='log' ;
  infile saslog filevar=filen end=eof ;
  anyerrors=0;
  do while (not eof and not anyerrors);
    input;
    if _infile_=:'ERROR' and index(_infile_,':') then anyerrors=1;
  end;
  if anyerrors;
  keep filename ;
run;

Re: scan multiple files and write list of file name in which contain error

nnl3256 — Thu, 05 Apr 2018 13:23:45 GMT

Thank you so much, Tom. it works exactly as I expected.