Scan all logs in a folder and filter Errors and Warnings

[ammirabile]

Hi guys,

I tried to find something related to this topic but i couldn't find anything that solved my problem.

I am trying to read all logs that I am saving in Unix while running programs using shells. 

Basically, I need a program to scan for all files in a folder and print the errors and warnings in a table, I am new to this kind of work so any help will be appreciate.

PS: PIPE is not available to use.

Thanks

[andreas_lds]

You should post some lines of a log file (using the {i}-icon) and what you expect as result.

Have a look at the documentation of the function dopen, together with dread, dnum and dclose you can write a datastep extracting all filenames in a directory. If all file-names end with .log you can skip this step an use something like:

data work.WarningsAndErrors;
  infile "PATH/*.log" filename=_filename;
  input;

  /* now the variable _infile_ contains one line of a log-file */
  /* command to parse the line */
  /* conditional output if something interesting was found */
run;

The variable _filename contains the name of the file currently read. If you need that information in the table you have to copy it to a another variable.

[Kurt_Bremser · Accepted solution]

So you can log on directly and run your SAS jobs from the commandline, but you're locked out from using the commandline from SAS? Idiocy squared.

Run this from the commandline:

find /path -name \*.log -print -exec grep -e WARNING -e ERROR {} \; > $HOME/errwarn.lst

and then read $HOME/errwarn.lst with SAS. Note that you will get all names of log files, but only those with ERRORs or WARNINGs will have follow-up lines.

[ammirabile]

Thanks @Kurt_Bremser, this command will be enough for now.

[RW9]

If your new at this, perhaps this isn't the best task to start with?

Anyways, you can extract program run information by surrounding a run of the code with proc scaproc.  This procedure analyses all the code, inputs/outputs etc. and is far more effective than anything you will come up  with as there are many nuances which may affect your code.

http://support.sas.com/documentation/cdl/en/proc/61895/HTML/default/viewer.htm#a003199745.htm

If not, then you would need to read in each file, using SAS functions (as you don't have pipe):

http://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a000209538.htm

Then parse the file picking out various text parts.  

[Patrick]

@Kurt_Bremser's suggestion is a good one.

If you're running your SAS programs via batch then you could also add PROC SCAPROC (can be added to the batch command via parameter options allowing to run additional SAS command added to the called program).

PROC SCAPROC scans the log for you and you then could collect the result in a table.

[Tom]

I find that simple scans for the text ERROR or WARNING generates too many false positives, such as finding the code lines that produce error message instead of the actual error messages themselves.  SAS error messages contain ERROR in column one and contain a colon, although sometimes with extra characters between the two.

I have this script I have used for years to pull out key information from SAS logs.  It will optionally also search for selected non-error messages that indicate potential problems.  Like uninitalized variable warnings.

It also pulls key information lines from the head and tail of the log.

#!/bin/ksh
#----------------------------------------------------------------------
# saslog [-n] [-w] filelist
#----------------------------------------------------------------------
# Summarize a SAS log by getting header line, version number, scan for
# ERROR lines and tail with time spent.
#
# Options:
#  -n   Use the -n option on the egrep command to add line numbers
#
#  -w   Also scan for WARNING lines and NOTE lines that are potential
#       problems: Uninitialized variables, merge with multiple datasets
#       with repeating by groups, automatic character to number
#       conversion, substr out of bound, insufficient decimal places and
#       unknown formats.
#
#----------------------------------------------------------------------
# Example:
#
#  >saslog pr*_chk.log xyz.log
#  prdgchk_chk.log:
#  1 The SAS System 17:29 Wednesday, November 28, 2007
#  NOTE: SAS (r) Proprietary Software Release 8.2 (TS2M0)
#  ERROR: Execution terminated by an ABORT statement at line 284 column 91
#  ERROR: SAS ended due to errors.
#  ERROR: Errors printed on page 12.
#   real time 0.22 seconds
#   cpu time 0.20 seconds
#
#  prevchk_chk.log:
#  1 The SAS System 11:38 Friday, November 9, 2007
#  NOTE: SAS (r) Proprietary Software Release 8.2 (TS2M0)
#   real time 0.31 seconds
#   cpu time 0.23 seconds
#
#  xyz.log:
#  File not found
#
#----------------------------------------------------------------------
#
export LC_COLLATE=C
#-----------------------------------------------------------------------
# Search for any saslog options
#-----------------------------------------------------------------------
stop=0
number=
error='^ERROR.*:'
while [[ $# -gt 0 && $stop -eq 0 ]]; do
  case $1 in
    -w) error="$error"'|^NOTE: Format .* was not found'
        error="$error"'|^NOTE: .* repeats '
        error="$error"'|^NOTE: .* uninitialized '
        error="$error"'|^NOTE: .* converted '
        error="$error"'|^NOTE: .* decimal '
        error="$error"'|^NOTE: .* SUBSTR'
        error="$error"'|^WARNING.*:'
        error="$error"'|^      One or more lines were truncated\.'
        shift ;;
    -n) number=-n
        shift ;;
    *)  stop=1 ;;
  esac
done
#
#
for file in $*; do
#----------------------------------------------------------------------
# If $file does not exist and $file.log does then use that
#----------------------------------------------------------------------
  if [[ ! -f $file && -f ${file}.log ]] ; then file=${file}.log ; fi
#----------------------------------------------------------------------
# Display the filename with a COLON at the end.
#----------------------------------------------------------------------
  echo $file:
  if [ -f $file ] ; then
#----------------------------------------------------------------------
# Print the first line compressing the spaces between words.
#----------------------------------------------------------------------
    head -1 $file | sed 's/[^!-~][^!-~]*/ /g'
#----------------------------------------------------------------------
# Print the SAS (r) line with the version and release.
#----------------------------------------------------------------------
    grep 'SAS (r)' $file | head -1
#----------------------------------------------------------------------
# Print any lines that look like SAS error messages.
#
# NOTE: The cut command removes overstrike lines caused by OVP option.
#----------------------------------------------------------------------
    egrep $number -e "$error" $file | cut -d$'\r' -f1
#----------------------------------------------------------------------
# Print the SAS termination message, removing any titles, blank lines
# and compressing blanks and non printing characters.
#----------------------------------------------------------------------
    tail -20 $file | sed '1,/^NOTE: The SAS System used:/{d;}' \
      | grep -v SAS | sed 's/[^!-~][^!-~]*/ /g' | grep -v '^ *$'
  else
    echo 'File not found'
  fi
  echo
done