Catalina.out housekeeping

[shoin]

In large environments, catalina.out grows exponentially.  I am interested to explore what admin community has done to keep the size of catalina.out in check.  I have heard of using logrotate as well, has any one tested that?  environments are both WIN and *NIX

Thank you in advance

[AndreasWindisch]

Hey,
 
my solution is quite simple.
On a weekly base i clear the catalina.out, server.log and tm*.log.
 
Our systems are running on Linux X64.
 
Snippet of the script:

# Log cleanup on specific weekday
#
confweb=/opt/sas/Lev1/web
dayofweek=$(date '+%u')
case "$dayofweek" in
  2|4|7)
  echo -e "\n  Middle tier log clean up"
  truncate $(find "$confweb/Web/WebAppServer" -mindepth 3 -maxdepth 3 -name 'catalina.out') --size 0
  truncate $(find "$confweb/Web/WebAppServer" -mindepth 3 -maxdepth 3 -name 'server.log') --size 0
  truncate $(find "$confweb/Web/WebAppServer" -mindepth 3 -maxdepth 3 -name 'tm*.log') --size 0
  ;;
  *)
  echo -e "\n  No Middle tier log clean up"
  ;;
esac

Logrotation sounds like a advanced and interesting solution.

Best regards,
Andreas

[Mark_sas]

Here are some recommendations:

On Windows, most of the log messages are going to server.log. By default, that file is defined to be “rolling," meaning older messages will be purged. Hence, server.log should not become too large and you shouldn't have to modify your configuration.

On UNIX, most of the log messages are going to catalina.out. One option to make this file rotatable is by replacing server.log with catalina.out in config files.  For example, in SASConfig/Lev1/Web/WebAppServer/SASServer1_1/lib/log4j.xml, you'll find the appender definition:

   <appender name="FILE" class="org.apache.log4j.RollingFileAppender">

In it, you can replace server.log with catalina.out:

   <param name="File" value="${catalina.base}/logs/catalina.out"/>

This should make catalina.out a rolling file.

You can also control the size of the log file by updating the parameters in this appender.

[shoin]

@Mark_sas, much thank you intriguing. In my instance at /sas/config/Lev1/Web/WebAppServer/SASServer1_1/lib

I find:

<!-- <appender name="FILE" class="org.apache.log4j.FileAppender"> -->
<appender name="FILE" class="org.apache.log4j.RollingFileAppender">
<appender-ref ref="FILE"/>

I was not clear if I add a line as you suggested since I do not see server.log in the tag.

Thank you for your time!

[Mark_sas · Accepted solution]

Note: This solution has been modified to be more complete.

In order to maintain the size of the catalina.out file and keep that file from growing too large, you can periodically delete the catalina.out file when the corresponding SAS Web Application Server is stopped. When the SAS Web Application Server restarts, it will be writing to a new catalina.out file. You can make a step like this part of your regular maintenance and clear out the logs in the <SAS_CONFIG>/Lev1/Web/WebAppServer/SASServerX_Y/logs/ directory when the SAS Web Application Server(s) are stopped. That should be a feasible solution to maintain the size of that file (and the other logs in that directory also). You should not delete or rename the catalina.out file when it is actively being written to, so make sure to do this when the SAS Web Application Server is stopped.

In terms of log rotation - There is no default way to enable the catalina.out log file to roll over, from a SAS Web Application Server configuration standpoint. This file is not controlled by log4j mechanisms (like the other SAS Web Application Server log files), but rather it is a redirection of SYSOUT and SYSERR. There is no way from a SAS configuration standpoint to configure this log to roll over.

If you want to enable this log to roll over, you could look into using some other third-party or system log rotation utilities (like logrotate) that would re-name the catalina.out file at some interval. You can find some more information about that online. The SAS Web Application Server is based on a tomcat web application server, and catalina.out is not specific to the SAS Web Application Server but rather is part of the underlying tomcat server configuration. So if you do a search for something like 'tomcat catalina.out rotation' you'll find some more information about this and examples of how to use something like logrotate.
(Using a log rotation utility like this is not something SAS can offer support on, but you could look into this on your own if enabling the catalina.out file to roll over is a priority.)

[SteveInTally]

Interesting thread.

We have a SAS Visual Analytics implementation with a RollingFileAppender configured in the $CATALINA_BASE/lib/log4j.xml pointing to server.log.  This file is usually around 5-6MB in size.

And we also have configured in the $CATALINA_BASE/bin/setenv.sh a CATALINA_OUT pointing to catalina.out that is typically around 40GB in size and full of DEBUG level statements.  Oddly enough I cannot locate where any logging is currently configured to DEBUG level.  I've GREP'd the entire $SASCONFIG tree looking for xml files with levels set to DEBUG and cannot find any.

I'd like to lower the size of that catalina.out file if possible, or do away with it completely if the necessary diagnostics are already being routed to the server.log file.