SAS Viya Logging

[#1]

Log files are a really important source of information for administrators who need to monitor and troubleshoot SAS environments. In this blog I will look at logging in SAS Viya 3.2, answering questions such as where are the logs located?, how do you configure logging? and how do you find the relevant logs to troubleshoot a problem? 

Firstly, with all these new servers and microservices, where are the logs?

On each machine in a VIYA the deployment log files are stored in the configuration directory in:

/opt/sas/<deployment_name>/config/var/log/

The deployment name is set at install time. For the rest of the blog we will assume our deployment name is viya. That would result in the base directory for the logs being /opt/sas/viya/config/var/log/.

You can also access the logs from the standard Linux log location /var/log where a symlink viya links to the log directory within the configuration directory.

 Within the log directory each service/server will have its own directory named for the server/service (there can be a lot of them).

Then within that directory, for each instance of the server/service, there is a sub-directory which contains the logs.

The screenshot shows the logs for the identity service in /opt/sas/viya/config/var/log/identities/default.

In SAS VIYA there are some differences, and similarities with SAS 9.4 in how logging is configured, administered and operates. In SAS VIYA different types of components use different methods for controlling logging. The components can be broken into three categories

• SAS VIYA Servers, for example the Workspace Server, Spawner, CAS Server
• Microservices and web applications, for example Visual Analytics, the identity service or the authorization service.
• SAS Configuration Server

In this article I will look at the first two.

For SAS Viya servers logging is controlled much like SAS 9.4 using the SAS logging facility. Logging configuration is very flexible and is controlled by xml files in each servers configuration directory. The logging facility allows detailed changes to the logging configuration including changes to log file names, level of messages displayed, output destination, and format of messages.

The CAS server itself uses the logging facility. Logging is configured in the

"…/config/etc/cas/default/logconfig.xml" which is referenced by the cas.cfgloc option in the casconfig.lua file in the same directory.

CAS adds some CAS specific loggers designed to output messages about CAS, and some CAS specific conversion patterns which can add detail to the messages in the CAS logs. For the VIYA servers and CAS, a server restart is required to pick up changes to the logging configuration.

SAS Viya Microservices and Web Applications work differently. Logging levels for each microservice and web application are set in SAS Environment Manager. SAS Environment Manager supports:

• adding a logger
• changing the level of message output by an existing logger (e.g INFO to DEBUG)

You cannot currently change the format of the messages written to the log, send the log to a different destination, rollover the logs on a time-based schedule or change the name of the log file.

The log files for microservices are named in the format:

sas-sas-service_name_start_date_start_time.log.

The logging levels of microservices and web application can be changed dynamically in SAS Environment Manager without the need to restart most services (if you change a jvm parameter you will need to restart the service). VIYA microservice and web application loggers are documented here.

Let's take a quick look at adding a logger to get more information in the log. In this example we will add a new logger for the folder service. The folder service controls the folders where users store there content in VIYA 8.1.

In SAS Environment Manager as a SAS Administrator select Resources > Configuration > Definitions

In the list of definitions, select logging.level

Select the new icon and specify the following parameters:

• Services: select the services to which the logging level applies in our case the folder service
• Level: specify OFF, INFO, FATAL, WARN, ERROR, DEBUG, TRACE, or ALL we will set it to DEBUG
• Select the logger name to add. The logger names are hierarchical classification of messages. In our case we will select com.sas.folders to get messages about folders.

 Click Save to save our changes.

If you navigate to the log directory for the folder service you will see that debug messages are immediately generated and output to the folder service log.

If you want to see a demo of this process have a look at this youtube video.

You cannot delete a logger in SAS Environment Manager, however you should adjust the logging level of your new logger to INFO when you are done troubleshooting. Logging at levels higher than INFO can generate large volumes of messages.

As I already mentioned each VIYA server, microservice, and web application creates a separate log in a separate directory. In addition to this in most VIYA environments the log directories will be on multiple machines. This can make it challenging just to find the relevant logs to troubleshoot an issue. There is an interesting paper from SGF this year which looks at using Python to collect logged information about VA report performance from the microservice logs. Ansible a tool used to deploy SAS Viya is can also be used to automate saerching and retrieving logs in a multi-machine environment.

The VIYA administratiom documentation suggest using operating system commands to identify the log of interest:

• sudo grep ERROR: /var/log/sas/viya/*/default/* returns all logs with ERROR for services on the machine
• ls -ltr */* | grep "todays_date“ returns all logs that have changed today

Hopefully, this overview of SAS Viya logging will help with your troubleshooting efforts For more information you can see the logging section of the SAS VIYA 3.2 Administration guide. In addition I know that R&D are working hard on more robust tools for log collection and analysis which will be available in future releases.

[muse1988]

This article is very helpful. I am wondering if you also knew how to locate the specific logfiles of a job that has run recently. My understanding is log files of a certain time period are zipped up and store in /var/log/ etc, but I am trying to locate a specific job's log file (once run.)