A first look at logging in the new SAS Viya
The way the new SAS Viya (2020.1 and later) log messages are collected and made available to SAS administrators is very different from the way they were handled in SAS Viya 3 or SAS 9. In this post, I'll describe the overall idea of how logging works in SAS Viya 4, without getting into too much technical detail.
SAS Viya services run inside containers, which are in turn deployed in pods in a Kubernetes cluster. You could draw it like this:
A Viya service runs inside a Docker container, which is in a Kubernetes pod, on a node, in a Kubernetes cluster.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Let's look at how those log messages get from the process to the administrator's attention.
Logs go to stdout and stderr, not SAS .log files
Generally speaking, processes running inside containers in Kubernetes pods cannot write log files or anything else to a persistent file system. It is a feature of Kubernetes that most pods are supposed to be able to be automatically started and stopped as needed, with as many copies of each pod as Kubernetes is told to run, that can be run on the node which Kubernetes thinks they can run best. The services provided by those pods should continue to operate, with users, and to some extent administrators, being mostly unaware of all this orchestration. Just writing log files to a fixed location on the host machine's disk does not make much sense.
A few pods like those serving CAS, SAS Infrastructure Data Server (Postgres), SAS Configuration Server (Consul) and SAS Message Broker (Rabbitmq) are allocated Kubernetes 'persistent volumes' which are basically managed filesystem storage, so that they can save their state when the pods are stopped. But those pods are in a minority, and all SAS services write log messages. Using persistent volumes to write out log files is definitely not the right way to do things in Kubernetes.
Instead, SAS Viya 4 deployments follow the Kubernetes standard pattern: services and processes write log messages out to stdout and stderr. No more named log files in a specific path. That may take quite some getting used to!
Sidenote: This post is concerned with what happens to log messages once they have been written out by a service or process. SAS Administrators also need to think about managing log thresholds which control the least severe classification of log message that gets written out by SAS Viya services. If we lower the threshold, e.g. to debug or trace, the service will write out more - sometimes many more - log messages under the same workload. If we increase the threshold e.g. to warning or error, the service will write out fewer messages but we miss some useful details. This aspect of logging is something a Kubernetes administrator will know far less about than a SAS administrator should.
Kubernetes picks those messages up, and saves them in pod logs. Pod logs can be accessed from the Kubernetes command line interface kubectl, or via REST calls to the same Kubernetes API that kubectl and more or less everything else in Kubernetes uses. But out of the box Kubernetes doesn't do much more with them than that. What's more, SAS administrators might not have access to kubectl or the REST API for the cluster where Viya is hosted.
Even the Kubernetes administrator, who presumably knows the kubectl command line interface very well, will not much like using kubectl to search through hundreds of thousands of log messages every day. So, we need something else to give SAS Administrators and other users a way to get to those log messages.
There are many ways to manage and surface Kubernetes logs. One is to combine three tools as something generically called the EFK stack. There are many off-the-shelf and self-assembly ways to combine these tools, but as a general approach it is popular and well-established.
The EFK Stack: Elasticsearch + Fluentd/Fluent Bit + Kibana
An EFK stack collects log messages from the pod logs on each Kubernetes node. Some implementations of EFK stacks use a component called Fluentd; other implementations, including ours, opt to use its lightweight sibling, Fluent Bit. Fluent Bit in our EFK stack performs a similar function to sas-watch-log in the SAS Viya 3 operations infrastructure.
In our implementation, Fluent Bit runs on every Kubernetes node, and streams pod logs over the Kubernetes internal network into Elasticsearch, a text-indexing-and-searching engine (dare I say database?) which stores the log messages and their contextual data. This contextual data includes fields such as timestamp, source process, container, pod, namespace, node, severity, process id, process owner etc. Elasticsearch stores all this for each log message, along with highly-searchable indexes. The last part of the 'stack', Kibana, reads data from Elasticsearch, applies management policies and permissions to it, and presents the collated, re-formatted and better-organised log messages to administrators a nice web interface.
Customers who have a lot of experience with Kubernetes, or have already incorporated Kubernetes into their IT ecosystem before they deploy SAS Viya 4 will almost certainly already have some sort of logging solution for their cluster. It may be a flavor of the EFK stack. Those customers will likely use their existing logging toolset to manage and work with SAS Viya logs. But they may need our help to fine-tune the way SAS log messages are collected, re-formatted, stored in separate indexes per SAS deployment, and presented in Kibana or another tool's reports.
Customers who do not already have Kubernetes and are adopting it at the same time as deploying SAS Viya 4 may choose to use a lightly-customised deployment of the EFK stack, which constitutes the logging part of our SAS Viya Monitoring for Kubernetes project.(https://github.com/sassoftware/viya4-monitoring-kubernetes)
Logs are created by every SAS Viya service, in each container in each pod, on every node where a SAS Viya pod runs in the cluster. They are gathered first in Kubernetes pod logs.
Every process, including every Viya process, in every container in every pod in the Kubernetes cluster writes out log messages to stdout and stderr. These are collected first by Kubernetes in pod logs.
When our SAS Viya Monitoring for Kubernetes project's logging solution is deployed to the cluster, it puts a Fluent Bit component on each node in the cluster. It uses a Kubernetes object called a daemonset to ensure each node gets one instance of Fluent Bit.
The EFK stack is deployed on the Kubernetes cluster, placing Fluent Bit on each node. Fluent Bit streams log messages from the pod logs into Elasticsearch, from where they can be surfaced in Kibana.
The EFK stack is deployed on the Kubernetes cluster, placing Fluent Bit on each node. Fluent Bit streams log messages from the pod logs into Elasticsearch, from where they can be surfaced in Kibana.[/caption]
As part of the SAS Viya Monitoring for Kubernetes project, our version of the EFK stack has had some significant configuration work done on the Fluent Bit component, to adjust the format and metadata for some log messages and quite a bit else besides. We have also configured Elasticsearch's ingest pipeline to store log messages from different SAS Viya deployments in different Elasticsearch indexes - its equivalent of tables.
Log messaged watched by Fluent Bit are streamed to an Elasticsearch ingest pipeline, which splits into separate indexes per SAS deployment
Because log messages from different SAS Viya deployments in the same Kubernetes cluster are put in different Elasticsearch indexes, and because Kibana and Elasticsearch allow different users and group to be granted different levels of access to indexes, you can control who sees logs for each SAS Viya deployment in the cluster. Where two customers called e.g. ACME and ABCO are deployed in the same Kubernetes cluster, you can manage administrator access to them in the way illustrated in this diagram:
The ACME and ABCO SAS administrators can be granted access to see log messages for their own deployments, and nothing else in the Kubernetes cluster. However, a Kubernetes cluster administrator may be granted access to see and manage log messages from both SAS Viya deployments, as well as other sets of logs from other pods in the cluster.
I really like that the logging subsystem provided by the EFK stack in SAS Viya Monitoring for Kubernetes does not depend on any SAS service to continue working. In SAS Viya 3, the Operations Infrastructure depended on CAS, among other things, to surface log messages in SAS Environment Manager. This means that if CAS is down, you should see error messages in Kibana from CAS, whereas in Viya 3, you likely would not see error (or any!) messages in the Logs page of SAS Environment manager.
Our SAS Viya Monitoring for Kubernetes implementation of the EFK stack does not require an additional license fee for SAS customers who choose to use it. However, the components do of course consume cluster resources and are therefore not entirely 'free' to run - those cluster resources have a cost. Cloud vendors generally make it quite easy to see what the running costs are for each namespace. This allows the customer to see the running costs of the logging solution they deploy and use.
Find more articles from SAS Global Enablement and Learning here.
See you next time!
