Exporting logs from SAS Viya on Kubernetes – a survey of five available methods

6 Likes

"Can you send me the logs?"

A familiar phrase to anyone who has ever asked for help with an error or performance issue in SAS.

For SAS 9 and earlier, and for SAS Viya 3.x, this usually means finding and sending one or more '.log' files on the filesystem.

However, in the latest releases of SAS Viya - sometimes referred to SAS Viya 4 to distinguish it from SAS Viya 3.x - the process of getting logs to send to someone else is a little different. SAS Viya services and jobs running on Kubernetes do not write out log files in the way they did in earlier releases. Instead, services and jobs stream log messages to stdout and stderr, and it's up to Kubernetes and the Kubernetes administrator to decide what - if anything - to do with them.

This is definitely a better design pattern for a scaleable, dynamic, resilient application running on Kubernetes. But we have to learn how to work with logs in a new way.

Just to clear up a potential point of confusion, in this post I'm not talking about the SAS logs you get when you run a SAS program that might contain data step and macro code, procedures and so on. The way those logs work in SAS Viya 4 has evolved slightly, but when you run a program in SAS Studio or SAS Model Studio, you still get the logs and the rest of the results much as you did before. SAS batch jobs, and even interactive 'line mode' in SAS Viya 4 still return SAS logs much as earlier versions, even if the details have changed a little. In this post, I'm talking about the logs output by server processes, running in Kubernetes pods, that your client application (like SAS Studio, SAS Visual Analytics, SAS Environment Manager or the SAS Command Line Interface) connects to.

Five tools for exporting logs

There are several way to export SAS Viya log messages. Each of them have some pre-requisites in terms of software that needs to be installed, or credentials you need to use them. And none of them is the 'best' way by itself, so it's worth knowing them all:

Mirantis Lens
The SAS Viya Log Download Tool from the SAS Viya 4 Administration Resource Kit
The SQL Workbench in Kibana from SAS Viya Monitoring for Kubernetes
kubectl logs
The experimental Export Logs tool (getlogs.sh) in SAS Viya Monitoring for Kubernetes

Mirantis Lens

If you are a Kubernetes administrator, or at least have some access to the Kubernetes cluster, you can export container logs from Mirantis Lens, a very slick and capable user interface for Kubernetes clusters. This is definitely the easiest way, but it has two limitations:

You can only export logs for one container at a time.
You can only export logs from the ~10 MB cache that Kubernetes stores for each running or recently ended pod. If the pod containing the service whose logs you want to see is very busy and outputs a lot of logs, old messages can be dropped from Kubernetes' cache before you get to see them, to make way for more recent log messages.
You only get the pod logs, not the log documents that SAS Viya for Kubernetes streams to Elasticsearch, enriched with additional fields for better context. At least you know (or can easily find) that context by looking at the pod's attributes (name, node etc). when you download its log messages.

To use Mirantis Lens, you need a Windows, MacOS or Linux machine, and a Kubernetes kube config file that gives you access to at least see SAS Viya pods in the cluster.

This clip shows how:

(view in My Videos)

How to export logs in Lens. Look for the download button, bottom right, in the log view.

Select the full-screen icon to see a larger version.

The SAS Viya Log Download Tool from the SAS Viya 4 Administration Resource Kit

The SAS Viya 4 Administration Resource Kit is an excellent collection of tools for administrators and anyone who deploys or maintains SAS Viya 4. One of them, the SAS Viya Log Download Tool is a utility that will download all (or a selection) of pod logs from a single Kubernetes namespace. You should specify a SAS Viya namespace, and you can optionally specify:

which pods to download logs from (otherwise all pods)
a limit to the number of messages to download from each pod (otherwise all available messages)
where to save out the logs (otherwise, it writes them to a directory named sas-k8s-logs in the current working directory)

Its limitations are that:

As with Mirantis Lens, you can only export logs from the ~10 MB cache that Kubernetes stores for each running or recently ended pod.
Being a command tool, not everyone will find SAS Viya Log Download Tool quite so easy to use as a visual UI like Mirantis Lens.
You only get the pod logs, not the log documents that SAS Viya for Kubernetes streams to Elasticsearch, enriched with additional fields for better context. At least you know (or can easily find) that context by looking at the pod's attributes (name, node etc). when you download its log messages.

However, it's strengths are that:

people who are comfortable with a command line, and have a beginner's familiarity with python and kubectl will find it easy to use, and
you can script it, with all the usual benefits this brings: the potential to reduce manual effort through automation, aid repeatability/reduce mistakes and the fact that a script can be scheduled, triggered in response to an event, or run ad-hoc.

To use SAS Viya Log Download Tool you need a bash shell environment with python 3, the kubectl Kubernetes CLI, and a Kubernetes kube config file that gives you access to at least see SAS Viya pods in the cluster.

Here's an example of how you could use it - there are lots more in the Log Download Tool's documentation:

python3 viya-ark.py download-pod-logs -n sas --output-dir="/path/to/report/"

The SQL Workbench in Kibana from SAS Viya Monitoring for Kubernetes

Our SAS Viya Monitoring for Kubernetes project provides an excellent monitoring and logging solution for SAS Viya on Kubernetes. The logging module is SAS' take on the EFK stack, using Elasticsearch, Fluent Bit and Kibana, and is customized for SAS Viya. It is easy for a Kubernetes administrator to install into the cluster, and it's great for collecting, categorising, exploring and displaying SAS Viya log messages.

This logging solution's limitations are that:

It is a containerised Kubernetes application. The Kubernetes administrator has to deploy it into the cluster before you can use it.
The Elasticsearch database can get quite large.
By default Elasticsearch keeps logs for 3 days - this may be longer than Kubernetes keeps some of the pod logs, but shorter than we became used to in earlier versions of SAS, where logs may have been kept permanently until the administrator did something to clear them down, or for e.g. 30 days.
To export log messages you have to write queries by hand and submit them through Kibana's SQL Workbench interface. This requires knowledge of both SQL and the way SAS Viya logs are organized in Elasticsearch indexes.
The SQL Workbench user interface is occasionally a bit flaky, especially when exporting larger numbers of log messages.

However, for exporting logs, this tool has some very significant advantages of over the other two methods:

Our logging stack, specifically our custom-configured Fluent Bit agent adds additional context fields to the raw log message 'documents' output by services to pod logs. This context includes the namespace, the pod, the container name and other things that are essential to understanding the message properly.
It also 'fixes' some quirks in the way services output or categorize lot messages. Those issues are not fixed in the pod logs.
It also has Kubernetes event messages, and sends them to Elasticsearch as if they were log messages, with logsource=KUBE_EVENT. This is very useful for debugging pods that won't start - and whose services therefore don't output any log messages.
Elasticsearch doesn't keep the last 10 MB of messages from each pod. Instead, it keeps the last 3 days of messages from each pod. It is sometimes the only place you can still see older messages from pods which output a lot of messages, and cycle through their 10 MB pod log cache quickly. Conversely, for pods which do not output many messages, sometimes this means there are fewer messages from that pod in Elasticsearch than in the pod logs.
It is easy to retrieve log messages from more than one pod at once. In fact, this is the default, unless you specify a pod name (or something which has the same effect as specifying the pod name) in the SQL where clause. Since SAS Viya has many instances of services with multiple pods, this allows you to see what happened in any of those pods instead of having to look in each of them one at a time. Think about messages output by CAS worker pods on large CAS deployments - you might not want to look through the logs for each worker node individually.
It also allows you to see a unified series of log messages spanning multiple services in the same set of results. SAS Viya is a highly distributed application and in some cases this is the only way to see logs for an interesting sequence of events that span several services.

To use Kibana's SQL workbench, you only need a web browser and login credentials! Well, you need SQL skills and a bit of knowledge of our indexes too.

This screenshot shows an example SQL query, and the Download button:

After running a query in Kibana's SQL Workbench page, you can download the results as a CSV, or in the other formats shown.

Select the image to see a larger version.
Mobile users: To view the image, select the "Full" version at the bottom of the page.

Three things to note:

Most of Kibana takes time filters and shows results in your local browser's time zone. Behind the scenes, Elasticsearch works in UTC, and the SQL Workbench does too. You will need to convert between your local time zone and UTC if you filter on @timestamp in your SQL where clause.
Currently, you are limited to querying one Elasticsearch index in SQL Workbench. We configure our logging solution to have one index per namespace, per UTC day. So queries spanning UTC days need to be split.
By default, SQL queries retrieve 200 rows. Add a LIMIT statement to the very end of your SQL query to change this, e.g. to a higher number.

kubectl logs

You can of course export pod logs using kubectl logs, so it deserves a brief mention. kubectl logsis easily the least capable tool of all covered in this post. Its main advantage is being simple, and Kubernetes admins already have it and likely know how to use it.

Its limitations are that:

Being a command tool, not everyone will find kubectl easy to use.
You can only export logs from one pod, or pods in the same group
It has limited filtering options.
You only get the basic pod logs, the same as with Mirantis Lens and the SAS VIya Log Download Tool. No extra context fields, no KUBE_EVENT messages, and none of the other fixes we make to the log documents in SAS Viya Monitoring for Kubernetes.

To use kubectl, you need it to be installed in your shell, and a Kubernetes kube config file that gives you access to at least see SAS Viya pods in the cluster.

Search the web for more on how to use it, but for example, this command would output logs for the sas-logon-app to a file called sas-logon-app.ndjson:

kubectl logs -l app=sas-logon-app --namespace=sasviya > sas-logon-app.ndjson

The experimental Export Logs tool (getlogs.sh) in SAS Viya Monitoring for Kubernetes

The latest release of SAS Viya Monitoring for Kubernetes at the time of writing is release 1.0.5, published on 15th March 2021. It includes a new, experimental tool, called the Export Logs tool. I have not had much time to experiment with it yet, and will write more about it in a future post, but it looks really nice. See the documentation here. To shamelessly copy and paste from that documentation:

To download the log messages, run the getlogs.sh script. This is the syntax for the script:

getlogs.sh -ns | -- namespace namespace [query options] [time period options] [output options] [connection options] [other options]

The -- namespace namespace option is required. This value specifies the Kubernetes namespace corresponding to the SAS Viya deployment for which you want to obtain log messages.

It runs in Bash on Linux, but not currently on MacOS. Please bear in mind that this new tool is experimental, and is likely to change over the next few releases of the project.

Its limitations are that:

Being a command tool, not everyone will find SAS Viya Log Download Tool quite so easy to use as a visual UI like Mirantis Lens.
It behaves a little quirkily when you limit the number of rows you want, but your query will span multiple Elasticsearch indexes, and you may get more rows than you asked for as a result.

Bonus Stern!

As my wife keeps telling me, more Sterns are definitely a good thing. However, this particular Stern (no relation, honest!) can be very useful both in live analysis of logs, and capturing of relevant info: https://github.com/wercker/stern. From its readme: "Stern allows you to tail multiple pods on Kubernetes and multiple containers within the pod. Each result is color coded for quicker debugging." When you want to watch something live, and you know the specific pod or even the container that will generate the message (even if there are many copies of that pod running on your cluster), Stern is terrific for observing those log messages in real time.

As with kubectl logs, the SAS Viya Log Download Tool and Lens, you only get the basic pod logs. You won't see the extra context fields, KUBE_EVENT messages, or other improvements made to log JSON documents in SAS Viya Monitoring for Kubernetes.

Conclusions

So, which is best? Honestly, there is no tool among this set which is 'the best' in all use cases.

I really like the SAS Viya Monitoring for Kubernetes logging stack, and Kibana. They have the best log data, and the richest tools for filtering, but SQL Workbench is a somewhat advanced tool, with a bit of a learning curve.

Lens is just gorgeous to use, but it's log export is pretty basic and limited to one pod at a time. For simple stuff, it's perfect.

The SAS Viya Log Download Tool is great for dumping all logs to a zip file. If you don't know which log you need someone to look at, this is the tool for you.

The last three also only offer the raw, un-enhanced pod logs with less context, and you have to hope the messages you want are within the most recent 10 MB cache.

I'm excited to learn more about the new Export Logs tool in SAS Viya Monitoring for Kubernetes, and every Kubernetes admin knows kubectl logs already.

So really, they are all useful, and I encourage you to become familiar with all these methods, so that when someone asks "Can you send me the logs?", your answer should be "Yes!"

See you next time!