"Can you send me the logs?"
A familiar phrase to anyone who has ever asked for help with an error or performance issue in SAS.
For SAS 9 and earlier, and for SAS Viya 3.x, this usually means finding and sending one or more '.log' files on the filesystem.
However, in the latest releases of SAS Viya - sometimes referred to SAS Viya 4 to distinguish it from SAS Viya 3.x - the process of getting logs to send to someone else is a little different. SAS Viya services and jobs running on Kubernetes do not write out log files in the way they did in earlier releases. Instead, services and jobs stream log messages to stdout and stderr, and it's up to Kubernetes and the Kubernetes administrator to decide what - if anything - to do with them.
This is definitely a better design pattern for a scaleable, dynamic, resilient application running on Kubernetes. But we have to learn how to work with logs in a new way.
Just to clear up a potential point of confusion, in this post I'm not talking about the SAS logs you get when you run a SAS program that might contain data step and macro code, procedures and so on. The way those logs work in SAS Viya 4 has evolved slightly, but when you run a program in SAS Studio or SAS Model Studio, you still get the logs and the rest of the results much as you did before. SAS batch jobs, and even interactive 'line mode' in SAS Viya 4 still return SAS logs much as earlier versions, even if the details have changed a little. In this post, I'm talking about the logs output by server processes, running in Kubernetes pods, that your client application (like SAS Studio, SAS Visual Analytics, SAS Environment Manager or the SAS Command Line Interface) connects to.
There are several way to export SAS Viya log messages. Each of them have some pre-requisites in terms of software that needs to be installed, or credentials you need to use them. And none of them is the 'best' way by itself, so it's worth knowing them all:
If you are a Kubernetes administrator, or at least have some access to the Kubernetes cluster, you can export container logs from Mirantis Lens, a very slick and capable user interface for Kubernetes clusters. This is definitely the easiest way, but it has two limitations:
To use Mirantis Lens, you need a Windows, MacOS or Linux machine, and a Kubernetes kube config file that gives you access to at least see SAS Viya pods in the cluster.
This clip shows how:
How to export logs in Lens. Look for the download button, bottom right, in the log view.
Select the full-screen icon to see a larger version. |
The SAS Viya 4 Administration Resource Kit is an excellent collection of tools for administrators and anyone who deploys or maintains SAS Viya 4. One of them, the SAS Viya Log Download Tool is a utility that will download all (or a selection) of pod logs from a single Kubernetes namespace. You should specify a SAS Viya namespace, and you can optionally specify:
Its limitations are that:
However, it's strengths are that:
To use SAS Viya Log Download Tool you need a bash shell environment with python 3, the kubectl Kubernetes CLI, and a Kubernetes kube config file that gives you access to at least see SAS Viya pods in the cluster.
Here's an example of how you could use it - there are lots more in the Log Download Tool's documentation:
python3 viya-ark.py download-pod-logs -n sas --output-dir="/path/to/report/"
Our SAS Viya Monitoring for Kubernetes project provides an excellent monitoring and logging solution for SAS Viya on Kubernetes. The logging module is SAS' take on the EFK stack, using Elasticsearch, Fluent Bit and Kibana, and is customized for SAS Viya. It is easy for a Kubernetes administrator to install into the cluster, and it's great for collecting, categorising, exploring and displaying SAS Viya log messages.
This logging solution's limitations are that:
However, for exporting logs, this tool has some very significant advantages of over the other two methods:
To use Kibana's SQL workbench, you only need a web browser and login credentials! Well, you need SQL skills and a bit of knowledge of our indexes too.
This screenshot shows an example SQL query, and the Download button:
Three things to note:
You can of course export pod logs using kubectl logs, so it deserves a brief mention. kubectl logsis easily the least capable tool of all covered in this post. Its main advantage is being simple, and Kubernetes admins already have it and likely know how to use it.
Its limitations are that:
To use kubectl, you need it to be installed in your shell, and a Kubernetes kube config file that gives you access to at least see SAS Viya pods in the cluster.
Search the web for more on how to use it, but for example, this command would output logs for the sas-logon-app to a file called sas-logon-app.ndjson:
kubectl logs -l app=sas-logon-app --namespace=sasviya > sas-logon-app.ndjson
The latest release of SAS Viya Monitoring for Kubernetes at the time of writing is release 1.0.5, published on 15th March 2021. It includes a new, experimental tool, called the Export Logs tool. I have not had much time to experiment with it yet, and will write more about it in a future post, but it looks really nice. See the documentation here. To shamelessly copy and paste from that documentation:
To download the log messages, run the getlogs.sh script. This is the syntax for the script:
getlogs.sh -ns | -- namespace namespace [query options] [time period options] [output options] [connection options] [other options]
The -- namespace namespace option is required. This value specifies the Kubernetes namespace corresponding to the SAS Viya deployment for which you want to obtain log messages.
It runs in Bash on Linux, but not currently on MacOS. Please bear in mind that this new tool is experimental, and is likely to change over the next few releases of the project.
Its limitations are that:
As my wife keeps telling me, more Sterns are definitely a good thing. However, this particular Stern (no relation, honest!) can be very useful both in live analysis of logs, and capturing of relevant info: https://github.com/wercker/stern. From its readme: "Stern allows you to tail
multiple pods on Kubernetes and multiple containers within the pod. Each result is color coded for quicker debugging." When you want to watch something live, and you know the specific pod or even the container that will generate the message (even if there are many copies of that pod running on your cluster), Stern is terrific for observing those log messages in real time.
As with kubectl logs, the SAS Viya Log Download Tool and Lens, you only get the basic pod logs. You won't see the extra context fields, KUBE_EVENT messages, or other improvements made to log JSON documents in SAS Viya Monitoring for Kubernetes.
So, which is best? Honestly, there is no tool among this set which is 'the best' in all use cases.
I really like the SAS Viya Monitoring for Kubernetes logging stack, and Kibana. They have the best log data, and the richest tools for filtering, but SQL Workbench is a somewhat advanced tool, with a bit of a learning curve.
Lens is just gorgeous to use, but it's log export is pretty basic and limited to one pod at a time. For simple stuff, it's perfect.
The SAS Viya Log Download Tool is great for dumping all logs to a zip file. If you don't know which log you need someone to look at, this is the tool for you.
The last three also only offer the raw, un-enhanced pod logs with less context, and you have to hope the messages you want are within the most recent 10 MB cache.
I'm excited to learn more about the new Export Logs tool in SAS Viya Monitoring for Kubernetes, and every Kubernetes admin knows kubectl logs already.
So really, they are all useful, and I encourage you to become familiar with all these methods, so that when someone asks "Can you send me the logs?", your answer should be "Yes!"
See you next time!
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning and boost your career prospects.