Are updates available for my deployed Viya software? A use case for alerting on log data

[#1]

In my previous article, we saw how AlertManager can be used to set up alerts based on SAS Viya metrics collected by Prometheus as part of SAS Viya Monitoring for Kubernetes.

However, Viya's alerting capabilities don't stop there. It's also possible to set up alerts based on log messages captured by the EFK stack. In this article, we'll explore Kibana's alerting functionality and outline the process for defining alerts based on log data to address a specific use case: automatic notifications of available updates for your deployed Viya software.

As David Stern wrote previously, when SAS Viya Monitoring for Kubernetes is deployed alongside SAS Viya, log messages (stdout from SAS Viya pods, jobs, etc.) are collected and consolidated by Fluent Bit and stored in the Elasticsearch engine (SAS Viya Monitoring for Kubernetes uses the Open Distro for Elastic distribution). Kibana is then used to surface the captured log data in its web-based UI. Aside from just displaying logs, Kibana has a number of other useful features, including some built-in monitoring and alerting capabilities that administrators will find useful. The documentation is a good place to start for understanding the available features (and asking questions on the forum), but let's look at our specific use case. How can we use Kibana's alerting functionality to tell when updates are available for our Viya deployment?

The sas-update-checker Kubernetes cronjob is included by default in a SAS Viya deployment, and is scheduled to run automatically every Sunday evening. The job compares the deployed software with the software available in the SAS Container Registry. If there is no difference, you are assumed to be running the latest software, and a message like the following message is printed to the log:

The report command started
Deployed release 'lts-2020.1-20210211.1613070436601':
        Support level is 'SUPPORTED'
        Support ends '2022-11-18T15:34:03.151Z'
No new release available for deployed version 'lts-2020.1'.
No new release available for deployed cadence 'lts'.
The report command completed successfully

If updates are available, however, the output will look something like this:

The report command started
Deployed release 'stable-2020.0.5-20200918.1600465441168':
Support level is 'SUPPORTED'
Support ends '2021-01-17T03:01:41.267Z'
New release available for deployed version 'stable-2020.0.5': 'stable-2020.0.5-20200924.1600985745738'.
New content available at: 'stable-2020.0.5-20200924.1600985745738'.
Different versions:
'sas-annotations' version '2.5.0-20200824.1598272777757' has an available update '2.5.1-20200910.1599741481752'
'sas-backup-agent' version '2.10.6-20200904.1599240159114' has an available update '2.10.8-20200921.1600690954675'
'sas-connect' version '4.11.1-20200908.1599597211869' has an available update '4.11.1-20200921.1600711980180'
'sas-files' version '2.33.4-20200903.1599168366641' has an available update '2.33.6-20200922.1600741413515'
'sas-job-flow-scheduling' version '2.2.3-20200903.1599155515778' has an available update '2.2.5-20200922.1600743219690'
'sas-localization' version '1.4.3-20200916.1600292890148' has an available update '1.4.3-20200918.1600443688202'
'sas-search' version '2.27.5-20200903.1599168034067' has an available update '2.27.7-20200922.1600746056812'
No new release available for deployed cadence 'stable'.
The report command completed successfully

That's helpful information, but unless an administrator goes to the logs and looks specifically for those messages, they are easily missed. Here's where alerting can help bridge the gap.

Let's set up an alert to look for the phrase "New content available". Alerts can be configured from the Alerting page in Kibana.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

On the alerting dashboard, we're prompted first to create a monitor. The documentation tells us that monitors are jobs that query the Elasticsearch engine on a schedule. Click the button to Create monitor.

Specify a name, and then scroll down to the Define Monitor section. There are three methods of definition to choose from: visual graph, extraction query, and anomaly detector.

• Visual graph allows for a query to built "visually" by specifying query parameters and viewing the results in chart form as shown below. 

• Anomaly detection can be used to search for changes/fluctuations in the logging data. That's beyond the scope of this article, and not required for our simple use case.
• Extraction query allows for an expression to be crafted using Elastic's query DSL. Enter the query, and click Run to see the results displayed in the Extraction Query Response window on the right. See example below.

Note that for all options, an index must be specified. Let's enter viya_logs-*, matching the "default" index pattern for Viya applications.

At the bottom of the page, define a schedule at which you would like your monitor to query the data. Since the sas-update-checker runs once a week, let's run the monitor at the same frequency.

After clicking Create, we can move on to defining a trigger. A trigger defines the conditions that need to be met for an alert to fire. After specifying a name and severity, you can specify the conditions using Painless, the built-in scripting language. In this example, we can keep the default value for the trigger condition - we are simply looking for (counting) any instances of that phrase being written the logs (ctx.results[0].hits.total.value > 0 is pre-filled in the Trigger Condition window).

If we click Run, the condition is evaluated and the result printed in the Trigger Condition Response window. In this case, the condition is true, indicating that the phrase was found (and updates are available!).

Now we need to define a destination. The version of Kibana deployed with SAS Viya Monitoring for Kubernetes as at lts-2020.1 includes options for sending alert notifications via Slack, Amazon Chime, and custom webhook. In future releases, there will be a built-in facility for configuring mail servers as alert destinations. It's also possible to use webhooks to forward notifications to email addresses - we'll look at that in a future article. For this demo, let's configure a webhook. Note that I'm using a third-party site to test my webhook.

The last step is to configure an action to tell Kibana to send a notification to our destination when the condition is met. You can customise the message's contents as shown below.

After we've created the trigger, we're all done. When the monitor next runs (remember we specified a 7 day interval at the beginning), it will search for the message telling us an update is available. If it finds one, a HTTP POST message will be sent to our webhook address. For example:

This example was a basic one, but demonstrates the power of alerting on log data. Consider other examples, such as an alert being triggered on multiple failed login attempts, or when a particular data set is loaded and accessed. There are many potential applications where log alerts and other features built-in to the logging and monitoring components of the SAS Viya Monitoring for Kubernetes project can deliver exceptional value for administrators.

One final important note about updates - before you actually apply any updates, be sure to the read the documentation carefully. Pay close attention to the Deployment Notes section in the SAS Viya Administration Guide. Careful planning and preparation is crucial. 

My thanks go to Greg Smith and @DavidStern for their contributions. 

Thanks for reading. I hope the information provided has been helpful. Leave a comment below to ask questions or share your own experiences. Happy alerting!  

Find more articles from SAS Global Enablement and Learning here.

[JuanS_OCS]

Hi @AjmalFarzam ,

this is really great, and I think it gives a perfect example of a real world alert!

Now, in terms of the Viya updates, I've got a question: wouldn't it be even better this comes bundled out-of-the-box in the Admin dashboard? Most of software have got this messages not just in the logs, but in ready-to-go visual alerts. Just my 2 cents, an idea, perhaps for next release?

Best regards,

Juan

[AjmalFarzam]

Hi @JuanS_OCS, good question. Let me try to get an answer for you. There's a new 'notifications' plugin to the sas-viya CLI coming soon, but I'm not sure it will do exactly what you're asking here. I'll make some enquiries and provide an update when I have more information. 

Thanks!