Sending email notifications for alerts based on SAS Viya log messages
In an earlier post I wrote about setting up alerts on SAS log data captured and stored by the logging components of the SAS Viya Monitoring for Kubernetes framework. I also wrote about a limitation in the current version of Elasticsearch that ships with SAS Viya Monitoring for Kubernetes as at lts-2020.1), which does not allow alert notifications to be sent via email. Rather, the options are limited to webhook, Slack and Amazon Chime. In this post, we'll look at a workaround, which will allow for notification emails to be distributed for log alerts.
Disclaimer
We'll be using another third party (open source) tool here, which isn't provided or supported by SAS, or included with SAS Viya Monitoring for Kubernetes. This approach is merely a workaround for delivering alert notifications via email in the absence of a suitable destination type in Kibana that allows the sending of emails natively. Later versions of Elasticsearch, which will inevitably be incorporated into SAS Viya Monitoring for Kubernetes, do have an "email" destination type, so this functionality should become available eventually in any case without the need for this workaround.
Webhook alerts handler
While searching for a solution, I came across a couple of possible options. This post outlines what I consider to be the simplest approach. All that is needed beyond a Viya deployment that has SAS Viya Monitoring for Kubernetes deployed, is a docker image called odfe-alerts-handler. This handy little utility is essentially a web server that accepts a webhook containing email parameters as input, and connects to a mail server (connections for which are defined when the web server is started) to send a notification to an email address which is defined in the alert trigger in Kibana.
Let's see it in action.
1. Start odfe-alerts-handler
The odfe-alerts-handler is a docker image, so we start it with something like:
docker run --rm -p 30339:30339 youtous/odfe-alerts-handler --web.listen-address=":30339" --smtp.host="pdcesx21090.race.sas.com" --smtp.port=1025 --smtp.from="odfe@gelcorp.com" --smtp.default-subject="ODFE has fired an alert"
I picked a random port number, and entered my SMTP server's details. A message is displayed indicating that the alerts handler is up and running.
2. Define alert in Kibana
Next, we decide what log message we want to alert on and create an alert for it in Kibana. This process is outlined in detail in the ODfE doc page.
Assume we want to send an alert each time a certain word or phrase (e.g. a dataset name, a username, a specific error message) is printed to the logs. In Kibana's Alerting page, the monitor is created as per the normal process - nothing extra needs to be done to send an email alert. There are differences though when creating the destination and the trigger.
When defining the destination, select Custom webhook from the Type dropdown box. Then select the radio button to "Define endpoint by URL" and enter the address for the alerts handler, adding '/email' to the end of the URL.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Click Create to finish defining the destination, and move on to creating the trigger.
On the Create Trigger page, define the trigger's name and severity and review/validate the trigger condition and response as per normal. You only have to do things differently when it comes to configuring the actions. Once you give the action a name, select the destination you created from the dropdown box. When the 'Message' box appears, you must add the email address where the alert is to be delivered in a 'to:' parameter inserted before the message body. For example:
The three dashes separate the headers from the message body. You can click the link to "Send test message" to verify things are working as expected.
3. Validate
If set up properly, the recipient will receive an alert notification via email when the alert fires (or when "Send test message" is clicked).
If there are errors, they will be printed to the terminal window.
There you have it. A quick and easy workaround for sending email notification for alerts defined on log data.
Thank you for reading. Please leave a comment below to ask questions or share your own experiences.
