Picking up the Slack – Send SAS Viya alert notifications to Slack channels
Once again we have a post about alerting in SAS Viya! This time, we will look at what is required to get Prometheus AlertManager (when deployed as part of SAS Viya Monitoring for Kubernetes) to send alert notifications to Slack channels. Slack is one of several receiver options that AlertManager can integrate with; in addition to being able to send email notifications, it can also be hooked up to PagerDuty, OpsGenie, and any other location that can receive a webhook. In fact, behind the scenes, webhooks are precisely how AlertManager sends notifications to Slack. Unsurprisingly, there is some setup involved both in AlertManager as well as in Slack, so you may need help from an administrator to perform some of the steps outlined here.
Slack setup
We need to enable the Slack workspace to accept webhooks sent by Alertmanager. Thankfully, this is a straightforward exercise in Slack. Simply go to the Slack API site, log in, and click on Your Apps (api.slack.com/apps). From here, click the green button to Create New App.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Click the option to Create the App from Scratch, then give your app a name (for example, SAS Incoming Webhook App) and select the target Slack workspace.
On the next page, you can configure the app. We only need the app to receive webhooks and post them to the Slack channel. You can set that up by clicking the Incoming Webhooks tile under the Add features and functionality heading.
On the Incoming Webhooks page, click the toggle button to activate the feature, and then click the Add New Webhook button at the bottom of the page to generate the API URL.
First, you need to select from the dropdown box the channel you want the app to post incoming webhooks to.
When you click Allow, you're taken back to the Incoming Webhooks page, but this time the webhook URL and sample cURL request fields are populated.
You can try copying the cURL command executing it from your SAS environment to test that it works; it should result in the "Hello World" message being printed to your designated Slack channel. What is really important though, is the Webhook URL. This is the URL which must be supplied to AlertManager to establish the link with your Slack channel. Copy it, and keep it handy for the next steps.
AlertManager setup
The other piece of the puzzle is to get AlertManager to connect to the new Slack app. Open a terminal connection to your Kubernetes cluster where SAS Viya Monitoring for Kubernetes is deployed. Assuming you haven't already customised your AlertManager configuration, you can create and apply a new configuration. (If you do have an AlertManager configuration that you wish to retain and add your Slack connection information to, you will need to modify the configuration by first extracting and decoding the contents of the alertmanager-v4m-alertmanager secret in the ops4viyamon namespace.)
Create the YAML template with the new configuration containing the Slack Webhook URL as the value for the slack_api_url global setting (alternatively, you can specify api_url at the receiver block level). Also create a new receiver block (called slack-receiver in my example below), where you can define any other desired customisations and configuration options.
tee ~/alertmanagercfg.yaml > /dev/null << EOF
global:
smtp_smarthost: pdcesx02190.race.sas.com:1025
smtp_from: 'alertmanager@gelcorp.com'
smtp_require_tls: false
slack_api_url: 'https://hooks.slack.com/services/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'
route:
receiver: slack-receiver
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receivers:
- name: slack-receiver
slack_configs:
- channel: '#sas-viya-production'
EOF
Then create the YAML template to update the secret containing the encoded contents of the above AlertManager configuration fie (~/alertmanagercfg.yaml):
tee ~/alertmanager-secret.yaml > /dev/null << EOF
apiVersion: v1
data:
alertmanager.yaml: $(cat ~/alertmanager.yaml | base64 -w0)
kind: Secret
metadata:
name: alertmanager-v4m-alertmanager
namespace: ops4viyamon
type: Opaque
EOF
Apply the file to the cluster to update the secret: kubectl apply -f ~/alertmanager-secret.yaml
You can verify the new configuration has been applied by accessing the AlertManager UI and navigating to the Status page. Look for your customisations and make sure the slack_configs block is there.
You'll notice some other settings have also been added automatically. These settings control how notifications are formatted in Slack, and they can all be customised to suit individual requirements.
That takes care of the necessary AlertManager steps, so all that is left to do now is wait for an alert to fire, or to create a new alert just to test our setup (refer to my previous article to quickly create a basic alert). The threshold for the test alert should be low enough for it trigger/fire immediately, so the best way to verify is to log on to the Slack workspace and head to the channel we linked to.
The notification should appear as a message posted by the Slack app as shown above. It includes a count of the number of firing alerts and the name of the Prometheus (CR) that sent the notification.
This is a simple example that can be taken much further. For instance, it's possible to customise the information that is sent to Slack, tag individual users, modify text colours, insert images, or send alerts to multiple Slack channels, and so on. There are a couple of obvious customisations that would make sense, such as supplying the correct URL to AlertManager in the Slack notification; the default notification shown above is a clickable link to AlertManager's internal URL, which doesn't work from outside the Kubernetes cluster. The documentation is a good starting point to learn about all the possibilities, but there are also many other resources available online, many with specific examples.
As always, thanks for reading. Hopefully you found the information useful. Leave a comment below to ask questions or share your own experiences.
