Sending SAS Viya alert notifications to MS Teams

[#1]

Effective alerting requires that the right people are notified of issues at the right time and via the most appropriate channels. SAS Viya Administrators and DevOps engineers using Alertmanager to manage alerts in the SAS Viya platform can integrate Alertmanager with several third party party applications to distribute alert notifications. While native support is built-in for some applications, many others are capable of receiving information via webhooks (e.g. alert notifications sent by Alertmanager). In this post, we'll look at how Alertmanager can integrate with Microsoft Teams via webhook, allowing alert notifications to be automatically posted to a Microsoft Teams channel when alerts are triggered.

There are several ways to set up the integration. In any case, as there is no native support for Teams integration in Alertmanager, and because Teams requires messages to be in a certain format (one that Alertmanager doesn't produce its alerts in), a third-party tool must be deployed to act as an intermediary. There are many to choose from, but in this case I used an open-source utility called prometheus-msteams. It is essentially a lightweight web server which receives alert messages from Alertmanager and sends them to a Teams channel in the required format.

But first, the Teams channel must be configured. Generate the unique incoming webhook URL (which is where prometheus-msteams will send messages) in the desired Teams channel by right-clicking on the channel name and clicking Connectors.  

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

If it's not already added, click Add on the Incoming Webhook Connector. When it's added, click Configure.  

Provide a name and then click create to generate the unique URL. Copy/take note of this, as it is required later.  

Next, set up prometheus-msteams. There are several ways to deploy it, including as a binary or as a docker container. In my lab, I deployed it to my Kubernetes cluster (in the monitoring namespace deployed by SAS Viya Monitoring for Kubernetes) using the Helm chart. First, I added the chart repo as per the instructions with:

helm repo add prometheus-msteams https://prometheus-msteams.github.io/prometheus-msteams/

Then prepare the configuration file before deployment. This is where webhook URL from Teams must be specified.

tee ~/config.yaml > /dev/null << EOF
replicaCount: 1
image:
  repository: quay.io/prometheusmsteams/prometheus-msteams
  tag: v1.5.1

connectors:
# in alertmanager, this will be used as http://prometheus-msteams:2000/alertmanager
– alertmanager: https://sasoffice365.webhook.office.com/webhookb2/e501090a-af3d-426c-bf35-2374d5e7d211@b1c14d5c-3625-45b3-a430-9552373a0c2f/IncomingWebhook/4b5a85a99819453aae87cbce67ea6469/07d2cf3f-f87b-4c4c-a3d6-2f10f2993322
# # Enable metrics for prometheus operator

metrics:
  serviceMonitor:
    enabled: true
    additionalLabels:
      release: prometheus # change this accordingly
    scrapeInterval: 30s
EOF

To deploy, run:

helm upgrade --install prometheus-msteams --namespace v4mmon -f config.yaml  prometheus-msteams/prometheus-msteams

That results in: 

Release "prometheus-msteams" does not exist. Installing it now. 
NAME: prometheus-msteams 
LAST DEPLOYED: Wed Mar 15 06:52:49 2023 
NAMESPACE: v4mmon STATUS: deployed 
REVISION: 1 
TEST SUITE: None 
NOTES: ** Please be patient while the chart is being deployed ** 

To monitor the deployment, execute the following command: 

kubectl get pods -l app=prometheus-msteams --namespace v4mmon -w

Check that it has been deployed and is running:

kubectl get po -n v4mmon|grep msteams

The result:

prometheus-msteams-588777df8f-btmq2 1/1 Running 0 92s

The next step is to define a new receiver in the Alertmanager configuration for prometheus-msteams. In my lab, I made it the default receiver for all routes, so all alert notifications will get sent to Teams.

tee ~/alertmanager.yaml > /dev/null << EOF
global:
  smtp_smarthost: $(hostname -f):1025
  smtp_from: 'alertmanager@gelcorp.com'
  smtp_require_tls: false
  resolve_timeout: 5m
route:
  receiver: prometheus-msteams
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
receivers:
- name: prometheus-msteams
  webhook_configs:
  - url: "http://10.96.3.151:1975/alertmanager"
    send_resolved: true
EOF

Note that in this example, the URL is referring to the IP & port of the node where the prometheus-msteams pod is running. In my lab, I set up the prometheus-msteams service with a NodePort so that I could simulate test alerts using cURL (from outside the cluster).

Apply the updated configuration:

cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  alertmanager.yaml: $(cat ~/alertmanager.yaml | base64 -w0)
kind: Secret
metadata:
  name: alertmanager-v4m-alertmanager
  namespace: v4mmon
type: Opaque
EOF

Once the Alertmanager config is applied, it may take a minute to take effect (hint: keep an eye on the Status page in the Alertmanager UI). When it does, notifications will start appearing as posts in the Teams channel when the alerts begin firing.  

If you want to customize the way messages are posted, you can do that too by following the documented instructions and examples.

For more information refer to the GitHub page. Alternative approaches are also widely documented online.

If you'd rather use Slack than Teams, check out this previous post. Thanks for reading.

Find more articles from SAS Global Enablement and Learning here.