Effective alerting requires that the right people are notified of issues at the right time and via the most appropriate channels. SAS Viya Administrators and DevOps engineers using Alertmanager to manage alerts in the SAS Viya platform can integrate Alertmanager with several third party party applications to distribute alert notifications. While native support is built-in for some applications, many others are capable of receiving information via webhooks (e.g. alert notifications sent by Alertmanager). In this post, we'll look at how Alertmanager can integrate with Microsoft Teams via webhook, allowing alert notifications to be automatically posted to a Microsoft Teams channel when alerts are triggered.
There are several ways to set up the integration. In any case, as there is no native support for Teams integration in Alertmanager, and because Teams requires messages to be in a certain format (one that Alertmanager doesn't produce its alerts in), a third-party tool must be deployed to act as an intermediary. There are many to choose from, but in this case I used an open-source utility called prometheus-msteams. It is essentially a lightweight web server which receives alert messages from Alertmanager and sends them to a Teams channel in the required format.
But first, the Teams channel must be configured. Generate the unique incoming webhook URL (which is where prometheus-msteams will send messages) in the desired Teams channel by right-clicking on the channel name and clicking Connectors.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
If it's not already added, click Add on the Incoming Webhook Connector. When it's added, click Configure.
Provide a name and then click create to generate the unique URL. Copy/take note of this, as it is required later.
Next, set up prometheus-msteams. There are several ways to deploy it, including as a binary or as a docker container. In my lab, I deployed it to my Kubernetes cluster (in the monitoring namespace deployed by SAS Viya Monitoring for Kubernetes) using the Helm chart. First, I added the chart repo as per the instructions with:
helm repo add prometheus-msteams https://prometheus-msteams.github.io/prometheus-msteams/
Then prepare the configuration file before deployment. This is where webhook URL from Teams must be specified.
tee ~/config.yaml > /dev/null << EOF replicaCount: 1 image: repository: quay.io/prometheusmsteams/prometheus-msteams tag: v1.5.1 connectors: # in alertmanager, this will be used as http://prometheus-msteams:2000/alertmanager – alertmanager: https://sasoffice365.webhook.office.com/webhookb2/e501090a-af3d-426c-bf35-2374d5e7d211@b1c14d5c-3625-45b3-a430-9552373a0c2f/IncomingWebhook/4b5a85a99819453aae87cbce67ea6469/07d2cf3f-f87b-4c4c-a3d6-2f10f2993322 # # Enable metrics for prometheus operator metrics: serviceMonitor: enabled: true additionalLabels: release: prometheus # change this accordingly scrapeInterval: 30s EOF
To deploy, run:
helm upgrade --install prometheus-msteams --namespace v4mmon -f config.yaml prometheus-msteams/prometheus-msteams
That results in:
Release "prometheus-msteams" does not exist. Installing it now.
LAST DEPLOYED: Wed Mar 15 06:52:49 2023
NAMESPACE: v4mmon STATUS: deployed
TEST SUITE: None
NOTES: ** Please be patient while the chart is being deployed **
To monitor the deployment, execute the following command:
kubectl get pods -l app=prometheus-msteams --namespace v4mmon -w
Check that it has been deployed and is running:
kubectl get po -n v4mmon|grep msteams
prometheus-msteams-588777df8f-btmq2 1/1 Running 0 92s
The next step is to define a new receiver in the Alertmanager configuration for prometheus-msteams. In my lab, I made it the default receiver for all routes, so all alert notifications will get sent to Teams.
tee ~/alertmanager.yaml > /dev/null << EOF global: smtp_smarthost: $(hostname -f):1025 smtp_from: 'email@example.com' smtp_require_tls: false resolve_timeout: 5m route: receiver: prometheus-msteams group_wait: 30s group_interval: 5m repeat_interval: 12h receivers: - name: prometheus-msteams webhook_configs: - url: "http://10.96.3.151:1975/alertmanager" send_resolved: true EOF
Note that in this example, the URL is referring to the IP & port of the node where the prometheus-msteams pod is running. In my lab, I set up the prometheus-msteams service with a NodePort so that I could simulate test alerts using cURL (from outside the cluster).
Apply the updated configuration:
cat <<EOF | kubectl apply -f - apiVersion: v1 data: alertmanager.yaml: $(cat ~/alertmanager.yaml | base64 -w0) kind: Secret metadata: name: alertmanager-v4m-alertmanager namespace: v4mmon type: Opaque EOF
Once the Alertmanager config is applied, it may take a minute to take effect (hint: keep an eye on the Status page in the Alertmanager UI). When it does, notifications will start appearing as posts in the Teams channel when the alerts begin firing.
If you want to customize the way messages are posted, you can do that too by following the documented instructions and examples.
For more information refer to the GitHub page. Alternative approaches are also widely documented online.
If you'd rather use Slack than Teams, check out this previous post. Thanks for reading.
Find more articles from SAS Global Enablement and Learning here.
Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.
If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website.
Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning and boost your career prospects.