Setting Up Alerting in Grafana for SAS Viya
Grafana provides comprehensive insights into your SAS Viya deployment's health and performance using metrics collected by Prometheus. New capabilites were recently added to Grafana to improve its alerting capabilities. In this post, we will walk you through the process of configuring alerting for your SAS Viya deployment in Grafana.
Grafana requires Prometheus or another compatible data source set up to gather metrics from SAS Viya components. This is the default when SAS Viya Monitoring for Kubernetes is deployed.
The Grafana configuration is stored in a configMap (v4m-grafana) in the monitoring namespace. The config stored here includes things like your SMTP server host and port. If Grafana is already deployed, running a set of commands like the below will toggle on email connectivity and specify your mail server details in the configMap to allow Grafana to send alert notifications via email.
# get configmap in .yaml
kubectl --namespace monitoring get configmap v4m-grafana --output=yaml > /tmp/v4m-grafana-configmap.yaml
# update yaml file to enable smtp and enter mail server host:port
sed -r -i "/kind: ConfigMap/ i\ \ \ \ [smtp] \\
\ \ \ enabled = true \\
\ \ \ \ host = mailserver.gelcorp.sas.com:25" /tmp/v4m-grafana-configmap.yaml
# apply updated yaml
kubectl -n monitoring replace --filename=/tmp/v4m-grafana-configmap.yaml
# restart grafana
kubectl -n ${_monitoringNS} delete po -l app.kubernetes.io/name=grafana
If Grafana isn't deployed, it's possible to specify these config parameters (and others) pre-deployment in the user.env file provided with SAS Viya Monitoring for Kubernetes.
Before actually getting to creating alerts, you must configure the notification channels to receive alerts. First, log on to Grafana and navigate to the "Alerting" section by clicking on the bell icon in the left sidebar.
Note that the main "Alert rules" page displays any other alerts configured in Prometheus. While there are limited options for interacting with these alerts, having the ability to view all alerts in one interface is very convenient. The best part is that when SAS Viya Monitoring for Kubernetes deploys Grafana, it is already configured to use Prometheus as a data source for metrics, so these alerts appear automatically without any additional config.
Click on the "Contact points" tab. This is where we configure our channels. A default one, grafana-default-email, is pre-defined for us. Click the pencil icon to open the edit page, where you can select the contact point type (Email) and then specify the destination address. Click the Test button after doing so to make sure everything is set up correctly for the recipient to receive the notification.
Save the changes.
Now head back to the Alert rules tab. The new alert we create in this page will be a Grafana managed alert (i.e. not a PrometheusRule). Click on "+ New alert rule" under Grafana alerts to begin adding the alert. Define the alert conditions based on SAS Viya metrics collected by Prometheus (using either the builder or code methods). Grafana's interface for creating alerts provides a rich set of options for controlling all aspects of your alert condition, and includes the best features of the Prometheus Expression Browser and then some. Specify or select the appropriate metrics, choose how they are filtered and evaluated, and then set the threshold values and evaluation frequency. For example, you could set a rule to trigger when memory usage exceeds a certain value for a specified duration.
Craft an informative rule name and message as well as other descriptive attributes that clearly indicate the issue. You will also be required to select (or create) a folder and group for your rule. In the "Notifications" section, select the key-value labels that will connect the alert to contact points (the equivalent of Alertmanager's routing function).
Save the alert. When the evaluation period lapses, and the alert condition is met, the alert will begin firing and a notification will be sent.
Adjust the threshold values, duration, or other settings as needed based on your alerting strategy and requirements.
We'll look at next steps and additional options in a future post. Stay tuned.
Thank you for reading.
