Testing Prometheus alerts (and more) in SAS Viya with promtool

[#1]

promtool is a command-line based utility delivered with Prometheus that performs some functions SAS and Kubernetes administrators will find useful. An earlier post demonstrated amtool, the equivalent CLI tool for Alertmanager. Where amtool focuses on verifying the Alertmanager configuration (including validation of alert notifications), promtool is geared towards querying and validating metrics and rules. When creating custom alerts for your Viya deployment, it can be particularly useful for testing the alert rules before you deploy them.

promtool is included as part of the Prometheus deployment, but can also be installed separately (requires golang) and run from anywhere. When SAS Viya Monitoring for Kubernetes is deployed, promtool can be found inside the Prometheus pod in the monitoring namespace at /bin/promtool.

Let's look at some examples of promtool in action.

Query metrics instantly from the command line

Run an instant PromQL query against your metric data, as you would in the Prometheus Expression Browser UI. The below example queries memory usage for the sas-visual-analytics container and displays the raw time-series metric data.

promtool query instant http://prometheus.gelcorp.com 'container_memory_usage_bytes{container="sas-visual-analytics"} / (1024 * 1024 * 1024)'

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

Query metric data over a range

Similar to above, but with values displayed over a specified period. For example, view the same data but in 30 second intervals over the last 5 minutes:

promtool query range http://prometheus.gelcorp.com 'container_memory_usage_bytes{container="sas-visual-analytics"} / (1024 * 1024 * 1024)' --step=30s --start=$(date -d '5 minutes ago' +'%s') --end=$(date -d 'now' +'%s')

This can be useful for determining where your alert thresholds should be.

Check alert and recording rules

Verify that your Prometheus rules are well-defined before deploying them. Put the rules in a rules file, and then use promtool to check them for syntax errors and misconfigured arguments. This also provides a nice way of validating automatic changes in a CI/CD environment. In this example, rules.yaml looks like:

groups:
  - name: custom-viya-alerts
    rules:
      - alert: ViyaMemoryUsage
        annotations:
          description: Total Viya 4 namespace memory usage is more than 20% of total memory capacity.
          summary: Alerting on Viya 4 namespace Memory usage
          runbook: https://gelgitlab.race.sas.com/GEL/workshops/PSGEL260-sas-viya-4.0.1-administration/-/blob/master/04_Observability/img/alerting/runbook.md
        expr: ((sum by (node) (container_memory_usage_bytes{container!~"POD",namespace="gelcorp",pod=~"sas-.+"})) / (sum by (node) (kube_node_status_capacity{resource="memory"})) * 100) > 20
        labels:
          severity: critical
      - alert: LowAvailability
        expr: (avg_over_time(up{pod=~"sas-.+"}[1w]) * 100) < 70

The command to validate it:

promtool check rules ./rules.yaml

The result will either shows the number of rules detected or display any errors encountered with syntax or structure.

Query labels

Find available values for a specified label name.

promtool query labels http://prometheus.gelcorp.com 'severity'

Again, this one can also be helpful when creating alerts; specifically, when assigning labels to alerts for use when sending alert notifications.

Unit testing of alerts

Another very nice feature provided by promtool is the ability to unit test rules. You can simulate conditions to verify that the (new) alerts would be triggered once deployed (e.g. conditions that have never happened before). The detailed steps are beyond the scope of this post (there are many examples available online, including in the offical doc), but the process requires you to create a test file and then pass it in as an argument to the 'test rules' subcommand.

And more...

There are several other functions available with promtool, including a command to verify the Prometheus configuration (useful if making changes). For a full list of options and examples, consult in the built-in documentation or the official Prometheus documentation.

Thanks for taking the time to read. 

Find more articles from SAS Global Enablement and Learning here.