Alertmanager administration with amtool

[#1]

Anyone who’s tried modifying the Alertmanager configuration knows how tedious a process it can be. For one, indentation in the configuration.yaml (or alertmanager.yaml) is important, and very little feedback is provided by Kubernetes or Alertmanager if something is invalid or incorrect. As a result, it can be tricky to determine whether the updates have actually taken effect. When the kubectl command to apply the new configuration is run, a message indicates the configuration was updated. But if there are problems with the configuration, such as indentation or syntax issues, they don’t become apparent until the administrator sees that configuration hasn’t actually been updated (by looking the Alertmanager UI, or worse; by realising that alerts aren’t behaving as they should). Even then, it can be difficult to identify exactly why the configuration wasn’t updated. Where is the invalid element or misplaced line break? Introducing amtool, Alertmanager’s command-line companion that provides, among several other neat functions for interacting with alerts, a way for the configuration to be validated to quickly identify issues.

amtool is a CLI tool bundled with Alertmanager. In SAS Viya Monitoring for Kubernetes, it is deployed and accessible from inside the Alertmanager pod with a command like: kubectl -n v4mmon exec -it alertmanager-v4m-alertmanager-0 -- amtool [command]).

This may not be the most convenient way to run it though, as there are some caveats, of which an obvious one is the requirement for direct access to the cluster (and there are others). Alternatively, if you set up Ingress for Alertmanager (or a NodePort for the service), you can install the amtool CLI anywhere, or even run it as a standalone Docker container. In my lab, I cloned the Prometheus repo and compiled the amtool binary per the instructions (requires Golang to be installed).

Back to our primary use case. Assume we want to update the Alertmanger configuration to define a new receiver. First, let's use amtool to query the existing configuration: amtool --alertmanager.url=http://alertmanager.server.gelcorp.com/ config show


global:
  resolve_timeout: 5m
  http_config:
    follow_redirects: true
  smtp_hello: localhost
  smtp_require_tls: true
  pagerduty_url: https://events.pagerduty.com/v2/enqueue
  opsgenie_api_url: https://api.opsgenie.com/
  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
route:
  receiver: "null"
  group_by:
  - job
  continue: false
  routes:
  - receiver: "null"
    match:
      alertname: Watchdog
    continue: false
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
receivers:
- name: "null"
templates:
- /etc/alertmanager/config/*.tmpl

Let's try adding a new route and receiver by loading a new configuration file (alertmanager.yaml), which looks like:

global:
  smtp_smarthost: mail.gelcorp.com:1025
  smtp_from: 'alertmanager@gelcorp.com'
  smtp_require_tls: false
  resolve_timeout: 5m
route:
  receiver: sas-email-alert
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
receivers:
- name: sas-email-alert
  email_configs:
 - to: sasadmin@gelcorp.com
    headers:
      Subject: 'New Alert - SAS Viya'
      send_resolved: true
    require_tls: false

Create a new YAML manifest to update the secret which stores the base64-encoded Alertmanager configuration, and apply it:

tee ~/alertmanager-secret.yaml > /dev/null << EOF
apiVersion: v1
data:
  alertmanager.yaml: $(echo $(cat ~/alertmanager.yaml | base64 -w0))
kind: Secret
metadata:
  name: alertmanager-v4m-alertmanager
  namespace: v4mmon
type: Opaque
EOF

kubectl apply -f ~/alertmanager-secret.yaml -n v4mmon

In the output, we get the following message:

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

But a look at the Alertmanager configuration shows the old configuration, which doesn’t include our updates. Despite the message printed to the terminal, the new configuration was not applied.

Here’s where amtool can help us troubleshoot. If we run the below command, we can figure out what’s broken:

amtool --alertmanager.url=http://alertmanager.server.gelcorp.com/ check-config alertmanager.yaml

The culprit: line 13. Sure enough, the email_configs block has an invalid element; the "to" field is incorrectly indented.

After adding a space to correct it, we can try checking it again. This time, it looks more positive:

This is the config file we should apply to ensure our Alertmanager configuration update is successful.

There are many other useful functions provided by amtool, including:

• Displaying alert routes: amtool config routes show


• Testing alert routes (in this example, to see where an alert with a specified label will be sent to): amtool config routes test severity=warning
• Viewing alerts: amtool alert



as well as a bunch of other cool stuff like managing silences and testing new alerts.

amtool provides a simple, quick and effective method for managing Alertmanager configuration and artifacts. Instructions can be scripted and scheduled, and very little setup is required to get started. The built-in documentation contains a comprehensive list of possible commands available to the administrators (hint: you can also enable bash auto-completion with: amtool --completion-script-bash).

Thanks for reading.

Ajmal Farzam  

Find more articles from SAS Global Enablement and Learning here.