Configuring Basic Authentication for Prometheus and Alertmanager
In a default deployment of SAS Viya Monitoring for Kubernetes, Prometheus and Alertmanager are deployed without any authentication mechanism. That is, anyone with the URL can access the Prometheus Expression Browser and Alertmanager UIs. But there is a way to make things a little more secure. In this article, we'll look at how basic authentication can be configured for the monitoring applications.
Applications are not certainly considered 'secure' simply with the addition of basic authentication, but doing so does add a layer of simple security and access control. There are some gaps and caveats, including but not limited to:
- This particular method requires that ingress be configured for each application endpoint
- It's not necessarily completely safe/secure, even over HTTPS
- Credentials are stored in a Kubernetes secret, which is base64 encoded (i.e. can be easily decoded)
- It requires maintenance (when passwords need to be changed)
Even so, basic authentication is probably better than no authentication, at least in some cases. Note though, that in many cases, there may not be a need to expose Prometheus and Alertmanager at all. They can function just fine in-cluster without ingress (though NodePorts should be disabled as a security measure, which will be soon be the default setting for SAS Viya Monitoring for Kubernetes) and in cases where external access is required for short periods, port-forwarding can be a good option.
This page of the GitHub repository provides more information on deploying logging and monitoring components with ingress. Samples are also provided for both host and path ingresses.
We will need to make some adjustments to the ingress configuration to implement basic-auth. But first, we need to create/define credentials in a password file. If installed, we can use htpasswd:
# below command will prompt for a password
htpasswd ./auth marcelo
... or we can run something like the following for the same result:
printf "marcelo:$(openssl passwd -crypt Biel$a)\n" >> auth
To add more user credentials, so that multiple users can access the applications with different credentials, simply re-run the command with the additional usernames and passwords. The resulting 'auth' file, which contains a list of users and encoded passwords (remember to limit access to it!), will be used to create a new Kubernetes secret. Run the following to create the secret in the monitoring namespace ("ops4viyamon" in this example):
kubectl create secret generic basic-auth --from-file auth -n ops4viyamon
Next, the ingress needs to be modified to use the credentials in the secret to implement basic authentication. Edit the ingresses (in the monitoring namespace) for Prometheus and Alertmanager by adding the following keys to the annotations block:
nginx.ingress.kubernetes.io/auth-realm: Authentication Required
nginx.ingress.kubernetes.io/auth-secret: ops4viyamon/basic-auth
nginx.ingress.kubernetes.io/auth-type: basic
The quick/lazy way to test this is to use Lens or kubectl edit ing, but the changes will not persist if you redeploy/upgrade. The better way would be to edit the ingress configuration in user-values-prom-operator.yaml. In any case, the result should end up looking something like:
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Once that's done for both the Prometheus and Alertmanager ingresses, we're ready test with a cURL command or by hitting the URL in a browser. This time, we should get a prompt for credentials:
Now, users will only be able to access the interfaces if they authenticate using the credentials specified.
Grafana has its own authentication layer, but there is something else to consider for Grafana. While Grafana connects to Prometheus (the datasource for the dashboards), the dashboards still work without additional configuration after basic authentication has been implemented, because the connection to Prometheus uses the internal ClusterIP service address to connect directly, and not via the ingress controller. Remember that we've configured basic-auth for the NginX controller rather than for Prometheus. The latter is also possible though, as Prometheus has built-in support for basic authentication, which doesn't require ingress at all (although there seems to be an issue getting it working for Alertmanager). Refer to this guide and this example for more information. Note that using another method may require changes to the connection information in the Grafana datasource config (to supply basic auth credentials).
The NginX documentation also contains more information, but of course it's also possible to use another ingress controller (e.g. Apache), or to configure the ingress controller to start in a container in the Prometheus pod in a Sidecar proxy pattern. Possibly a topic for a future article.
Find more articles from SAS Global Enablement and Learning here.
