Deploying SAS Viya monitoring for Kubernetes to a dark site

2 Likes

When you deploy SAS Viya onto a Kubernetes cluster it will need access to the container images that are hosted by SAS. These images are accessible through a SAS hosted container registry. This registry is accessible through the internet. If your cluster is allowed to access the internet then deployment is easy. However there are situations in which direct access to the internet from a Kubernetes cluster is not allowed. For a SAS Viya deployment you can create a mirror registry and download the container images to that registry. That process is described here in the official SAS documentation.

However how does that work for SAS Viya Monitoring for Kubernetes? In this blog I’m going to show you how you can configure the monitoring components of SAS Viya Monitoring for Kubernetes, to deploy to a Kubernetes cluster that has no access to the internet.

What is a dark site?

By default, when you deploy a Kubernetes cluster to one of the cloud providers, you will end up with a cluster that has internet access. Meaning that, your cluster will be able to access all the public container registries and download the available container images to run certain applications. This can be convenient, but this is not allowed in every situation.

A dark site is a site where the cluster will not be able to access the public container registries, because it has no connection to the internet. To be able to run applications on these Kubernetes cluster, you will need to create a private container registry and download all container images into that private registry.

What is SAS Viya monitoring for Kubernetes?

SAS Viya monitoring for Kubernetes is a github repository that contains assets to deploy the monitoring and logging components to monitor your SAS Viya deployment. Out of the box it contains dashboards that give you insight in what’s going on in your Viya deployment.

Under the hood SAS Viya monitoring uses components like Prometheus, Grafana and AlertManager. All these components are deployed through a Helm chart. Helm is a package manager for Kubernetes and makes the management of your Kubernetes application easy. It allows you to make modification to deployments. For example, modifying the location of where the images by the chart are pulled from.

For deploying to a dark site that is exactly what we are going to use!

How do I deploy SAS Viya monitoring for Kubernetes to a dark site?

To deploy SAS Viya monitoring for Kubernetes to a dark site, you can simply follow these steps

Download all container images used by the helm chart to your own container registry
Customize the deployment by modifying the default location of the container registry
Deploy SAS Viya monitoring for Kubernetes

In this blog I will focus on deploying the monitoring part. However similar steps can be used to deploy the logging part. Let’s have a closer look at each of these steps

Download container images

Before we start with the actual deployment of the monitoring part, we first need to download all of the container images used by the helm chart, to a container registry that the Kubernetes cluster can access. This is a list of the images that need to be downloaded

k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.2.1
quay.io/prometheus-operator/prometheus-operator:v0.51.2
quay.io/prometheus-operator/prometheus-config-reloader:v0.51.2
quay.io/thanos/thanos:v0.17.2
quay.io/prometheus/prometheus:v2.30.3
quay.io/prometheus/alertmanager:v0.23.0
quay.io/kiwigrid/k8s-sidecar:1.14.2
quay.io/prometheus/node-exporter:v1.2.2
docker.io/grafana/grafana:8.2.1
docker.io/bats/bats:v1.4.1
docker.io/library/busybox:1.31.1
docker.io/curlimages/curl:7.73.0
docker.io/prom/pushgateway:v1.3.0

Please be aware that the versions of the images mentioned here are specific to the Helm chart version that is used by SAS Viya Monitoring for Kubernetes. At the time of writing this blog the version that was used was 19.2.2. If a different version is used, the associated version numbers of the images might change. To find out which helm chart version is currently used check this link

To view the version used for the images in a specific version of the Helm chart past this url in your browser.

https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack/<VERSIONUMBER>

Replace <VERSIO NUMBER> with the correct version number of the Helm chart. Once you have opened the website in your browser, click on the button default values as shown here in the screenshot.

This will generate all of the default values for that specific version of the Helm chart and it allows you to browse through the contents to find out which version for a specific image is being used. You can search through the default values by using the search box. This allows you to filter on terms like image or tag, to find the information that you are looking for.

Customize the deployment

I'm going to assume that you already know how to apply customizations to the deployment of SAS Viya Monitoring for Kubernetes. If not please read this link here. If you want to know what can be modified, check the samples that are provided in the repository. To apply our modifications we are going to use the user-values files. These files will allow us to modify the default location of the container images, to point to the correct container registry. In our case this will be the registry where we stored the images.

For the monitoring part we need two user-values files that need to be stored in the USER_DIR directory

user-values-prom-operator.yaml
user-values-pushgateway.yaml

Creating the user-values-prom-operator file

To create this file please execute the below command

cat <<EOF> $USER_DIR/monitoring/user-values-prom-operator.yaml
#Place overrides for the Prometheus Operator Helm Chart Here

# Prometheus Operator Helm Chart
# https://https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
#
# CRDs
# https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md
#
# Default Values
# https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml

# This sample demonstrates path-based ingress

# NOTE: Edit hostnames and ingress port

global:
  imagePullSecrets:
   - name: <IMAGE-PULL-SECRET>

prometheusOperator:
  admissionWebhooks:
    failurePolicy: Fail
    enabled: true
    ## A PEM encoded CA bundle which will be used to validate the webhook's server certificate.
    ## If unspecified, system trust roots on the apiserver are used.
    caBundle: ""
    ## If enabled, generate a self-signed certificate, then patch the webhook configurations with the generated data.
    ## On chart upgrades (or if the secret exists) the cert will not be re-generated. You can use this to provide your own
    ## certs ahead of time if you wish.
    ##
    patch:
      enabled: true
      image:
        repository: <YOUR-REGISTRY>/ingress-nginx/kube-webhook-certgen
        sha: ""
        pullPolicy: IfNotPresent

  ## Prometheus-operator image
  ##
  image:
    repository: <YOUR-REGISTRY>/prometheus-operator/prometheus-operator
    sha: ""
    pullPolicy: IfNotPresent
  ## Alertmanager image to use for alertmanagers managed by the operator
  ##
  alertmanagerDefaultBaseImage: <YOUR-REGISTRY>/prometheus/alertmanager
  ## Prometheus-config-reloader
  ##
  prometheusConfigReloaderImage:
    repository: <YOUR-REGISTRY>/prometheus-operator/prometheus-config-reloader
    sha: ""
  ## Thanos side-car image when configured
  ##
  thanosImage:
    repository: <YOUR-REGISTRY>/thanos/thanos
    sha: ""

prometheus:
  # Disable default configuration of NodePort
  service:
    type: ClusterIP
    nodePort: null
  ingress:
    annotations:
      kubernetes.io/ingress.class: nginx
    enabled: true
    hosts:
    - <INGRESS-HOST>
  prometheusSpec:
    routePrefix: /prometheus
    externalUrl: https://<INGRESS-HOST>/prometheus
    ## Image of Prometheus.
    ##
    image:
      repository: <YOUR-REGISTRY>/prometheus/prometheus
      sha: ""

alertmanager:
  # Disable default configuration of NodePort
  service:
    type: ClusterIP
    nodePort: null
  ingress:
    annotations:
      kubernetes.io/ingress.class: nginx
    enabled: true
    hosts:
    - <INGRESS-HOST>
    paths:
    - /alertManager
  alertmanagerSpec:
    routePrefix: /alertManager
    externalUrl: https://<INGRESS-HOST>/alertManager
    ## Image of Alertmanager
    ##
    image:
      repository: <YOUR-REGISTRY>/prometheus/alertmanager
      sha: ""

grafana:
  # Additional configuration necessary to serve Grafana from a subpath
  "grafana.ini":
    server:
      protocol: http
      domain: <INGRESS-HOST>
      root_url: https://<INGRESS-HOST>/grafana
      serve_from_sub_path: true
  # Disable default configuration of NodePort
  service:
    type: ClusterIP
    nodePort: null
  ingress:
    annotations:
      kubernetes.io/ingress.class: nginx
    enabled: true
    hosts:
    - <INGRESS-HOST>
    path: /grafana
  image:
    repository: <YOUR-REGISTRY>/grafana/grafana
    sha: ""
    pullPolicy: IfNotPresent
   ## Optionally specify an array of imagePullSecrets.
   ## Secrets must be manually created in the namespace.
   ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
   ##
    pullSecrets:
     - <IMAGE-PULL-SECRET>
  testFramework:
    enabled: true
    image: "<YOUR-REGISTRY>/bats/bats"
    imagePullPolicy: IfNotPresent
  downloadDashboardsImage:
    repository: <YOUR-REGISTRY>/curlimages/curl
    sha: ""
    pullPolicy: IfNotPresent
  initChownData:
    ## If false, data ownership will not be reset at startup
    ## This allows the prometheus-server to be run with an arbitrary user
    ##
    enabled: true
    ## initChownData container image
    ##
    image:
      repository: <YOUR-REGISTRY>/library/busybox
      sha: ""
      pullPolicy: IfNotPresent
  ## Sidecars that collect the configmaps with specified label and stores the included files them into the respective folders
  ## Requires at least Grafana 5 to work and can't be used together with parameters dashboardProviders, datasources and dashboards
  sidecar:
    image:
      repository: <YOUR-REGISTRY>/kiwigrid/k8s-sidecar
      sha: ""
    imagePullPolicy: IfNotPresent
kube-state-metrics:
  image:
    repository: <YOUR-REGISTRY>/kube-state-metrics/kube-state-metrics
    pullPolicy: IfNotPresent
  imagePullSecrets:
   - name: "<IMAGE-PULL-SECRET>"
prometheus-node-exporter:
  image:
    repository: <YOUR-REGISTRY>/prometheus/node-exporter
    pullPolicy: IfNotPresent
  serviceAccount:
   # Specifies whether a ServiceAccount should be created
   create: true
   # The name of the ServiceAccount to use.
   # If not set and create is true, a name is generated using the fullname template
   imagePullSecrets:
    - name: <IMAGE-PULL-SECRET>
EOF

Creating the user-values-push-gateway file

To create this file please execute the below command

cat <<EOF> $USER_DIR/monitoring/user-values-pushgateway.yaml
# Chart: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-pushgateway
# Default values: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus-pushgateway/values.yaml

image:
  repository: <YOUR-REGISTRY>/prom/pushgateway
  pullPolicy: IfNotPresent

# Optional pod imagePullSecrets
imagePullSecrets:
 - name: "<IMAGE-PULL-SECRET>"

# persistentVolume:
#   enabled: true
#   size: 2Gi
#   storageClass: myStorageClass
EOF

Replacing the variables in both user-values files

Now that we have created both files we can modify the files to reflect the environment which we are going to deploy to. There are basically three things we need to modify

Ingress: you need to provide the fully qualified domain name assigned to the IP of the ingress controller
Registry: you need to provide the fully qualified domain name of your container registry
Secret: you need to provide the name of the secret that contains the login information for your registry

With regards to the secret. Make sure this secret is already available in the namespace in which you are about to deploy SAS Viya Monitoring for Kubernetes. For more information on how to create a secret for your container registry, please see this link

export secret=<YOUR SECRET>
export ingress=<YOUR INGRESS FQDN>
export REGISTRY=<YOUR REGISTRY>

#Values for prometheus operator chart
sed -i "s|<IMAGE-PULL-SECRET>|$secret|g"  $USER_DIR/monitoring/user-values-prom-operator.yaml
sed -i "s|<YOUR-REGISTRY>|$REGISTRY|g"  $USER_DIR/monitoring/user-values-prom-operator.yaml
sed -i "s|<INGRESS-HOST>|$ingress|g"  $USER_DIR/monitoring/user-values-prom-operator.yaml

#Values for prometheus push gateway
sed -i "s|<IMAGE-PULL-SECRET>|$secret|g"  $USER_DIR/monitoring/user-values-pushgateway.yaml
sed -i "s|<YOUR-REGISTRY>|$registry|g"  $USER_DIR/monitoring/user-values-pushgateway.yaml

Deploy SAS Viya Monitoring for Kubernetes

Now that we have configured the two user-values files, we are ready to deploy the monitoring part of SAS Viya monitoring for Kubernetes.

cd $HOME
git clone https://github.com/sassoftware/viya4-monitoring-kubernetes.git --branch 1.1.2 
cd viya4-monitoring
monitoring/bin/deploy_monitoring_cluster.sh
monitoring/bin/deploy_monitoring_viya.sh

Once these commands have completed successfully, the deployment will pull all necessary images from the container registry that you provided instead of the default public container registries. You can now access Grafana via the following URL

https://<YOUR-FQDN>/grafana

I've used path based routing in my example to make all monitoring applications available. If you would like to use host based routing, please check out the following example available here

Conclusion

Not all sites have the ability to access the internet by default. It's a good thing that the Helm chart used to deploy SAS Viya Monitoring for Kubernetes, offers you the flexibility to override the default values used in the chart by providing your own container registry. It takes a bit of work to get all of the images in your own registry, but once you have all required images in there, you are good to go!

Thank you for reading. I hope the information provided in this article has been helpful. Please leave a comment below to ask questions or share your own experiences.