When you deploy SAS Viya onto a Kubernetes cluster it will need access to the container images that are hosted by SAS. These images are accessible through a SAS hosted container registry. This registry is accessible through the internet. If your cluster is allowed to access the internet then deployment is easy. However there are situations in which direct access to the internet from a Kubernetes cluster is not allowed. For a SAS Viya deployment you can create a mirror registry and download the container images to that registry. That process is described here in the official SAS documentation.
However how does that work for SAS Viya Monitoring for Kubernetes? In this blog I’m going to show you how you can configure the monitoring components of SAS Viya Monitoring for Kubernetes, to deploy to a Kubernetes cluster that has no access to the internet.
By default, when you deploy a Kubernetes cluster to one of the cloud providers, you will end up with a cluster that has internet access. Meaning that, your cluster will be able to access all the public container registries and download the available container images to run certain applications. This can be convenient, but this is not allowed in every situation.
A dark site is a site where the cluster will not be able to access the public container registries, because it has no connection to the internet. To be able to run applications on these Kubernetes cluster, you will need to create a private container registry and download all container images into that private registry.
SAS Viya monitoring for Kubernetes is a github repository that contains assets to deploy the monitoring and logging components to monitor your SAS Viya deployment. Out of the box it contains dashboards that give you insight in what’s going on in your Viya deployment.
Under the hood SAS Viya monitoring uses components like Prometheus, Grafana and AlertManager. All these components are deployed through a Helm chart. Helm is a package manager for Kubernetes and makes the management of your Kubernetes application easy. It allows you to make modification to deployments. For example, modifying the location of where the images by the chart are pulled from.
For deploying to a dark site that is exactly what we are going to use!
To deploy SAS Viya monitoring for Kubernetes to a dark site, you can simply follow these steps
In this blog I will focus on deploying the monitoring part. However similar steps can be used to deploy the logging part. Let’s have a closer look at each of these steps
Before we start with the actual deployment of the monitoring part, we first need to download all of the container images used by the helm chart, to a container registry that the Kubernetes cluster can access. This is a list of the images that need to be downloaded
Please be aware that the versions of the images mentioned here are specific to the Helm chart version that is used by SAS Viya Monitoring for Kubernetes. At the time of writing this blog the version that was used was 19.2.2. If a different version is used, the associated version numbers of the images might change. To find out which helm chart version is currently used check this link
To view the version used for the images in a specific version of the Helm chart past this url in your browser.
Replace <VERSIO NUMBER> with the correct version number of the Helm chart. Once you have opened the website in your browser, click on the button default values as shown here in the screenshot.
This will generate all of the default values for that specific version of the Helm chart and it allows you to browse through the contents to find out which version for a specific image is being used. You can search through the default values by using the search box. This allows you to filter on terms like image or tag, to find the information that you are looking for.
I'm going to assume that you already know how to apply customizations to the deployment of SAS Viya Monitoring for Kubernetes. If not please read this link here. If you want to know what can be modified, check the samples that are provided in the repository. To apply our modifications we are going to use the user-values files. These files will allow us to modify the default location of the container images, to point to the correct container registry. In our case this will be the registry where we stored the images.
For the monitoring part we need two user-values files that need to be stored in the USER_DIR directory
To create this file please execute the below command
cat <<EOF> $USER_DIR/monitoring/user-values-prom-operator.yaml #Place overrides for the Prometheus Operator Helm Chart Here # Prometheus Operator Helm Chart # https://https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack # # CRDs # https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md # # Default Values # https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml # This sample demonstrates path-based ingress # NOTE: Edit hostnames and ingress port global: imagePullSecrets: - name: <IMAGE-PULL-SECRET> prometheusOperator: admissionWebhooks: failurePolicy: Fail enabled: true ## A PEM encoded CA bundle which will be used to validate the webhook's server certificate. ## If unspecified, system trust roots on the apiserver are used. caBundle: "" ## If enabled, generate a self-signed certificate, then patch the webhook configurations with the generated data. ## On chart upgrades (or if the secret exists) the cert will not be re-generated. You can use this to provide your own ## certs ahead of time if you wish. ## patch: enabled: true image: repository: <YOUR-REGISTRY>/ingress-nginx/kube-webhook-certgen sha: "" pullPolicy: IfNotPresent ## Prometheus-operator image ## image: repository: <YOUR-REGISTRY>/prometheus-operator/prometheus-operator sha: "" pullPolicy: IfNotPresent ## Alertmanager image to use for alertmanagers managed by the operator ## alertmanagerDefaultBaseImage: <YOUR-REGISTRY>/prometheus/alertmanager ## Prometheus-config-reloader ## prometheusConfigReloaderImage: repository: <YOUR-REGISTRY>/prometheus-operator/prometheus-config-reloader sha: "" ## Thanos side-car image when configured ## thanosImage: repository: <YOUR-REGISTRY>/thanos/thanos sha: "" prometheus: # Disable default configuration of NodePort service: type: ClusterIP nodePort: null ingress: annotations: kubernetes.io/ingress.class: nginx enabled: true hosts: - <INGRESS-HOST> prometheusSpec: routePrefix: /prometheus externalUrl: https://<INGRESS-HOST>/prometheus ## Image of Prometheus. ## image: repository: <YOUR-REGISTRY>/prometheus/prometheus sha: "" alertmanager: # Disable default configuration of NodePort service: type: ClusterIP nodePort: null ingress: annotations: kubernetes.io/ingress.class: nginx enabled: true hosts: - <INGRESS-HOST> paths: - /alertManager alertmanagerSpec: routePrefix: /alertManager externalUrl: https://<INGRESS-HOST>/alertManager ## Image of Alertmanager ## image: repository: <YOUR-REGISTRY>/prometheus/alertmanager sha: "" grafana: # Additional configuration necessary to serve Grafana from a subpath "grafana.ini": server: protocol: http domain: <INGRESS-HOST> root_url: https://<INGRESS-HOST>/grafana serve_from_sub_path: true # Disable default configuration of NodePort service: type: ClusterIP nodePort: null ingress: annotations: kubernetes.io/ingress.class: nginx enabled: true hosts: - <INGRESS-HOST> path: /grafana image: repository: <YOUR-REGISTRY>/grafana/grafana sha: "" pullPolicy: IfNotPresent ## Optionally specify an array of imagePullSecrets. ## Secrets must be manually created in the namespace. ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ ## pullSecrets: - <IMAGE-PULL-SECRET> testFramework: enabled: true image: "<YOUR-REGISTRY>/bats/bats" imagePullPolicy: IfNotPresent downloadDashboardsImage: repository: <YOUR-REGISTRY>/curlimages/curl sha: "" pullPolicy: IfNotPresent initChownData: ## If false, data ownership will not be reset at startup ## This allows the prometheus-server to be run with an arbitrary user ## enabled: true ## initChownData container image ## image: repository: <YOUR-REGISTRY>/library/busybox sha: "" pullPolicy: IfNotPresent ## Sidecars that collect the configmaps with specified label and stores the included files them into the respective folders ## Requires at least Grafana 5 to work and can't be used together with parameters dashboardProviders, datasources and dashboards sidecar: image: repository: <YOUR-REGISTRY>/kiwigrid/k8s-sidecar sha: "" imagePullPolicy: IfNotPresent kube-state-metrics: image: repository: <YOUR-REGISTRY>/kube-state-metrics/kube-state-metrics pullPolicy: IfNotPresent imagePullSecrets: - name: "<IMAGE-PULL-SECRET>" prometheus-node-exporter: image: repository: <YOUR-REGISTRY>/prometheus/node-exporter pullPolicy: IfNotPresent serviceAccount: # Specifies whether a ServiceAccount should be created create: true # The name of the ServiceAccount to use. # If not set and create is true, a name is generated using the fullname template imagePullSecrets: - name: <IMAGE-PULL-SECRET> EOF
To create this file please execute the below command
cat <<EOF> $USER_DIR/monitoring/user-values-pushgateway.yaml # Chart: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-pushgateway # Default values: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus-pushgateway/values.yaml image: repository: <YOUR-REGISTRY>/prom/pushgateway pullPolicy: IfNotPresent # Optional pod imagePullSecrets imagePullSecrets: - name: "<IMAGE-PULL-SECRET>" # persistentVolume: # enabled: true # size: 2Gi # storageClass: myStorageClass EOF
Now that we have created both files we can modify the files to reflect the environment which we are going to deploy to. There are basically three things we need to modify
With regards to the secret. Make sure this secret is already available in the namespace in which you are about to deploy SAS Viya Monitoring for Kubernetes. For more information on how to create a secret for your container registry, please see this link
export secret=<YOUR SECRET> export ingress=<YOUR INGRESS FQDN> export REGISTRY=<YOUR REGISTRY> #Values for prometheus operator chart sed -i "s|<IMAGE-PULL-SECRET>|$secret|g" $USER_DIR/monitoring/user-values-prom-operator.yaml sed -i "s|<YOUR-REGISTRY>|$REGISTRY|g" $USER_DIR/monitoring/user-values-prom-operator.yaml sed -i "s|<INGRESS-HOST>|$ingress|g" $USER_DIR/monitoring/user-values-prom-operator.yaml #Values for prometheus push gateway sed -i "s|<IMAGE-PULL-SECRET>|$secret|g" $USER_DIR/monitoring/user-values-pushgateway.yaml sed -i "s|<YOUR-REGISTRY>|$registry|g" $USER_DIR/monitoring/user-values-pushgateway.yaml
Now that we have configured the two user-values files, we are ready to deploy the monitoring part of SAS Viya monitoring for Kubernetes.
cd $HOME git clone https://github.com/sassoftware/viya4-monitoring-kubernetes.git --branch 1.1.2 cd viya4-monitoring monitoring/bin/deploy_monitoring_cluster.sh monitoring/bin/deploy_monitoring_viya.sh
Once these commands have completed successfully, the deployment will pull all necessary images from the container registry that you provided instead of the default public container registries. You can now access Grafana via the following URL
I've used path based routing in my example to make all monitoring applications available. If you would like to use host based routing, please check out the following example available here
Not all sites have the ability to access the internet by default. It's a good thing that the Helm chart used to deploy SAS Viya Monitoring for Kubernetes, offers you the flexibility to override the default values used in the chart by providing your own container registry. It takes a bit of work to get all of the images in your own registry, but once you have all required images in there, you are good to go!
Thank you for reading. I hope the information provided in this article has been helpful. Please leave a comment below to ask questions or share your own experiences.
Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.
If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website.
Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning and boost your career prospects.