Viya Deployment in Upstream Open-Source Kubernetes : using local-storage provisioner
My colleague, Edoardo Riva explained last year, in an excellent blog post, how to use a local storage provisioner in Openshift. A similar technique can be used in an Upstream Open-Source Kubernetes cluster, to expose local disks attached to the Kubernetes nodes and leverage the available storage for SAS Viya volumes.
This is especially useful for two specific SAS Viya components that are installed as part of the Viya deployment and require block storage: Redis and OpenSearch.
As this post became longer than initially intended, I’m adding a little "ToC" below to help you navigate through its sections 😊
A gentle reminder on the Viya dynamically provisioned volumes
Why do I need local block storage ?
Local storage requirements in Upstream Open-Source Kubernetes
Could you show me how it works ?
Mimic directly attached disks in RACE
Configure Redis and OpenSearch
Can I switch from NFS-based to local-storage in an existing environment ?
A gentle reminder on the Viya dynamically provisioned volumes
The SAS Viya platform requires various types of persistent data to function. During the deployment of SAS Viya, a lot of distinct volumes are automatically and dynamically provisioned and made available to Kubernetes as "Persistent Volumes" for the various Viya components (Consul, RabbitMQ, CAS, backups, etc…)
As an example, the table below lists the typical Persistent volume claims with the capacity claim, and required access modes by category.
When you deploy SAS Viya, if you haven’t set specific instructions (as noted there) in your kustomization.yaml file to point to a specific storage class for some of your volumes, then the "default" storage class is used for ALL these persistent volumes.
This "default" storage class, that Kubernetes refers to, is configurable. It likely could be pointing to a local block storage implementation (usually the case in Public Clouds) or specifying an NFS-based storage (for example provided by the nfs subdir external provisioner).
Here is an example of how it looks like, if you have been using the IaC and DaC tools to deploy SAS Viya in an Upstream Open-source Kubernetes cluster :
In this case, the "default" storage class is provisioned by the nfs-subdir-external-provisioner-default which corresponds to exported directories from an NFS server.
While it may work (NFS can be used for both RWX and RWO access type volumes), it is usually not a good idea to use a single "default" storage class for all your volumes.
Why do I need local block storage ?
While NFS-based storage is acceptable for several of the volumes that are automatically created as part of the SAS Viya deployment, it is not adequate for Redis and OpenSearch components.
It is respectively noted in the SAS documentation here:
and there:
as well as in the official Redis configuration recommendations
and OpenSearch documentation:
So, as written everywhere, using NFS-based storage for Redis and OpenSearch in production system is not recommended due to potential locking and performance issues.
In addition, at least two recent "file access" issues have been observed in the field with this message in the Redis pod's logs : "Fatal: can't update cluster config file." The problem was encountered during a Viya version update in AKS with Redis using Azure Files and was solved by setting up an additional (block) storage class for Redis.
While not reported in our Technical support systems, the same error was also encountered during the update of the Viya version in an Upstream Open-Source Kubernetes platform at a customer in Germany during a deployment.
These "file locking" issues confirm the need to avoid NFS-based storage for Redis.
OK so, now that you are convinced, let's see how to get local block storage for Redis and OpenSearch in Upstream Open-Source Kubernetes !
Local storage requirements in Upstream Open-Source Kubernetes
While not explicitly mentioned in the "Platform-specific Requirements" page, since the Redis and OpenSearch volumes require block storage, it naturally becomes a requirement to provide directly-attached block storage for any Kubernetes cluster.
- When SAS Viya is deployed in Public Clouds it is much easier to fulfill this requirement, the default storage class is typically adequate for Redis and OpenSearch (Azure Disks, GP2, GCE PD, etc…).
- But, when SAS Viya is deployed on-premise, it can be more complicated :
- Each Kubernetes node, where the Redis or OpenSearch pods could be scheduled, should have directly-attached volumes that could be used for the objects that these components need to persist.
- It is something that usually requires planning and adds up in the infrastructure costs (in the on-premise infrastructures, local disks or Enterprise grade SAN are typically configured for block storage).
As an example, I was recently contacted by colleagues (from SAS Germany) working on a customer deployment of SAS Viya in Open-Source Kubernetes where this requirement had not been identified in advance. It now results in the urgent need to provision and attach disks to the Kubernetes nodes to meet our official requirements ☹
The IaC for Kubernetes requirements page has a storage section to spell out what is needed and how the local-storage storage class can be used for specific Viya volumes:
If this requirement is fulfilled (disks with empty partitions attached to the Kubernetes nodes), the IaC tool will find the disks, format and mount them on the nodes, so the local storage provisioner can make them available through a storage class.
However, remember that the customers clusters are not necessarily created with the IaC Tool…So these attached disks on the Kubernetes nodes, and the way to make them available as block storage to Kubernetes, is something that should always been discussed during your Architecture design phase if you are deploying SAS Viya on-premise.
Could you show me how it works ?
Yes, I can show you how it works and if you have a subscription to the SAS Viya Deployment workshop in learn.sas.com you can even practice 😊
However, in order to build this hands-on where we can use the local storage provisioner and a local-storage storage class in our lab environment (based on VMware hosts) I had to use a special trick…
Mimic directly attached disks in RACE
In a real customer situation you would either fulfill the IaC requirement (disks with empty partitions attached to the Kubernetes nodes) and let the IaC tool format and mount the disks or mount your own Filesystems (from local disks or SAN storage) on the hosts.
Since we don't have the options to get such disks layout on our RACE machines, we need to simulate "fake" mounted disks.
Based on an example provided in the sig-storage-local-static-provisioner project we create several volumes directories (under /mnt/sas/volumes and mount tmpfs filesystems for each of them.
mkdir -p /mnt/sas/volumes
for vol in vol1 vol2 vol3; do
mkdir -p /mnt/sas/volumes/\$vol
sudo mount -t tmpfs -o size=15G,nr_inodes=15k \$vol /mnt/sas/volumes/\$vol
done
The tmpfs filesystem (extension of ramfs) is a File System that keeps all its files in virtual memory and is temporary (in the sense that no files are created on the hard drive).
The script above is for one node, but you can easily place it in a file, then use ansible to distribute and run it across the kubernetes nodes that require local storage.
Important : Keep in mind that what we do here in these lab environments to simulate real local disks is not recommended in a real production environment.
See the local volumes
What is important to understand is that : the local storage provisioner installed by the IaC tool (or that you can install by yourself if the cluster was installed by the customer) is using Kubernetes DaemonSets, running on the nodes, which are constantly looking for mount points that could be used as local volumes.
So as soon as the new Filesystems appear on the Kubernetes nodes, they are automatically detected by the local storage provisioner pods running on the nodes, and if you check your persistent volumes, you should now see something like the following :
We have 9 available volumes that correspond to our vol1, vol2 and vol3 mount points on 3 nodes.
Note that the local-storage storage class is always installed by the IaC tool for open-source Kubernetes with the following specifications:
A key part of the equation here is the WaitForFirstConsumer "VolumeBindingMode".
It means that the volumes won't be allocated until the first pods request them and bind them through a PersistentVolumeClaim. The volumes, locally available on a given node, are taken by the first pods scheduled and starting on that particular node. It has an impact in the planning/provisioning of the volumes depending on the expected pods placement.
Configure Redis and OpenSearch
Now, the last thing to do is to make Redis and OpenSearch point to our local storage (instead of using the default storage class) and validate that the Redis and OpenSearch volumes are bound to our local PVs !
Here are the Kustomize Patch Transformers that I have been using to force OpenSearch to use the local-storage Storage Class:
tee ~/project/deploy/gelenv-localstorage/site-config/localvolumes/opendistro-storage.yaml > /dev/null << EOF
---
apiVersion: builtin
kind: PatchTransformer
metadata:
name: sas-opendistro-storage-class-transformer
patch: |-
- op: replace
path: /spec/defaultStorageClassName
value: local-storage
target:
kind: OpenDistroCluster
---
apiVersion: builtin
kind: PatchTransformer
metadata:
name: sas-opendistro-volumesize-transformer
patch: |-
- op: replace
path: /spec/nodes
value:
- name: default
replicas: 1
roles:
- master
- data
volume:
resources:
requests:
storage: 12Gi
heapsize: 2G
target:
kind: OpenDistroCluster
EOF
Note that in the second part of the PatchTransformer we are also changing the requested storage size for OpenSearch from the default value (128GB) to 12GB. If you use real local storage, you should not change the default value, but if you are just testing with memory-based filesystem (as I do in the Hands-on), then it might be useful.
For Redis, we simply need to change the storage class name :
tee ~/project/deploy/gelenv-localstorage/site-config/localvolumes/redis-storage.yaml > /dev/null << EOF
---
apiVersion: builtin
kind: PatchTransformer
metadata:
name: redis-modify-storage
patch: |-
- op: add
path: /spec/storage
value:
size: 1Gi
class: local-storage
target:
kind: DistributedRedisCluster
name: sas-redis-server
version: v1alpha1
group: redis.kun
EOF
But by default, there are 6 Redis instances in a Viya deployment, so you need to make sure that you have enough available local volumes on the nodes where the Redis pods will be scheduled. Finding a local PV now become an additional constraint for the Kubernetes scheduler…if it can't find a suitable node with an available local volume, then the Redis pod cannot start.
At the end, after having referenced the PatchTransformer files from you main kustomization.yaml file, built your manifest and applied it (assuming you are deploying with the manual method), the OpenSearch and Redis pods should be ready for service and it should look like this in your Persistent Volumes view :
Note that the patch Transformer examples to change the OpenSearch and Redis default storage class are documented and available in the README files.
Can I switch from NFS-based to local-storage in an existing environment ?
Now, you might have already deployed your environment using unsupported NFS-based volumes for OpenSearch and Redis, and now you (or your customer) realize that we are not following the official SAS requirements. So, you might wonder : "can I change it without breaking my environment ?"
The answer is yes for Redis and OpenSearch (unlike other Viya components) because their internal databases can be considered as ephemeral storage since they are rebuilt every time these services start.
However, it requires an outage. You have to stop the SAS Viya platform deployment (using the sas-stop-all cronjob), remove the existing OpenSearch and Redis volumes, remove the OpenDistroCluster and DistributedRedisCluster instances. Then you can apply your Kustomize changes, and then restart all the SAS Viya platform deployment (using the sas-start-all cronjob).
If you want to experiment, here is the link to the SAS Deployment workshop where we do all these tasks.
Conclusion
I hope you have found this post useful. The main takeaway here, is that it should not be forgotten (during the Architecture design phases) that local block storage is a requirement for some Viya platform components (so it should be made available in your Kubernetes cluster).
Each Kubernetes platform (in the Cloud or On-prem) offer various storage options but they should be reviewed/tested in order to establish the Viya storage specifications : for all the volumes created during the Viya deployment but also for the data that need to be accessed from SAS Viya during the Analytics lifecycle.
While we have shown how to setup and use the local storage for Redis and Opendistro in this post, note that it could also be used for other things like the CAS Disk cache (instead of the hostPath volume type).
Find more articles from SAS Global Enablement and Learning here.
