Kubernetes Storage Patterns for SASWORK – part 3

1 Like

We're back for part 3 of this series describing storage patterns in Kubernetes for SASWORK.

This is where we step through the nitty-gritty details so that you can configure SASWORK to use the desired volume type and storage provider at your site.

--- See this series: [ Part 1 | Part 2 | Part 3 ]

Prerequisites

The general rule is that the site's IT team is responsible to provide the infrastructure to host the SAS Viya platform. Besides virtual machines, networking, and Kubernetes, this does include the site's choices for storage. As a reminder, when it comes to storage for the operational aspects of the SAS Viya platform, we expect the site to provide persistent RWO volumes (used by one node at-a-time) as well as persistent RWX volumes (shared storage used by multiple nodes concurrently).

And while SASWORK can be configured to use the standard RWO volumes (or RWX for Checkpoint-Restart), if it's possible for the site to provide local disk, then that's recommmended for cost, simplicity, and stable performance.

We expect that most of the operational volumes for SAS Viya are likely to be dynamically provisioned as they're needed when the pods come online. This means that your site will already have their desired storage provider(s) setup and they should be able to provide a corresponding storage class definition for each.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

If you're keeping count, we're tracking on three possible storage classes here. Placeholders are used for names since those are arbitrarily decided by the Kubernetes administrator:

RWO-STORAGE-CLASS

- Typically block storage from cloud provider

RWX-STORAGE-CLASS

- Typically shared file storage from cloud provider

- Or NFS, or clustered file system, or custom storage

LOCAL-STORAGE-CLASS

- Typically local disk on instance

- Needs mount and format (and RAID, optionally)

- If not available, use RWO storage

This is ideally where we'd like to begin our work for this effort. Picking up the example storage classes provided by the site and referring to them moving forward.

The plan to configure

The SAS runtime pods are instantiated as sas-batch-server, sas-compute-server, and sas-connect-server, depending on use-case. Each of those pods are described by a PodTemplate that explains to Kubernetes everything it needs to stand up a pod for it to run. This includes naming the container image to pull, request and limit values for CPU and RAM, networking details, and for this discussion, the mount point locations for the various volumes the pod needs.

The SAS runtime container specifies an internal mountpoint at the path "/viya". That's the top-level directory in the containerwhere SAS will place its collection of SASWORK subdirectories and files.

Out-of-the-box, SAS Viya is configured to rely on emptyDir volumes for SASWORK. This means that the container's "/viya" mountpoint will map onto the node's root volume at the typical location where Kubernetes places emptyDir volumes.

We need to modify the PodTemplate to delete that emptyDir "/viya" volume and then redefine the "/viya" volume to use the storage we prefer.

SAS provides documented guidance on getting this done. Refer to the README files for an explanation and then dig into the deployment assets for example manifest at "$deploy/examples/sas-programming-environment/storage/change-viya-volume-storage-class.yaml".

Our approach assumes you're familiar with using kustomize to configure the SAS Viya deployment. Specifically, place desired YAML manifest files in your site-config directory and then link to them in the "transformers:" section of the kustomization.yaml.

The modifications following are shown as separate YAML documents (preceded with "---"). These can be combined into a single manifest file as makes sense.

Delete the "/viya" volume from SPRE pod templates

The first edit we'll make to the SPRE pod templates is to delete the original "/viya" volume that relies on emptyDir.

---
# Delete existing viya volume from SPRE templates
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: delete-viya-volume
patch: |-
  apiVersion: v1
  kind: PodTemplate
  metadata:
    name: change-viya-volume-storage-class
  template:
    spec:
      volumes:
        - $patch: delete
          name: viya
target:
  kind: PodTemplate
  annotationSelector: "sas.com/kustomize-base=sas-programming-environment"

A few things to note here:

This is a PatchTransformer, specifically a stategic merge patch.
It targets PodTemplates by keying off the annotation that selects "sas.com/kustomize-base=sas-programming-environment". The sas-batch-server, sas-compute-server, and sas-connect-server share this annotation - so this one patch addresses all three PodTemplates.
The "patch" it's making is to delete the volume referred to by the container as "viya".

Add the new "/viya" volume to use Generic Ephemeral Volumes

The next step is to define a new "/viya" volume and tell it to use a Generic Ephemeral Volume instead.

---
# Add new viya volume to sas-compute-job-config
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: add-compute-viya-volume
patch: |-
  - op: add
    path: /template/spec/volumes/-
    value:
      name: viya
      ephemeral:
        volumeClaimTemplate:
          spec:
            accessModes: [ "ReadWriteOnce" ]
            storageClassName: LOCAL-STORAGE-CLASS
            resources:
              requests:
                storage: 64Gi # Adjust size based on your SASWORK requirements
target:
    kind: PodTemplate
    annotationSelector: "sas.com/kustomize-base=sas-programming-environment"

Note:

This is another PatchTransformer
The volume spec is a little more complicated than emptyDir - specifying an "ephemeral" volume claim template now
The ephemeral volume claim specifies a storage class name - shown here referring to the one that can use local disk on the node. And it specifies the size of the volume, 64 gibibytes.

Caveats:

If your site doesn't provide local disk for SASWORK, then you can use an RWO-STORAGE-CLASS instead. You could also stick with emptyDir or implement hostPath, of course.
Some local disk storage provisioners - like Kubernetes Local Persistence Volume Static Provisioner project and Rancher's local-path provisioner project - don't enforce the volume size limit. This can actually be a good thing for SASWORK. Whichever provisioner/CSI driver your site provides, you should follow up to find out if the volume size is enforced and plan accordingly.

Sidebar: Why is it good if the local volume provisioner doesn't enforce the volume limit?

Recall that it is likely multiple instances of the SAS runtime could be running on a single Kubernetes node. They might all specify the same top-level directory path for their SASWORK. So, the SAS runtime is careful to uniquely name each SASWORK location to prevent collisions. That's cool.

The other thing though, is that sometimes SASWORK is heavily used and other times not so much. If the local storage provisioner enforces the volume size, then that effectively creates a hard limit on the number of concurrent SAS runtime pods per node. That's fine, if that's what you want. But if your SAS runtime workload is varied in how heavily it uses SASWORK, then not enforcing the volume size means you can rely on other controls to manage the number of SAS runtime pods per node - just make sure that there's sufficient total disk space where SASWORK will be placed.

Be more specific

With two relatively simple patches, we were able to change the "/viya" volume definition for three different SPRE pod templates. But what if you want to specify something different for sas-batch-server and for sas-compute-server and for sas-connect-server?

Instead of this:

target:
    kind: PodTemplate
    annotationSelector: "sas.com/kustomize-base=sas-programming-environment"

You could define a specific "/viya" volume patch for each SPRE pod template, then target them specifically by using one of the following:

target:
  kind: PodTemplate
  name: sas-compute-job-config
 
target:
  kind: PodTemplate
  name: sas-batch-pod-template
 
target:
  kind: PodTemplate
  name: sas-connect-pod-template

Assuming the plan is to change all three SPRE pod templates to use different storage classes, then your site-config file might contain four YAML documents:

patch to delete "/viya" emptyDir
targets "sas.com/kustomize-base=sas-programming-environment"

patch to add "/viya" Generic Ephemeral Volume "alfa"
targets "sas-compute-job-config"

patch to add "/viya" Generic Ephemeral Volume "bravo"
targets "sas-batch-pod-template"

patch to add "/viya" Generic Ephemeral Volume "charlie"
targets "sas-connect-pod-template"

What about Checkpoint-Restart?

Checkpoint-Restart functionality is only available for the sas-batch-server. And you might recall that it requires an appropriate RWX volume because an interrupted batch job might later resume on a different Kubernetes node and it will need to recover those SASWORK files.

This is fundamentally a different kind of storage than what we typically see for SASWORK on RWO volumes. We don't want each instance of sas-batch-server to get its own dedicated physical volume to back SASWORK. Instead, we want them all to share one volume that can be mounted and re-used repeatedly.

First of all, we will create a static PVC:

---
# Shared RWX PVC for SASWORK to enable checkpoint-restart functionality
# The RWX access mode allows multiple compute pods to mount simultaneously
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: SASWORK-RWX-PVC
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: RWX-STORAGE-CLASS
  resources:
    requests:
      storage: 64Gi # Adjust size based on your SASWORK requirements

This simply defines a PersistentVolumeClaim. You can apply it directly with kubectl if you like. Or, if you want something that runs automatically as part of your kustomization process, then save it as its own file in your site-config directory and link to it in the "resources:" section of the kustomization.yaml file.

Note:

This is not a PatchTransformer. It’s a resource definition. Do not include it in the YAML manifest for the PatchTransformers of the "/viya" volume we're discussing.
Be sure to provide appropriate RWX shared storage for this volume. Plain old NFS is not supported for this use-case (even if it does work in small-scale implementations).
This PVC is defined once and will remain until its specifically deleted. It calls a storage class, and so the very first time it's referenced, then Kubernetes will be able to dynamically provision a new volume. After that, subsequent pods that refer to this PVC will get the exact same volume.

Now we just need to configure the sas-batch-server to use this new static PVC:

---
# Add new viya volume to SPRE templates
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: add-spre-viya-volumes
patch: |-
  - op: add
    path: /template/spec/volumes/-
    value:
      name: viya
      persistentVolumeClaim:
        claimName: SASWORK-RWX-PVC
target:
  kind: PodTemplate
  name: sas-batch-pod-template

Note:

This PatchTransformer targets only the sas-batch-server's PodTemplate.
It specifies that the "/viya" volume should refer to the PVC named "SASWORK-RWX-PVC" (or whatever you call it at your site).
Be sure to address other SPRE PodTemplates as needed.

Bonus round

Have you heard of the SAS Cloud Analytics Service (or CAS)? I figured you had. As the flagship in-memory analytics engine in the SAS Viya platform, it has gained some notariety.

In that case, you might be aware that it has a special scratch space it relies on as a backing store - it's called CAS_DISK_CACHE. It benefits from using local disk (or RWO volumes) much like SASWORK.

Well, the SPRE has been extended with new functionality informally known as the Enhanced Compute Engine (or ECE). Simply put, the ECE enables running CAS actions inside the SPRE. This brings the performance of in-memory analytics where it can apply to smaller scale tasks most efficiently. But it also needs its equivalent of CAS_DISK_CACHE, what I refer to as ECE Cache.

For the ECE Cache, the SPRE PodTemplate defines the "/tmp" volume. Like "/viya", it defaults to an emptyDir, but it can also be changed to utilize a Generic Ephemeral Volume. Well, you know exactly how to do that now following the examples above. Note that you'll never need to specify an RWX volume for "/tmp" and so I expect that the first examples where we simply target "sas.com/kustomize-base=sas-programming-environment" to modify all SPRE PodTemplates should be sufficient.

References and Resources

In addition to the links above, the GEL team also provides courses on learn.sas.com:

SAS Deployment Learning Subscription: Installing and Configuring the SAS Viya platform for Kubernetes and different cloud platforms.
SAS Architecture and Security Learning Subscription: Considerations for planning the technical architecture and security for the SAS Viya platform.

Find more articles from SAS Global Enablement and Learning here.