Announcing Project Mountpoint – Guidance to configure storage for SAS Viya
The SAS Viya platform doesn't just need "storage" — it needs the right storage for the job, whether that’s fast, shared, or temporary. We also know site IT teams have strong opinions (and security concerns) about their environments, often rightfully avoiding risky setups like hostPath. SAS Viya is designed for storage flexibility through the use of standard Kubernetes conventions. That's a pretty broad range to consider though - so, now we have new tools to describe storage for Viya's use that are simple to implement and easy to extend when needed.
Project Mountpoint aims to provide the tools and guidance to streamline configuration of the SAS Viya platform for implementing your choice of storage providers in the Kubernetes environment.
Where Project Mountpoint fits in your environment
Project Mountpoint mostly provides content aimed at the "configuration" side of a SAS Viya deployment. With the right configuration collateral, then your site IT team can implement the storage solutions they prefer and that align with Viya's requirements.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
There are a lot of different needs for storage. Project Mountpoint is new today, but already growing to cover more scenarios. We already offer several powerful tools and practices to help.
Three Starter Storage Classes
Project Mountpoint introduces the concept of Three Starter Storage Classes (3SSC). These are implemented as drop-in, no-edit configuration to function independently of the site, of the cloud service, and of the backend physical storage. They're abstracted from those constraints in their definition on purpose. This makes them portable to be used anywhere. Yes, anywhere.
What are these Three Starter Storage Classes?
- viya-standard-sc: Use for RWO persistent storage
- viya-shared-sc: Use for RWX shared storage
- viya-scratch-sc: Use for temporary storage (instead of emptyDir)
The project provides a YAML file containing multiple YAML documents that configure the various SAS Viya services to refer to the 3SSC. While functional as it is, the YAML file should be considered as an example that can be modified for deployments with additional Viya services needing storage that might not be in the current definition.
SAS Viya will be configured to use the Storage Classes named above to create Kubernetes Persistent Volumes Claims (PVC) to get dynamically-created Persistent Volumes (PV) for both RWO Persistent and RWX Shared types of storage for the operational needs of the software services. For scratch space (i.e., temporary), Generic Ephemeral Volumes are defined for the powerful analytic runtimes that rely on scratch space to be fast and performant.
Multiple storage providers for SASWORK
Speaking of scratch space, the storage backing SASWORK is a critical architectural consideration. If implemented poorly, then many aspects of user interactions with analytic processes can feel undesirably slow. For sites that are coping with that kind of situation, we offer a process where they keep their current SASWORK storage, but also define additional backend storage providers for side-by-side(-by-side) comparison of job runs. This only requires configuration changes in a live environment with zero interruption to the current processes, workflows, or system availability.
The individual steps to do this aren't especially challenging, but can be tedious:
Project Mountpoint provides a tool to make this easy. With a single command, you can define the entire chain above. Once that's set, then your end users simply specify the "SPRE" context they desire in SAS Studio, for a SAS Batch job, etc. which corresponds to the new backend storage for SASWORK to test out.
By having the ability to see how different storage backends affect SASWORK performance at your site, then you can ascertain the winning SASWORK implementation and follow through with disabling the unwanted configurations. This can have significant impact not just on speed and performance, but in some cases might allow the overall system design to transition to an architecture that not only reduces storage costs but also enables elastic features along with improved availability.
RWX storage for checkpoint-restart
The concepts above can build and combine to provide more flexibility and features. One area that might be overlooked at some sites is the proper storage configuration to support Checkpoint-Restart functionality in SAS Batch Servers.
The purpose of Checkpoint-Restart is for your SAS program code to resume where it left off when it was interrupted (either by preemption or host failure). This can save hours of rework for very long-running batch jobs. This also means that we need to preserve the state of files and processing at the point where the job was interrupted. In particular, we focus on SASWORK here. For one thing, it’s the default location of the Checkpoint Library that knows where things were when the job was interrupted. But it might also contain data sets, catalogs, utility files or whatever was in-flight when the job was running in its original execution. We want to get back to those files. So, SASWORK needs to be backed by a persistent RWX shared volume that is concurrently accessible to the nodes labeled for compute workloads.
With Project Mountpoint's targeted implementation to configure multiple storage providers for SASWORK, then we can define a Batch Context that users can specify when running jobs that use the Checkpoint-Restart functionality. Project Mountpoint goes farther than that by also providing detailed validation exercises so you can positively confirm that Viya is using that storage as intended.
See for yourself
If you'd like a chance to kick the tires and see what the big deal is, then go look at the Three Starter Storage Classes first. Configuring Viya's use of storage is just one of the many steps a site must perform when deploying the SAS Viya platform. Project Mountpoint explains where in context of the overall deployment specifically - and for the exact details, then refer to the 3SSC Deployment Examples. Those prove the point that a single drop-in, no-edit configuration file to set Viya up with 3SSC can work with any cloud provider or on-prem environment.
---
SIDEBAR:
Also note there's a bonus 3SSC deployment exercise for Microsoft Azure which steps through setting up Azure Container Storage v2 (ACS) as the backend for the viya-scratch-sc storage class. There's no faster I/O in Azure - nor a cheaper way to run it - than that. And it's super simple to set up, too.
Project Mountpoint is intended to be storage provider agnostic - preferring the site IT team to decide the storage backend for themselves. But, as far as scratch space in Azure-based deployments of the SAS Viya platform, Azure Container Storage has become this author's first recommended approach to try.
---
Take the Project Mountpoint approaches for a spin and let us know what you think. We're certain it will improve your ability to configure SAS Viya for any storage you prefer. And if you have questions or problems, then you can submit an Issue in Github or reach out to me directly.
Find more articles from SAS Global Enablement and Learning here.
