The interdependence between SAS Viya and the Kubernetes platform versions with each of them having different release lifecycles can sometimes be problematic.   To add to the fun, SAS Viya uses the term "update" to indicate moving to the next cadence version whereas Kubernetes uses the term "upgrade" to indicate moving to the next minor release version.   ☹️   If you move to a Kubernetes version without paying attention to the Viya LTS version that you are running, you may find yourself out of SAS support. And conversely if you move to a new Viya version without taking care of the Kubernetes version you could end up being out of Kubernetes support…   This dashboard below should gives you an idea of the release cycle for both platforms…     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   However the fact that, at a given time Kubernetes version "A" and SAS Viya version "B" are both supported individually does not mean that this SAS Viya version "B" is supported for this specific Kubernetes version "A" (remember that each SAS Viya version is only supported for 3 Kubernetes versions).   In this post, we want to remind some important guidelines to ensure that you remain on a "supported update path" and we also explore a very "real" customer use case of "mixed versions deadlock" I have been recently facing and see how we could address it  😊     The "Kubernetes upgrade workflow" for SAS Viya   The good news is that SAS is aware of this strong dependency between the SAS Viya and Kubernetes versions and release lifecycles and, as a result, provides specific, detailed and frequently maintained documentation to help customers deal with it.   I particularly like this table (available in the Virtual Infrastructure Requirements section of the SAS documentation), where you can see the versions dependencies between SAS Viya and Kubernetes at a glance:   You can immediately understand from the table that: whether you decide to update either SAS Viya or upgrade Kubernetes to its next version, you may also have to make a change to its counterpart to remain supported.   As an example, imagine a customer running SAS Viya LTS 2024.09 on the extended support version of AWS EKS (Elastic Kubernetes Service) 1.28 and in June 2025, they decide to update to the latest SAS Viya LTS 2025.03. From the table above, we can see that they also need to move to EKS 1.29 (at least) to remain supported.   But in this case, which platform should I perform first? SAS Viya or EKS ?   The "Kubernetes upgrade workflow" section of the SAS documentation addresses this very question!   It corresponds to scenarios that are regularly tested by SAS and covers 3 different situations (that I have tried to summarize in the table below) which correspond to different assumptions and conditions.   Scenario:   S1-Upgrade Kubernetes without updating the SAS Viya platform.     S2-Update the SAS Viya platform and upgrade Kubernetes in the same time frame.   S3-Update the SAS Viya platform and upgrade Kubernetes in the same time frame. Assumption: The target Kubernetes version is supported for your current SAS Viya version. Your current Kubernetes version IS NOT supported for the newer release of the SAS Viya platform to which you are updating Your current Kubernetes version IS supported by the newer SAS Viya release. Condition: Make sure that your target Kubernetes version is supported by BOTH the currently deployed version of the SAS Viya platform and the newer version to which you plan to update. Make sure that target Kubernetes version is supported by the newer SAS Viya version to which you plan to update Steps:   1. Viya Backup 2. Stop All Viya services 3. Upgrade the Kubernetes cluster to a supported version. 4. Verify dependent components versions (ingress-nginx, etc.) 5. Restart All Viya services   1. Viya Backup 2. Stop All Viya services 3. Upgrade the Kubernetes cluster to a supported version. 4. Verify dependent components versions (ingress-nginx, etc.) 5. Restart all SAS Viya platform services. 6. Update the SAS Viya platform software.     1. Viya Backup 2. Update the SAS Viya platform software 3. Stop All Viya services 4. Upgrade the Kubernetes cluster 5. Verify dependent components versions (ingress-nginx, etc.) 6. Restart all SAS Viya platform services.   The game here is to determine in which scenario you are, based on the customer’s current versions of Kubernetes/SAS Viya and their upgrades/updates plans.   So back to our example : "a customer running SAS Viya LTS 2024.09 on the extended support version of AWS EKS 1.28 and they want to upgrade to the latest SAS Viya LTS 2025.03."   So, if you look at the version and scenario tables above (or simply at the 3 bullet points in the official documentation), in which scenario would they be ?   Take your time to see if you could find the answer and solve this puzzle 😊   It is not always immediate and requires a bit of logical thought.   OK...I’m sure you have found the answer : Yes, they could be in scenario S1 or S2. Because they meet the assumptions and conditions for both scenarios.   So they can either decide to separate the 2 operations : first upgrade to EKS 1.29 or 1.30, then update SAS Viya (S1) OR they can decide to upgrade Kubernetes and update the SAS Viya platform in the same time frame (S2).    The Kubernetes upgrade workflow really helps to follow the upgrade and update steps in the right order, ensuring that the customer remains in the "Standard Support" situation the whole time.   However, there can be some special situations where your customer’s use case does not fit in any of the described scenarios  ☹️…we’ll talk about that in the last section of the post.                     The need to find the common k8s version between Viya LTS N and Viya LTS N+1   There is one key condition to remain on the "supported update path" that is not clearly spelled out in the documentation but is very important to understand :   When you upgrade Kubernetes to a new version, it is imperative that the target Kubernetes version is supported with your current and future Viya version.   In other words, when looking at the version table (above) you must identify a target Kubernetes version that is supported across your current and future Viya versions…the target Kubernetes version must be in common for the Viya N and Viya N+1 versions.   For example, if you were on Viya LTS 2024.03 with Kubernetes 1.27, you could not move to Kubernetes 1.28 then 1.29 without updating SAS Viya. You would need to go first to Kubernetes 1.28, then update the Viya LTS version to LTS 2024.09 and then upgrade Kubernetes to 1.29. That’s the only way to remain supported during these operations.   As we can see in the table, if there was a "2-steps" move from 1.27 to 1.29 without updating SAS Viya at the 1.28 milestone, they we would be running Viya LTS 2024.03 on K8s 1.29 (which is not supported).                         An interesting customer use case   The context   Here is a bit of background information, provided by our customer (a bank in Germany testing SAS Viya in a development environment before moving to production) :   Setup/Use case:   Deployment on OpenShift OpenShift Platform is managed by another Team Update-Strategy of OpenShift-Platform: Only update every second (even) release   Example to illustrate the problem:   OpenShift version 4.14 installed LTS 2024.03 supports OpenShift versions 4.13.x - 4.15.x (SAS Help Center: Platform-Specific Requirements) LTS 2024.09 supports OpenShift versions 4.15.x - 4.16.x (SAS Help Center: Platform-Specific Requirements) OpenShift version is being upgraded to 4.16. What are we doing with SAS Viya? We cannot update to LTS 2024.09 after the upgrade (2024.03 unsupported - with 4.16) and not prior to the upgrade (2024.09 unsupported - with 4.14).   Some observations on the customer requests   OK... so first a few remarks on this information provided by the customer.   First of all, they are running SAS Viya on the Red Hat OpenShift Container Platform (OCP). In OpenShift, the version scheme is a bit different: we talk about versions 4.x while in the standard opensource Kubernetes we use versions 1.x.y. However OCP is built on Red Hat Enterprise Linux (RHEL) and Kubernetes. We know, for example that OCP 4.17 uses Kubernetes 1.30, OCP 4.18 uses Kubernetes 1.31 and so on… Then we can also notice that the customer is well aware of the requirement to remain in a "supported" situation at all times. Their last comment leads us to think that they are already aware of the documented "Kubernetes upgrade workflow". While it is not technically possible to directly upgrade from OCP 4.14 to 4.16 (just like for the standard Kubernetes versions, you cannot skip OCP versions when upgrading their software. It would be a violation of the upgrade policy for kubeadm as explained in the IBM documentation there), the customer explained that the team in charge of the OCP cluster would not allow other operations while they perform the upgrade operations and would only reopen and expose the cluster once moved to the N+2 version (4.16, 4.18, 4.20 etc.). The underlying reason of that policy is that Red Hat OpenShift "Extended Update Support" (that we generally recommend for customers choosing the SAS Viya LTS cadence) is only available for "even" version.   We can clearly see it in the OCP Life Cycle table below :     As noted in the OpenShift documentation, "Red Hat denotes all even numbered minor releases (eg 4.8, 4.10, 4.12) as Extended Update Support (EUS) releases".   This explains why the customer’s OpenShift strategy is to only work with "even" versions.   The bad and good and bad news…   First, to ease our life, let’s build the same supported K8s version tables but with the addition of OCP version :   SAS Viya Platform Version Supported Versions of Kubernetes Supported version of OpenShift 2025.03 (LTS) 1.29, 1.30, 1.31 4.16, 4.17, 4.18 2024.09 (LTS) 1.28, 1.29, 1.30 4.15, 4.16 2024.03 (LTS) 1.26, 1.27, 1.28 4.13, 4.14, 4.15 2023.10 (LTS) 1.25, 1.26, 1.27 4.12, 4.13, 4.14   As explained in the second section of the post, our "Kubernetes Upgrade workflow" assumes that there is, at least, one OCP version in common between the source LTS version and the target LTS version...   As we can see on the table above, between LTS 2024.03 and LTS 2024.09 there is only ONE common version, which is OCP 4.15 - however since the "odd" version of OCP were not exposed to the customer team they could only have either 4.14 or 4.16 - so the "supported update path" was not available for them !     After talking a little bit more with the customer though, we understood that they were able to manage the situation and update to Viya LTS 2024.09 (although they were on an "unsupported update path") in their Test environment but before going into production environment, but they were now asking if they will have this problem again...   So first there is a SAS R&D initiative where the plan for the future is to always provide support for 3 OCP versions in our Viya LTS releases (unlike what happened with 2024.09).   Here is a little projection of what OCP versions are planned for the future:     There is one good news here : The customer should not face this issue again for their next Viya updates to LTS 2025.09 and then to LTS 2026.03 (because each time there is an "even" version in common between their current and target version).     However … we can, unfortunately, also see that:  while there will always be common OCP versions and bridge each Viya LTS release, there will NOT always be the "even" numbered ones.    It's not possible to always have the OCP "even" version in common, given that Viya LTS releases 2x a year and K8s releases 3x a year. The math would not work out for that… ☹️     What are the alternatives?   So the only real solution, in order to always remain on the supported path, would be to ask for an exception to the customer’s standard process in order to be able to perform the SAS Viya update between the move from the odd release (ex: 4.21, 4.23 in the table) before upgrading to the even "EUS" version…   Another smart suggestion from a colleague is to dive a little bit more in the details of the Viya update operation and to break the steps as below:   Perform the "Before Deployment Commands" steps from the "Deployment notes" in the source environment: (ex: LTS 2026.03 in OCP 4.20), then stop SAS Viya, perform the OCP cluster upgrade (ex: from 4.20 to 4.21 then to 4.22), then get the new deployment assets and re-deploy SAS Viya (ex: LTS 2026.09 in OCP 4.22), restart the Viya platform and finally perform the "After Deployment Commands"...   Technically it would avoid being in the situation where we would run a SAS Viya version in a non-supported version of OpenShift...However it is not really one of the listed options in the "Kubernetes Upgrade Workflow"…   The caveats of this plan are the following:   This update scenario has never been tested by SAS. If there is an issue with Viya at the end, then it will be hard to know if it's coming from the OCP upgrade or the Viya update…many things could go wrong (for example with 3 rd party tools versions incompatibilities).   As a summary the problem describe here is caused by the quite unique way the Extended Support is handled in OCP (only "even" versions)...each SAS LTS version has support for 3 Kubernetes version but unlike RedHat, SAS does not limit support to our even versions. But unfortunately, as it sometimes happens, there is no magic silver bullet to solve this specific customer issue… But at least there are options to try.    That’s it for today, hopefully this little post gave a useful illustration of the potential challenges of intricate K8s upgrade/Viya updates operations and how to deal with them.   Find more articles from SAS Global Enablement and Learning here.   ... View more

A few months  ago, various SAS teams and groups reported sporadic or systematic issues related to the startup of the SAS Compute Server pod.   Several tickets were opened to describe this issue and the problem was even reported on the Azure Q&A board. The symptom takes various forms but the underlying cause is always the same : the compute session could not be created, because the sas-compute-server pod failed to start up within the default 60 seconds timeout.   The issue can be seen from different applications (failure to run pipelines in SAS Model studio, ad-hoc analysis on a table within SAS Information Catalog, etc.…)  because, of course, many components of the platform rely on the successful launch and execution of SAS Compute Server sessions.   Most of the time, the issue appears randomly, is not consistent and quite often a restart of the SAS Workload Management makes the problem go away...but not always !   Some brave individuals were able to understand the root cause of this problem (and how to avoid it) and that’s what we’ll describe here (with a bit of detective work) 😊   A little "ToC" should help you to navigate in this rather technical post...   What really happens : the underlying issue (COMPTRIAGE-1784) fsGroup, fsGroupChangePolicy, how are these things working? Does the recursive change of ownership always happen? What changes with a CSI driver?  Am I affected ? What are the factors? A reproductible and observable behavior What Can we do? Conclusion   What really happens : the underlying issue   After looking carefully at the logs and doing some research, a promising  lead explaining the compute server pod timeout was found : "For large volumes, checking and changing ownership and permissions can take a lot of time, slowing Pod startup."   This intuition was confirmed by additional troubleshooting…When looking at the Kubernetes kubelet's logs on the compute node for this environment it appeared that the mounting of a PVC for the sas-compute-server pod was taking longer than would be expected and was the cause of the init-containers not completing.   E0805 18:41:01.568119 3996 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[python-volume], unattached volumes=[], failed to process volumes=[]: context canceled" pod="rtr/sas-compute-server-caa26268-7d7c-43a4-87c4- 6a61f8b78489-5646" podUID="ec636326-1a66-4131-84e7-2584239b75a5"   One of the particularities of the python-volume (that  is mounted into the SAS Compute Server pod when "Integration with External Languages" is configured) is that it contains many folders and files (around 79 000 !). So, it is likely that some operations on the mounted volumes take too much time and cause the SAS Compute Server to time out…     fsGroup, fsGroupChangePolicy, how are these things working ?   At this point, a little reminder on how the mounted volume permissions are set in Kubernetes would not hurt 😊   As nicely explained in the Kubernetes official documentation :   “By default, Kubernetes recursively changes ownership and permissions for the contents of each volume to match the fsGroup specified in a Pod's securityContext when that volume is mounted.” - The goal is to set the volumes folder permissions to allow whoever is running the container’s process to access the underlying storage. However, the page also describes that changing ownership and permissions in the volume can take time (and significantly slow down the startup of the pod...), and explains that it is possible to use another field, called fsGroupChangePolicy , to control the way that Kubernetes checks and manages ownership and permissions for a volume. In summary if the value for  fsGroupChangePolicy is set to OnRootMismatch , then the recursive change of permissions only occurs if the permissions and the ownership of the root directory does not match with expected permissions of the volume.   Let’s look at an example of the SecurityContext definition for our SAS Compute Server to illustrate that.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   We can see that the fsGroup value is automatically set to the GID (Group ID) value returned by the identities services for the user who starts the compute server (note that in this example, the GID is a randomly generated hash value but in other cases it could be the POSIX "group id" attribute of the end user, fetched from the identity provider).   In addition the fsGroupChangePolicy value is set to OnRootMismatch (which means that if a volume root folder is already owned by the group with this GID 515741841, then the recursive change of permissions is skipped).     Does the recursive change of ownership always happen ?   Actually, the fsGroup specification in the Pod's SecurityContext does NOT always cause the recursive change of permissions, depending on the type of volume and File System.   As noted there, “For certain multi-writer volume types, such as NFS or Gluster, the cluster doesn’t perform recursive permission changes even if the pod has a fsGroup. Other volume types may not even support chown()/chmod(), which rely on Unix-style permission control primitives.”      From what we’ve seen in our environment, when using static provisioning and standard "nfs" volume type, the fsGroup is NOT enforced. It is also the case when you use the nfs-subdir-external-provisioner automatic provisioner to create your "NFS based" storage class.   However, things are different when the NFS folder is exposed and mounted through a Storage Class provided with a CSI driver….     What changes with a CSI driver ?   When the volume is mounted with a CSI driver, then the volume’s "permission and ownership change" behavior can be affected and change from what happens without a CSI driver.   When mounting a volume with the CSI driver, the CSI driver configuration itself can drive the way the fsGroup is implemented. The fsGroup effect is delegated to the CSI driver.   The fsGroupPolicy is the CSI driver’s field that determine the behavior.   When looking at the official Kubernetes CSI doc we can see that there are 3 possible values:     So with the File mode, the fsGroup effect is enforced, while with the None mode, the volumes are mounted with no ownership or modification changes. The last available mode,  ReadWriteOnceWithFSType only modifies the ownership and permission under 2 conditions : the fsType is defined and the PV’s access mode is ReadWriteOnce.     Am I affected ? What are the factors ?   What appears to be the combined configuration causing the computer server timeout issue is the sas-pyconfig Python volume (required when SAS Viya integration with external languages is configured) coupled with the installation of the new NFS CSI driver (which is now recommended by SAS, as discussed in a previous post…).   By default, the new NFS CSI driver (nfs.csi.k8s.io) is configured with a value of File for the fsGroupPolicy, which means that the recursive change of volume's files and folders ownership is enforced when fsGroup is defined in the pod’s Security context (which is the case for our SAS Compute Server podTemplate definition - as seen above).     However, note that, while having python integration configured is a common cause of the SAS Compute Server timeout (due to the number of files/folders), some teams also saw it failing due to mounts with some data folders. Integration with python is one example; but it could happen with other mounts as well (especially if there are a lot of folders/files to parse – which could cause permissions changes for all the volumes attached to the pod take longer than the pod timeout limit…).     A reproductible and observable behavior   We were able to reproduce the issue in our lab environment.   After having implemented both configurations ("Integration with External languages" and new NFS CSI driver for Kubernetes) and redeployed, a first attempt to start a SAS Studio session gives this error.     However, on the second attempt (with the same user), the startup of the SAS Compute Server is generally successful. It allows us to execute and navigate inside the sas-programming container and to confirm that the ownership of the sas-pyconfig mounted volumes has been changed.     It is, very likely the operation that took too long the first time (almost 80,000 files permissions to change !) and caused the time out of the Compute Server !   If we, now connect to the NFS server and look at the physical folders permissions, we can see that the owning group of the python-volume (and other writable volumes) actually depends on who is the last to have started a Compute server session…     Interestingly, we can also see this message in the kubelet logs (that confirms the root cause of the issue discussed in this blog post).   Sep 22 13:22:58 sasnode08 kubelet[3431]: W0922 13:22:58.874504 3431 volume_linux.go:49] Setting volume ownership for /var/lib/kubelet/pods/cc703b02-bba2- 4f5c-b946-913dcc60b2e9/volumes/kubernetes.io~csi/pvc-284a67a5-7fb1-42cd-b77d- c1f8fce88e4f/mount and fsGroup set. If the volume has a lot of files then setting volume ownership could be slow, see https://github.com/kubernetes/kubernetes/issues/69699   Finally, on an another "timeout" occasion, using this command we could see that, while the pod is trying to perform the next volume mount operation, a message appears and reports that the “pod startup duration” is too long 😊       At this point it looks like the detective work is over and that we have caught the main suspects 😊     What can we do ?   To prevent the problem from happening, the solution that was found so far was to change the behavior of the CSI driver by changing the fsGroupPolicy value.   We can manually update the value with the kubectl "edit" or "patch" command to change the fsGroupPolicy value from Files to either ReadWriteOnceWithFSType or None, as shown below:   kubectl patch csidriver nfs.csi.k8s.io -p '{"spec":{"fsGroupPolicy": "None"}}'   It is also possible when installing the CSI driver to disable the fsGroupPolicy.   For example in helm you can use the --set feature.enableFSGroupPolicy=false option. However note that the CSI driver fsGroupPolicy value changes from File to ReadWriteOnceWithFSType in this case.     Conclusion   If you are using a CSI driver (such as the newly recommended opensource NFS CSI driver) and have also configured the SAS Viya Platform integration with external languages (with the python volume – which contains a lot of folders and file), then you may noticed some random failures of your SAS Compute Server sessions.   In this case, it is likely that you are affected by the fsGroupPolicy defined for the CSI driver.   The problem was not observed with the older open-source NFS provisioner tool because it was using the Kubernetes native in-tree NFS support (meaning it just creates PV's with .spec.nfs populated), whereas the CSI driver does the mount itself.    You could avoid the random Compute Server session failure by setting the CSI driver’s fsGroupPolicy to None or ReadWriteOnceWithFSType.   Note that this change was implemented in the DaC (Deployment as Code) GitHub project in the release that was published at the end of September 2025, the fsGroupPolicy is now set toReadWriteOnceWithFSType by default. If you are not using the DaC project to install the NFS CSI driver but have installed it manually on your own (to comply with the latest recommendation from the SAS Documentation) you may also want to consider making this change in the CSI driver configuration.   Finally note that with the Viya November 2025 stable version (2025.11), more freedom is given to the SAS Administrators in order to get rid of any CSI driver constraint that would not allow to disable the fsGroup settings. A new configuration option, "fsgroup.enabled" allows to make the PodTemplate's fsGroup and fsGroupPolicy settings optional in the SAS Launcher configuration, so they could be completely disabled when the underlying storage system already enforces access control, or when the volumes are mounted with sufficiently open permissions  (e.g., 0777 or per-user subpaths).   I hope you enjoyed this post and learned a few things about Kubernetes and SAS Viya (I know I did !  😊)   Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Viya needs shared storage and the most common way to provide shared storage for a deployment of SAS Viya in Kubernetes is to rely on an NFS Server.   From the beginnings of SAS Viya, the recommended way to expose the NFS Server to SAS Viya in Kubernetes was to use the "NFS subdir external provisioner" (which corresponds to this open-source project).   From SAS Viya 2025.06, though, the "Hardware and Resource Requirements" section of the SAS documentation has been updated and now references the "NFS CSI Driver for Kubernetes" (CSI plugin name: nfs.csi.k8s.io) instead of the "NFS subdir external provisioner".   In parallel, the viya4-deployment project (aka "DaC" for "Deployment as Code") recently updated the NFS provisioner from the nfs-subdir-external-provisioner to the csi-driver-nfs provisioner. The change in the DaC project and was released with the v9.0.0 version.   In this post, we'll explore the "why?" and "how?" questions related to this change.     Why ?   While it has been very helpful for years, the nfs-subdir-external-provisioner project is now outdated, unmaintained and has known vulnerabilities.   A visit on the GitHub repository confirms that no new release for this open-source software has been produced since March 2023 :   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Here is a screenshot from a security scan of the nfs-subdir-external-provisioner image:     We can see that several vulnerabilities have been identified for this image…   However a new open-source tool that could provide the same functionality was identified by SAS and it is called the NFS CSI Driver.   Here is a screenshot of the associated GitHub repository page :     Looking at the release dates, we can see that the project is way more active, with new releases published approximatively every 3 months.     How ?   Now the question is how to install this new NFS CSI Driver tool ?     Install the csi-driver-nfs with the DaC   Historically, the DaC (viya4-deployment project) code offers a "baseline" option to deploy the SAS Viya deployment pre-requisites (which include the NFS provisioner).   At the time of this write up, the DaC tool already has the code to install the new NFS CSI driver (which is located in the  roles/baseline/tasks/nfs-csi-provisioner.yaml YAML file).   It could be a breaking change in the environment for a customer having deployed with an older DaC version, who wants to update to the version that install the new NFS provisioner. So before the csi-driver-nfs code was  pushed in release v9.0.0, additional guidance for the customers already running a "DaC-deployed" environment has been added.    If you update from an earlier DaC version than v9.0.0 to a new one, make sure you read this page before running the "baseline" playbook again.     Install the csi-driver-nfs manually   For those who don’t use the DaC (viya4-deployment) tool to deploy SAS Viya and/or the pre-requisites (baseline), it is possible to run the Helm commands directly to install the csi-driver-nfs application.   Here is an example :   helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts helm repo update helm install csi-driver-nfs-sas  \ --set driver.mountPermissions="0777" \ --set storageClass.create=true \ --set storageClass.name="sas-csi-nfs" \ --set storageClass.reclaimPolicy=Delete \ --set storageClass.volumeBindingMode=Immediate \ --namespace kube-system \ --version  4.11.0 \ --set storageClass.parameters.server=${NFSIP} \ --set storageClass.parameters.share=/export \ --set storageClass.parameters.subdir=\${pvc.metadata.namespace}/\${pvc.metadata.name}/\${pv.metadata.name} \ --set storageClass.mountOptions="{noatime,nodiratime,rsize=262144,wsize=262144,vers=4.1}" \ csi-driver-nfs/csi-driver-nfs   If the Helm installation is successful, you should see this, when you list the storage classes:     Notice the storageClass.parameters.subdir option that lets you specify how you want to organize the structure and names of the folders for the data stored in the Persistent Volumes.    It means that if we, now, create a new Persistent Volume Claim using the sas-csi-nfs storage class, a folder with the "Persistence Volume name" is created under a folder with the with the "Persistence Volume Claim name", itself under a folder with PVC namespace, under the /export root folder on the NFS server.   For example it would look like that: However for a customer already running a Viya environment with the “old” NFS subdir-external provisioner, what would be the impact of switching from the "old" version to the new csi-driver-nfs ?     Switching from "nfs-subdir-external-provisioner" to "csi-driver-nfs"   So what are the steps to switch from the nfs-subdir-external-provisioner to the NFS CSI Driver  and what is the impact on the volumes for an existing environment ?   We have explored this scenario in the "Appendix" section below.   Basically :   We stopped the environment, Deleted the existing Storage Class associated to the old NFS provisioner, Installed the new NFS CSI Driver and attach it to the Storage Class name (used by SAS Viya) Restarted the environment.   and it turns out that there are NO impact at all on the existing Persistent Volumes : they are left untouched and are still mounted as when they were initially created with the old nfs-subdir-external-provisioner. The data remains there and the Kubernetes pods are still pointing to the same physical location on the NFS host for the existing Persistent Volume. As far as Viya knows, it should find its data right where it left it.    The reason is that in Kubernetes, a PersistentVolume (PV) has immutable properties which includes the volume ID, after creation. This means you cannot modify the volume's underlying storage details (like the specific disk or location) after it's been created.   So when we follow the procedure above, we are not really changing anything for what already exists in the environment. While there is now a new NFS provisioner installed and configured in the environment, it is not used for the existing Persistent Volumes.   If we wanted to truly move from using the old NFS provisioner to using the new NFS CSI driver, then we would need to DELETE and recreate all the PVs with the new provisioner. In such case we would need a way to recover what already existed in the PVs (either using backup/restore or setting the PV reclaim policy to RETAIN)...but I haven't tested any of these scenarios.     Observed differences   So what are the technical differences between the old NFS provisioner and the new one ? In one of the https://github.com/kubernetes-csi discussion, I found this comment with a pretty good answer to the question :   "nfs-subdir-external-provisioner use the in-tree NFS support, meaning it just creates PV's with .spec.nfs populated. Kubernetes natively supported many different kinds of storage (Ceph RBD, AWS EBS and other cloud specific storage). That supports is now getting moved out of core kubernetes into "drivers" which projects can support on their own, where the CSI standard says what the control plane will require of you. In simple terms the CSI driver calls mount instead of core kubernetes/kubelet itself."   Then it is explained that while the "in-tree" NFS support will not be removed for now, there are still some advantages such as the Volume snapshot support, and future extra features (such as NFS authentication with credentials, etc.).   Here are some other differences that we’ve noticed in our tests:   In the Helm installation command, the quotes around mount parameters (such as rsize, wsize or vers) can cause problems with the CSI driver installation (while it does not with the NFS subdir provisioner).   Options to organize the PV content in the NFS subfolders Default behavior with the nfs-subdir-external-provisioner : <NFS path>/<pvc namespace>-<pvc name>-<pv name> Example:   However you can change this behavior with the  pathPattern option   Default behavior with the CSI driver: <NFS path>/<pv name> Example:   However you can easily change this behavior with the subdir option    More pods ! While there was only one pod (with one container) created as part of the nfs-subdir-external provisioner, (as shown below),     The CSI driver setup creates many more of them, the NFS controller pod is managed by a ReplicaSet, while other pods are managed by a Daemonset object and run on each node.     Conclusion   If you are about to deploy a brand new SAS Viya environment, make sure you install the new recommended CSI NFS Driver.   If you are working in an environment where the old NFS subdir-external-provisioner has already been used to create and expose the Persistent Volumes, then switching to the new NFS CSI Driver does not change anything (until you create or recreate PVs).     Some lessons learned and recommendations:   Choose carefully your subdir (don’t choose the default values if you want to be able to easily identify the content of your Persistent volumes), Switching to the new CSI NFS driver doesn't change anything for your existing Persistent Volumes (until you start to use your new NFS provisioner to create new Persistent Volumes). Finally, you should also set at least 2 replicas for your CSI controller for better availability (in this case make sure that you have at least 2 nodes where the CSI Controller pods can be scheduled separately).   Special Thanks go to my colleagues, Rob Collum and Mike Goddard for sharing their findings on this topic and their collaboration in this post.     References   CSI Driver parameters CSI Driver helm charts Comparison to nfs-subdir-external-provisioner?       Find more articles from SAS Global Enablement and Learning here. ... View more

  With the 2024.07 release, SAS Mirror Manager introduced the ability to verify the integrity of SAS Viya container images. Then in 2024.10, it was improved further.    This feature provides the ability to check the Viya images authenticity and integrity, which is a critical security process for establishing trust with our customers deploying SAS Viya.   As noted in the official documentation, “SAS provides container image signatures for all official product images. These signatures can be used with SAS Mirror Manager and other tools in order to verify that images originated from SAS.”   In this post we’ll discuss the details of the Viya container image verification and see some examples.     What has changed?   SAS Mirror Manager verifies image signatures   Starting with 2024.07, SAS Mirror Manager verifies image signatures at the time when images are downloaded from SAS container repositories.   As illustrated in the diagram below, it means that each time you run the SAS Mirror Manager tool to pull the SAS Viya images from the official SAS container registry (cr.sas.com), a check of the images signatures is automatically performed. The public key included in the order’s certificates archive is used.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     SAS Mirror Manager mirror the image signature   Since the version 2024.10 of SAS Viya, SAS Mirror Manager can also mirror the image signature to some destination container registries (such as Azure Container Registry, Elastic Container Registry, and so on).   In the situation where you have made your Viya container images available into a private container registry (such as Azure Container Registry or JFrog), then you can also manually verify the associated signatures.   You can use an opensource tool called “Cosign” to perform the verification.       Requirements for image verification in a private registry   OCI version 1.1 SAS relies on the OCI version 1.1 specification for the SAS Mirror Manager image signature implementation. Signatures can be mirrored into any Container registry implementation that conforms to this specification.   As of today, the following Container Registry implementations support this version of the specification:   Azure Container registry Amazon Elastic Container Registry Google Artifact Registry   The image signing public key The signing key was added to new orders after May 2024, so older orders that were simply renewed will not have it in the corresponding certs.zip files. In the short term, if you are using an older order and want to verify the signatures of the images mirrored into a private registry, then current solution would be to request a new order. If a signing key is associated to your order, you should see a file named cosign_release_2023.pub when you unzip the order's certificate archive ("-certs.zip" file).   The Cosign open-source verification utility The Cosign source code and binaries can be found on the Cosign GitHub project.  You can install the cosign utility on many operating systems.   For example, on a RedHat Linux based operating system (RedHat Enterprise Linux, CentOS, Rocky Linux), you could simply run the commands below to download and install the RPM package:       Example   The SAS documentation provides the steps to perform the signature verification, and if you visit the latest version of the SAS Viya: Deployment on Google Kubernetes Engine hands-on (soon to be published in learn.sas.com), you can also find an example on how to do it against SAS Viya images that have been mirrored into the Google Artifact Registry.   Here are the steps :   Create the OAuth token to access the Google Artifact Registry When we run the Cosign tool, we need to provide a username and password to authenticate against the remote registry. With the Google Artifact registry, you can run a gcloud command to generate a short-lived OAuth access token to authenticate.     Install Cosign Download and use the appropriate package depending on your operating system. In our example we are running Cosign on Rocky Linux 9, so we simply download and install the RPM package (see the example above). Extract the image signing public key: You can simply use the unzip command to extract the public key from the order’s -certs.zip file. Verify the image signature with Cosign: In the example below, we collect the sas-logon-app image name, with its exact path in the registry and the associated image tag. Then we run the Cosign command using the syntax that is provided in the SAS documentation.       Finally, when you run the verification command, you should see something like this:       It confirms that the container image signature has been verified with success!     Conclusion   That’s it for today. This approach to validate the source and integrity of SAS container images improves the security posture of our software. Sites deploying SAS Viya can reliably ensure that the software they're installing comes from SAS in its original form without modification or replacement by intermediaries.   Special thanks to Elliot Peele for his support in the write up of this post !     Find more articles from SAS Global Enablement and Learning here. ... View more

The Ingress Controller is a critical requirement and key component of the SAS Viya infrastructure. In the first part of the post (that you should read first 😊 ), I explained the concept of the Ingress Controller, made the distinction between "Ingress NGINX" and "NGINX Ingress", then discussed the version, how to install and configure ingress-nginx .   In this second and last part, I will focus on some changes that are needed to allow SAS Viya to work with the latest version of ingress-nginx (1.12 and later). We will also see that several CVEs (including a critical one) were recently fixed in the ingress-nginx project and what is the impact for SAS Viya.   Here is a little "ToC" table that lists the key points, for this second part of the post 😊   Ingress NGNIX 1.12 breaking changes Ingress Common Vulnerability Exposures (CVEs) Am I affected by this security issue? Remediation: Upgrade version Can I always upgrade to a patched ingress-nginx version to fix these security issues? The NSG rule deletion issue Before ingress-nginx 1.12 ingress-nginx 1.12 and after Solution Conclusion     Ingress NGNIX 1.12 breaking changes   Quite recently, a note has been added in the documentation in case you were using ingress-nginx 1.12.x and later for your Kubernetes cluster.   Some default values have been changed beginning with 1.12 and the new default values could cause problems and failures to deploy SAS Viya. (Remember that if you plan to deploy SAS Viya in Kubernetes 1.31, ingress-nginx 1.12.x or later is mandatory).   Some additional configuration of the ingress-nginx controller is required :   For all Viya versions, you’ll need to set the value for annotations-risk-level to Critical If you deploy SAS Viya LTS 2024.09, or SAS Viya Stable 2024.10 or 2024.11 the strict-validate-path-type parameter must be set to false in the ConfigMap for the ingress-nginx controller in your Kubernetes cluster (when ingress-nginx 1.12.x or later is used). From stable 2024.12, SAS Viya works with the strict path validation that is enabled by default (strict-validate-path-type set to true).   The table below provides the commands to patch the ConfigMap depending on the SAS Viya version:   Viya Version Commands Any stable of LTS version before stable 2024.12 kubectl patch cm ingress-nginx-controller -n name-of-namespace -p '{"data":{"annotations-risk-level":"Critical", "strict-validate-path-type":"false"}}'; From  2024.12 kubectl patch cm ingress-nginx-controller -n name-of-namespace -p '{"data":{"annotations-risk-level":"Critical"}}';   Note : If the strict-validate-path-type configuration was modified for a previous SAS Viya platform deployment running ingress-nginx 1.12.x, it can be reverted with:   kubectl patch cm ingress-nginx-controller -n name-of-namespace -p '{"data":{"strict-validate-path-type":"true"}}';     Ingress Common Vulnerability Exposures (CVEs)   Many versions of ingress-nginx are affected by several security vulnerabilities that could make it easy for attackers to take over your Kubernetes cluster.   See the Kubernetes announcement related to these vulnerabilities: Ingress-nginx CVE-2025-1974: What You Need to Know.   As a consequence, your SAS Viya platform deployment might be exposed to this vulnerability.   This security issue has the following associated CVEs:   CVE-2025-1974 (Critical) CVE-2025-1098 (High) CVE-2025-1097 (High) CVE-2025-24514 (High) CVE-2025-24513 (Medium)   The best and easiest remedy is to upgrade to the new patched release of ingress-nginx .     Am I affected by this security issue ?   You can run this command to get the ingress-nginx version that you are running in your cluster:   kubectl get pods -A -l app.kubernetes.io/name=ingress-nginx -o jsonpath="{.items[*].spec.containers[?(@.name=='controller')].image}"   You should see something like this :   registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32c   Look at the version after the "controller" word, in this example we are running ingress-nginx 1.12.1.     Remediation: Upgrade version   If, for your customer environment, you obtain a version listed below, SAS strongly recommends that you upgrade to a patched version of ingress-nginx (supported by the Kubernetes version on which your cluster is running).      The SAS Technical Support has published an article for the customers with all the required guidelines, SAS® Viya® platform guidance for Kubernetes Ingress-NGINX Controller CVEs.     Can I always upgrade to a patched ingress-nginx version to fix these security issues ?   Customers currently running Viya LTS 2023.03 cannot upgrade to an ingress-nginx version that includes the fixes for the CVEs. SAS Viya LTS 2023.03 supports K8s 1.23-1.25 but unfortunately the ingress-nginx maintainers did not provide a fix for the CVEs that is supported on Kubernetes versions older than K8s 1.26…This is outside of SAS's control.   SAS recommendation is to upgrade to a newer Viya LTS/Kubernetes version combination. SAS Viya LTS 2023.10, LTS 2024.03, LTS 2024.09 will all work with the CVE patched version of Ingress NGINX.   Note: Viya LTS 2023.03 is EOL in May 2025 (As soon as LTS 2025.03 will be released - which is currently planned for the 15th of May 2025), so the update to a newer version of  SAS Viya is inevitable to remain in standard support.   In the latest DaC tool (viya4-deployment) version (8.2.0), the default version of ingress-nginx delivered has been changed to one that contains the fix for the CVEs.     The NSG rule deletion issue   Finally, there is something else that changed with ingress-nginx 1.12. It affects sites running SAS Viya in Azure where firewall rules must be added to allow additional inbound access to the Ingress external address (for example to allow access from Microsoft Entra ID when SCIM is configured).     Before ingress-nginx 1.12   Before 1.12, if you had to allow both inbound access to the Ingress Controller:   from Azure Entra ID (for the SCIM Client), AND from a range of IP address (corresponding to your clients IP addresses),   The proven practice was to :   Set the azure-allowed-service-tags annotation in your ingress-nginx controller definition (as shown below), for the inbound access from Azure Entra ID.     Then create your own Azure NSGs (Network Security Groups) - in the Azure portal or with the az CLI - to allow access from a specific range of client IP addresses…instead of using the LoadBalancerSourceRange specification in your ingress-nginx controller definition.   In the example below, you see the "manually created" NSG inbound security rule with a priority of 100 : it allows the access on port 80 and 443 from a range of IP addresses that correspond to our end-user’s IP addresses. But you also see the a rule generated (with a random name) from our azure-allowed-service-tag Ingress NGINX annotation which allow Azure Entra ID ("AzureActiveDirectory") to connect to our SAS Viya environment (for the SCIM push).     These guidelines are coming from this post written by Stuart Rogers.   In the post, Stuart explains in details, the reason to use "manually created" NSGs instead of the Ingress NGINX  controller.service.loadBalancerSourceRanges specification …basically loadBalancerSourceRanges uses IPTABLES to block any connection from outside of the defined range – hence preventing the access from Azure Entra ID !   This kind of setup has worked for several years. However it was recently discovered that this setup would not work any longer with ingress-nginx 1.12 !     ingress-nginx 1.12 and after   What happens now, if you implement this setup with ingress-nginx 1.12 or later is that our "manually created" NSG rules is silently removed by Azure after some time  ! (1 to 3 days).   One of our colleague, at SAS, discovered the issue and, after some investigation, found out (from the ingress-nginx-controller logs) that Azure/AKS was periodically "reconciling" the ingress controller firewall rules with the NSG, with the changes being executed by the AKS service principal :   Warning ConflictConfiguration 3m15s azure-cloud-provider Please use annotation service.beta.kubernetes.io/azure-allowed-ip-ranges instead of spec.loadBalancerSourceRanges while using service.beta.kubernetes.io/azure-allowed-service-tags annotation at the same time.     Solution   As suggested in the log message, the solution is to configure the ingress-nginx-controller service to use the   azure-allowed-ip-ranges annotation instead of creating the NSG rules directly in Azure.   This change allows Azure to properly create the required inbound NSG rules for both Entra/SCIM AND end-user source ranges.   Our colleague provided an example on how to patch a "DaC" deployed environment to implement the solution :   kubectl -n ingress-nginx describe svc ingress-nginx-controller # add the "azure-allowed-service-tags" annotation to allow Azure Entra ID inbound access (SCIM) kubectl -n ingress-nginx patch svc ingress-nginx-controller --type='json' -p='[{"op": "add", "path": "/metadata/annotations/service.beta.kubernetes.io~1azure-allowed-service-tags", "value":"AzureActiveDirectory"}]' # remove the loadBalancerSourceRanges specification kubectl -n ingress-nginx patch svc ingress-nginx-controller --type=json -p='[{"op": "remove", "path": "/spec/loadBalancerSourceRanges"}]' # add the azure-allowed-ip-ranges annotation to allow a range of IP adresses to kubectl -n ingress-nginx patch svc ingress-nginx-controller --type='json' \ -p='[ { "op": "add", "path": "/metadata/annotations/service.beta.kubernetes.io~1azure-allowed-ip-ranges", "value": "109.xxx.xxx.xxx/27,149.xxx.0.0/16,194.xxx.xx.xxx/28,10.0.0.0/16,13.xx.xxx.xxx/32,192.xx.xx.0/24,48.xxx.xxx.xxx/32,10.xx.xx.xx/16" } ]'   But if you are not using the "DaC" tool (viya4-deployment) and want to install ingress-nginx on your own with Helm, you can use the script below (provided by Stuart Rogers) as a reference:   helm install ingress-nginx --namespace ingress-nginx \ --set controller.service.externalTrafficPolicy=Local \ --set controller.service.sessionAffinity=None \ --set controller.config.use-forwarded-headers="true" \ --set controller.autoscaling.enabled=true \ --set controller.autoscaling.minReplicas=2 \ --set controller.autoscaling.maxReplicas=5 \ --set controller.resources.requests.cpu=100m \ --set controller.resources.requests.memory=500Mi \ --set controller.autoscaling.targetCPUUtilizationPercentage=90 \ --set controller.autoscaling.targetMemoryUtilizationPercentage=90 \ --set-string controller.service.annotations."service\.beta\.kubernetes\.io/azure-allowed-service-tags"="AzureActiveDirectory" \ --set-string controller.service.annotations."service\.beta\.kubernetes\.io/azure-allowed-ip-ranges"="10.244.0.0/16, 149.xxx.xxx." \ --version 4.12.1 \ ingress-nginx/ingress-nginx kubectl wait -n ingress-nginx --for=condition=available --all deployments --timeout=60s # Changed according to https://documentation.sas.com/doc/en/itopscdc/v_060/dplyml0phy0dkr/n02w9l89l6b6ttn1wncv617wndp3.htm kubectl patch cm ingress-nginx-controller -n ingress-nginx -p '{"data":{"allow-snippet-annotations":"true", "annotation-value-word-blocklist":"load_module,lua_package,_by_lua,location,root,proxy_pass,serviceaccount,{,},\\"}}'; # Changes in 2025.02 if version_ge "$NGINX_Version" "4.12.0"; then # Changed according to https://documentation.sas.com/doc/en/itopscdc/default/dplyml0phy0dkr/p1deewvd9xtqiyn168adq6fu9msv.htm if version_ge "$CADENCE_VERSION" "2024.12"; then kubectl patch cm ingress-nginx-controller -n ingress-nginx -p '{"data":{"annotations-risk-level":"Critical"}}'; else kubectl patch cm ingress-nginx-controller -n ingress-nginx -p '{"data":{"annotations-risk-level":"Critical", "strict-validate-path-type":"false"}}'; fi fi   In this case, the NSGs for both Entra ID and the range of allowed client IPs would be automatically generated in Azure as shown below:       Conclusion   That’s it for today…If you have read the whole post (part 1 and part 2) you should now be ready to move to the new ingress-nginx version 1.11.5 or 1.12.1 or later ! 😊 Thanks to Heather Hewlett, Tony Gianni and Stuart Rogers for sharing their knowledge and experience with the ingress-nginx setup and configuration.     Find more articles from SAS Global Enablement and Learning here. ... View more

The Ingress Controller is a critical requirement and key component of the SAS Viya infrastructure. This component can be seen as the "front door" or "entry point" of your SAS Viya deployment. SAS currently (as of April 2025) only supports the Ingress NGINX Controller (aka ingress-nginx) - with the exception of OpenShift routes if you are deploying SAS Viya in the Red Hat OpenShift platform.   There were some recent changes and announcements related to the Ingress NGINX Controller, so I thought it could be a good time to list and explain them.   As there was a lot to discuss I have split this post in two parts.   Here is a little "ToC" table for this first part to help you to navigate through this post 😊   What is the Ingress Controller? Ingress NGINX <> NGINX Ingress …A little clarification on the Terminology Which ingress NGINX version should I use? How do I install ingress-nginx? Vulnerability Mitigation for the "allow-snippet-annotation"   What is the Ingress Controller ?   In a Kubernetes cluster, an "Ingress" is the object that makes your web (HTTP or HTTPS) service available from the outside. It lets you map traffic to different backends based on rules that you define in the Ingress definition.   But in order for the Ingress resources to work, the cluster must have an Ingress Controller running (usually inside its own Kubernetes namespace). There are many implementations of the Ingress Controller component (Cloud plugins or generic, based on different technologies such as HAProxy, Istio, Envoy, etc.).   The Ingress Controller acts as a reverse proxy and load balancer. Its main role is to read the rules defined in Ingress resources and route incoming traffic to the appropriate services.   Ingress NGINX <> NGINX Ingress…A little clarification on the Terminology   There are two types of nginx ingress controllers with VERY VERY similar product names ☹:   One from the community (Ingress NGINX) The second from F5 (NGINX Ingress)   They are really different and the only one we support is the one from the community.     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   SAS supports the Ingress NGINX Controller which is commonly called ingress-nginx.   So here is the link to the project: ingress-nginx  to avoid any further confusion!   It is important, because if you install the wrong one (NGINX Ingress), then the annotations used in the SAS Viya ingresses definitions will not work…   Which version should I install?   The SAS Viya official documentation has a dedicated section to answer this question.   If you look at the latest version of the documentation (2025.04 at the time of this write up), you can see that the SAS Viya platform supports ingress-nginx 1.9.6 and later. Another important requirement is that ingress-nginx snippet annotations setting must be enabled.   However we can also read : "Make sure that the version of ingress-nginx that you are using is compatible with your cloud provider, and check for updates."   So aside the minimum version requirement and the need to allow snippet annotations, what really drives the ingress-nginx version is the Kubernetes version.   The ingress-nginx GitHub page has a reference table that tells you which Kubernetes versions are supported for each ingress-nginx versions.   See below an extract from this table:     It means that, for example, if you are running Viya 2025.03 on Azure AKS 1.31, you need to deploy at least ingress-nginx with v1.12. The corresponding Helm chart version is also included, which will come in handy in the next section.   How do I install ingress-nginx ?   Since ingress-nginx is a system requirement for SAS Viya, it should be the responsibility of the customer to deploy it.   However, the customer may ask for your help or for a test environment you may have to deploy it by yourself. The instructions to deploy ingress-nginx are provided in the "SAS® Viya® System Requirements" course available on learn.sas.com (see the extract below).     Using Helm is the most common way to install ingress-nginx. You need to provide the Helm chart version as shown in the example above.   Another point of reference, if you are looking for the commands to install ingress-nginx is the viya4-deployment GitHub project (aka "DaC" – for Deployment As Code).   The DAC playbook has a "baseline" role that includes the setup of ingress-nginx. If you look at the corresponding codebase , you’ll notice that several rules are now used to determine the correct version to install and implement specific configuration (CVE mitigation, annotations, strict path validation,…) depending on the Kubernetes version, Viya version and Cloud provider…   So, even if you don’t use it to install SAS Viya itself, it could be a good idea to use only the "baseline" feature of the DaC tool to ensure that you install the correct ingress-nginx version with the required configurations.   Vulnerability Mitigation for the "allow-snippet-annotation"   Back in 2021, a security issue was discovered in ingress-nginx where a user, who can create or update ingress objects, can use the custom snippets feature to obtain all secrets in the cluster! (see CVE-2021-25742 for more details)   However, the SAS Viya platform requires custom snippets to function properly.   So for each supported LTS and stable Viya version there is a set of instructions provided in the SAS documentation to explain how to mitigate the security risk by using an annotation-value-word-blocklist annotation. The  annotation-value-word-blocklist contains the characters/words that are well known of being used to abuse Ingress configuration and must be blocked. When an annotation is detected with a value that matches one of the blocked bad words, the whole Ingress won't be configured (source).   Here is an example on how to patch the ingress-nginx configMap object to implement this mitigation :   kubectl patch cm ingress-nginx-controller -n ingress-nginx -p '{"data":{"allow-snippet-annotations":"true", "annotation-value-word-blocklist":"load_module,lua_package,_by_lua,location,root,proxy_pass,serviceaccount,{,},\\"}}';   SAS strongly recommends to implement this mitigation to limit the risk of attackers exploiting this vulnerability. That's all for this first part of the post !   Now that you know what is the Ingress NGINX controller, which version to install and how to install it, you are ready for the next part where we will discuss the changes brought by the new ingress-nginx version (1.12) as well as some new security patches recently released to address several CVEs (Common Vulnerabilities and Exposures).     Find more articles from SAS Global Enablement and Learning here. ... View more

Last month, our team was contacted, by a colleague in the field, to help with a customer request.  The customer was interested in discussing the PSS (Pod Security Standards) settings in relation to a default SAS Viya installation.   More specifically, they were asking for guidance regarding several perceived security violations in their environment after the deployment of SAS Viya 2025.01.   Looking into the details of the "policies violation" report and with the help of colleagues experienced in this area (Alexander Koller, Carus Kyle) we were able to provide some guidance.   We've now had a number of queries about Kubernetes Pod Security Standards and other kind of Pod’s policies in a Kubernetes cluster, so we thought it may be the right time to write a little blog to discuss this topic through a customer use case. 🙂   First, a little reminder about PSS   As a Kubernetes administrator in charge of the Cluster security, you should implement Security Standards for your Pods. Until version 1.25, "Pod Security Policies" was the mechanism provided in Kubernetes, to implement these standards. However, while they allowed a very granular control of the pods, there were complaints about the complexity of this "fine grain" policies.   As a result, from Kubernetes 1.25 the "Pod Security Policies" were removed, and administrators now have the choice between:   Pod Security Admission : a built-in Kubernetes mechanism to enforce Pod Security Standards with 3 levels of isolation: privileged, baseline, and restricted. a 3rd party admission plugin, that they can deploy and configure themselves.   So, the security rules that exist in a Kubernetes Cluster (and that can prevent a SAS Viya deployment) are either coming from the Kubernetes PSA configuration or third-party plugins (such as GateKeeper/OPA, Kyverno, etc…).   One major difference between the deprecated "Pod Security Policies" and the new "Pod Security Admission with the Pod Security Standards" is that, once a PSS profile (baseline or restricted) is chosen, it is not possible to define exceptions.   List of policy violations   Now let’s come back to our customer’s case, so we can illustrate the PSS concept.   Here is the “policies violations” report as provided by the customer:   # Name Total Violations Description 1 container-must-have-limits 1 violations - enforcementAction: deny group: ''kind: Podmessage: >-container <sas-cas-server> memory limit <28Gi> is higher than the maximum allowed of <8Gi> name: sas-cas-server-default-controller namespace: sasviya version: v1 2 psp-pods-allowed-user-ranges   879 violations: - enforcementAction: deny group: '' kind: Pod message: >- Container hydrator is attempting to run without a required securityContext/runAsGroup. Allowed runAsGroup: {"ranges": [{"max": 1337, "min": 0}], "rule": "MustRunAs"} name: sas-annotations-6644dc6f99-cmz97 namespace: sasviya version: v1- enforcementAction: deny group: '' kind: Pod message: >- Container hydrator is attempting to run without a required securityContext/runAsUser name: sas-annotations-6644dc6f99-cmz97 namespace: sasviya version: v1 3 psp-allow-privilege-escalation-container   1 violations: - enforcementAction: deny group: '' kind: Pod message: 'Privilege escalation container is not allowed: sas-opendistro-sysctl' name: sas-opendistro-default-0 namespace: sasviya version: v1 4 psp-host-filesystem   1 violations: - enforcementAction: deny group: '' kind: Pod message: >- HostPath volume {"hostPath": {"path": "/mnt/nvme-disks", "type": ""}, "name": "cas-disk-cache"} is not allowed, pod: sas-cas-server-default-controller. Allowed path: [{"pathPrefix": "/foo", "readOnly": true}] name: sas-cas-server-default-controller namespace: sasviya version: v1 5 psp-privileged-container   1 violations: - enforcementAction: deny group: '' kind: Pod message: >- Privileged container is not allowed: sas-opendistro-sysctl, securityContext: {"allowPrivilegeEscalation": true, "capabilities": {"drop": ["ALL"]}, "privileged": true, "readOnlyRootFilesystem": true, "runAsNonRoot": false} name: sas-opendistro-default-0 namespace: sasviya version: v1   Let’s review each type of violation and try to understand what it means.   #1 container-must-have-limits   As the name suggests, this policy seems to enforce restrictions on the resource limits that can be set for a container.   The description (or violation’s message) is pretty clear here: the resource limit for the memory is set to 28Gi in our sas-cas-server-default-controller container. But it is higher than the maximum allowed of 8Gi.   The CAS server has unique attributes and requirements. Because it has been engineered to use all available resources to complete each request in the shortest amount of time, CAS is usually configured to run on dedicated nodes, using most of the available resources.   By default, CAS operates with auto-resourcing enabled, allowing the server to dynamically adjust resource usage. In such case, the CAS Operator scans node specifications and allocates around 80% of the available memory to CAS worker pods. It is likely the reason why the memory is set to 28Gi, which does not meet in this case, this custom and general policy of not exceeding the 8Gi for any container.   #2 psp-pods-allowed-user-ranges   According to its name and the associated description, this policy is related to the user account used to run the container.   When looking at the description, it looks, at first, like there is a specific issue with the sas-annotations pod and its hydrator container …But in hindsight, it is likely that this violation has been actually observed for most of the SAS Viya containers, because there are 879 violations of the policy 😊 (So it probably also includes the init containers referenced in the Viya pods)   According to the violation’s message, the broken rules here are that neither runAsGroup nor runAsUser specification are set in the Security Context of the containers.   In Kubernetes, a Security Context "defines privilege and access control settings for a Pod or Container". It can be used to force all the processes inside a container to run under the same user, add or remove Linux capabilities for a Container and many other things in terms of security controls...Refer to the Kubernetes documentation for more details on Security Contexts.   When looking at the securityContext specification of the hydrator container in the sas-annotation pod, we can, indeed, confirm that the absence of runAsGroup and runAsUser specifications:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   I'm using k9s in this example, but you could use a kubectl command to get the same information.     You can even use the script below to run a report across all the SecurityContext that are set for the pods and containers in a given namespace :     The Security Context can be set at both the Pod level and Container level. The report tells you, for each pod, if a PodSecurityContext is set and for each container inside the pod if a SecurityContext is set. When set, the securityContext specification is provided.   Most of the time for all the SAS Viya pods, you'll see that the runAsGroup or runAsUser attributes are not set in the SecurityContext:     The only pods (in this 2025.02 Viya deployment) with these attributes set, are the CAS and sas-pyconfig pods.   Example:     Here, the runAsUser field specifies that for any Containers in the CAS Controller Pod, all processes run with user ID 1001.   Although the runAsUser is not set for (almost) all the SAS Viya pods, if you run the ps -ef command inside a SAS Viya container, you would see that all the processes run under the sas account. It’s because in Kubernetes, when  runAsUser is not set inside the SecurityContext, the main process of the container runs with the user ID specified in the container image (USER instruction in the container’s image Dockerfile).   Additionally, while the runAsGroup or runAsUser attributes are not set, what we see in the report is that most of the SAS Viya pods have the runAsNonRoot attribute set to truein their PodSecurity context (which requires all the containers in the pod to be run as non-root users).   Interestingly, setting this runAsNonRoot attribute to true meets one of the policy requirements of the restricted isolation level in the Kubernetes Pod Security Standard definition…) - as you can see below...     #3 psp-allow-privilege-escalation-container   This policy is related to the privilege escalation (such as via set-user-ID or set-group-ID file mode) that, may be sometimes required for some of the SAS Viya components or for specific Viya customizations…   The message tells us that the “Privilege escalation container is not allowed” for the sas-opendistro-sysctl container inside the sas-opendistro-default-0 pod.   You can confirm it if you look at the container’s Security Context:     It is a known and documented issue with the deployment of OpenSearch inside the SAS Viya platform:     The offending container is created by the sysctl-transformer.yaml that is set by default in the "transformers:" section of the kustomization.yaml file :   sas-bases/overlays/internal-elasticsearch/sysctl-transformer.yaml   As documented there in the SAS Viya Platform Operation guide, the purpose of the transformer is to set the vm.max_map_count kernel parameter on the OpenSearch underlying host to ensure there is adequate virtual memory available for accessing the search indices.   The SAS Documentation explains that, instead of using the transformer, it is possible to set the vm.max_map_count in advance on the hosts where the OpenSearch pods will be running (stateful nodes).   While you could do it manually on the nodes using the commands provided in the SAS documentation (basically run the sysctl -w vm.max_map_count=262144 command as root on the host), you can plan it in the infrastructure provisioning, using a custom script (or "user data") to be run when the VM is provisioned (see examples in Azure  Google Cloud or AWS).   Also note that a PR (Pull Request) has been submitted in the IAC for Azure GitHub repository to enable Terraform to implement this setting when the Stateful nodes are provisioned.   #4 psp-host-filesystem   Once again, the policy violation is pretty clear: HostPath volume {"hostPath": {"path": "/mnt/nvme-disks", "type": ""}, "name": "cas-disk-cache"} is not allowed.   While it’s often the most performant option, the usage of the hostPath volume type is generally seen as a security vulnerability in Kubernetes.   According to the violation description, hostPath volume type has been configured for the CAS Disk Cache location.   A possible workaround is to: not mount the hostPath location directly to the pod, but instead, create a PersistentVolumeClaim pointing to a PersistentVolume using the local-storage or manual Storage classes (Alexander Koller also provides nice examples there).   If you are deploying SAS Viya in opensource Kubernetes, a local-storage should already exist for you.   If you are deploying in the Cloud, you can also explore different options for the CAS Disk Cache, such as the Kubernetes Generic Ephemeral Volumes. If SAS Viya is deployed in Azure, you could also use the emptyDir volume type with ephemeral OS disk in Azure (as demonstrated in this opened Pull Request that has been submitted in the IAC for Azure GitHub repository).   But I’ll stop here, as the topic of "avoiding the use of hostPath and NFS volumes in a SAS Viya deployment to comply with security standards" would deserve its own post 😊.   #5 psp-privileged-container   Once again, the issue here is with the OpenSearch sysctl container.   According to the message, what triggers the violations here are the privileged: true, and runAsNonRoot: false specifications in the container’s securityContext specifications. Note that, in the Kubernetes built-in PSSs, these settings are respectively forbidden with the baseline and restricted profiles.   As explained above, this violation is the result of the sysctl-transformer.yaml transformer that is included by default in the kustomization.yaml for the OpenSearch container and the same solutions/workarounds apply.   Summary and answers for the security violations   So, let’s summarize the violations and solutions or workarounds in a table (as they were provided to our colleague working with the customer).   This information will help the customer to understand the reason why some constraints were not observed during the Viya deployment and how to fix the issue.   # Name Description Notes/Solution 1 container-must-have-limits message: >- container <sas-cas-server> memory limit <28Gi> is higher than the maximum allowed of <8Gi> Policy Source: Gatekeeper custom policy.   The CAS CPU/memory resources settings (limits/request) can be changed, and the memory limit can be set below 8GB. However, it will likely impact the CAS topology, workload placement, performance and potentially limit the amount of data to be loaded.   To be clear: yes, we can tune the CAS container, so its memory limit meets the 8GB maximum allowed but does it fit with the Architecture design / requirements agreed with the customer regarding CAS? (Most of the time, CAS is the component in SAS Viya that is expected to fully utilize host RAM in most scenarios.)   If it does not, the customer could consider adding an exception to the policy for this specific pod.   2 psp-pods-allowed-user-ranges   message: >- Container hydrator is attempting to run without a required securityContext/runAsGroup. Allowed runAsGroup: {"ranges": [{"max": 1337, "min": 0}], "rule": "MustRunAs"} name: sas-annotations-6644dc6f99-cmz97message: >- Container hydrator is attempting to run without a required securityContext/runAsUser Policy Source: Gatekeeper custom policy.   This policy controls the user and group IDs of the container (more specifically runAsUser, runAsGroup, supplementalGroups, fsGroup specs...).   In SAS Viya, except for CAS and Compute pods, the runAsUser, runAsGroup attributes are not set.   As a workaround you can write a patchTransformer that adds the runAsUser attribute to the pod definition with the correct user ID 1001. Since most of the SAS Viya pods have the runAsNonRoot parameter set to true, asking the customer for an exception is another option to consider.   3 psp-allow-privilege-escalation-container   message: 'Privilege escalation container is not allowed: sas-opendistro-sysctl' Policy Source: Kubernetes PSS, restricted isolation level.   See the OpenSearch section in the SAS Viya operations guide.   4 psp-host-filesystem   message: >- HostPath volume {"hostPath": {"path": "/mnt/nvme-disks", "type": ""}, "name": "cas-disk-cache"} is not allowed, pod: sas-cas-server-default-controller. Policy Source: Kubernetes PSS, baseline isolation level.   If hostPath is not allowed, then the customer needs to find another Kubernetes volume type for CAS disk cache. Using the Generic Ephemeral Volume with an AWS managed disk may be a good solution.   Performance may be degraded compared to using local NVME drives though.   5 psp-privileged-container   message: >- Privileged container is not allowed: sas-opendistro-sysctl, securityContext: {"allowPrivilegeEscalation": true, "capabilities": {"drop": ["ALL"]}, "privileged": true, "readOnlyRootFilesystem": true, "runAsNonRoot": false} Policy Source: Kubernetes PSS, baseline and restricted isolation level.   See the OpenSearch section in the SAS Viya operations guide.   Conclusion   More and more customers, investing in Kubernetes, are now implementing cluster security policies, using the Kubernetes built-in Pod Security Standards and/or 3 rd party tools.   In this particular case, the customer was deploying SAS Viya in AWS.  They had a mix of PSSs, with the restricted isolation level and also specific Gatekeeper constraints.   Note that we have seen some customers cases where additional security rules were also enforced by turning on the Cloud vendor’s built-in policies engine, such as the Azure AKS Built-in policies…   On the other hand, the SAS Viya has very specific requirements, support a wide range of customization that may not meet the generic policies implemented in the cluster by the customer.   That’s the reason why it is important to understand exactly what the constraints are, in order to determine workarounds or required configuration changes in the SAS Viya platform (when available).   Sometimes, the violations of the policies in place are indicative of a real problem. Other times, like here, they simply need an explanation and sometimes an exception to the policies can be implemented.   While it is possible, in general,  for SAS Viya to run in a cluster where the Kubernetes built-in restricted PSS is enforced, it often requires to find alternatives to commonly used implementations (ex: CAS and Compute access to data through NFS or hostPath, enable host account on CAS) and some specific customizations may require workarounds or may even not be possible (ex: enable SAS Watchdog to monitor SAS sessions, executing python code in MAS, etc.).   That's it for today, I hope it was helpful !   References   SAS Viya Platform Operations documentation: Pod Security Admissions SecurityContext : https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ Pod Security Standards, profile details: https://kubernetes.io/docs/concepts/security/pod-security-standards/#profile-details     Find more articles from SAS Global Enablement and Learning here. ... View more

As a Deployment engineer - about to start the installation of a new Viya environment in Kubernetes - or as a Technical Architect - working on the storage design for a Viya project - you may hear, or be questioned by your customer about something called the "CSI drivers"...   When searching in the SAS Official documentation, you could be surprised to see no list of officially supported CSI drivers for Kubernetes! It is because SAS can unfortunately not test each and every technology associated to Kubernetes or Containers in every scenario of a SAS Viya deployment.   As my colleague, Rob Collum recently said in one of our team meetings : "Some things are tested and supported by SAS and some other things… can possibly work"…his remark was spot-on : a lot of technologies used in the Cloud or in Kubernetes platforms can be used and are likely to work with SAS Viya, but it does not mean that they have been officially tested by SAS for every kind of  SAS Viya deployment…     However, what we can do, as SAS professionals is to share the experience and the lessons learned in the field where customers have implemented proven technologies, even when they were sometimes not explicitly listed as fully tested and supported in the SAS Documentation.   There are several examples of that: Customers deploying Kubernetes platform under limited support (Tanzu, Rancher, etc.), customers using new network or storage container integration technologies, etc.   In this particular post, we discuss and share some feedback on the usage of the Container Storage Interface (CSI) drivers.   What are the CSI drivers?   In order to expose external storage to the containers running in the pods, Kubernetes initially came up with an "in-tree" volume plugin system (including things like awsElasticBlockStore, azureDisk or cinder).   But the code was part of the Kubernetes system which caused friction over time…it forced storage vendors to align with the Kubernetes release system, and caused reliability and security issues in the core Kubernetes binaries (with a code that was difficult to test and maintain for the Kubernetes maintainers).   To address this challenge, the CSI (Container Storage Interface) was developed as "a standard for exposing arbitrary block and file storage systems to containerized workloads on Container Orchestration Systems (COs) like Kubernetes".        Or simply put, a CSI driver is a standard that allows Kubernetes to interact with storage systems.   Nowadays, most of the original volume plugins (awsElasticBlockStore, azureDisk or cinder) are deprecated and the official Kubernetes documentation recommends to use the "out-of-tree" volume plugins implementing the CSI.   Once installed in the cluster, new available storage classes appear in Kubernetes and can be referenced by the SAS Viya Persistent Volume claims, volumeClaimTemplates or Custom resources.   In the Cloud Managed Kubernetes environments (AKS, EKS, GKE, etc.) the Cloud CSI drivers maybe be pre-installed and the associated storage classes are already available.   Why do we need them?   Cloud providers provide CSI drivers for a better integration with their storage managed services. In the "on-premise" world, Storage systems vendors provide CSI drivers for a better integration with their SAN or NAS storage systems.   The usage of the NFS subdir provisioner is generally more flexible than the CSI drivers and its usage is currently encouraged both in our official documentation and in the SAS "Deployment as Code" GitHub project (aka DAC).   However, we see more and more customers expressing concern about using "open-source community provided" software in their production environments and require the use of official Cloud supported CSI drivers instead.   In addition of the Cloud vendor support, there can be other benefits such as TLS encryption in-transit or at-rest, automatic reprovisioning when additional space is required, advanced integration with the Cloud file services (snapshots, backups, etc.).     CSI Drivers Known limitations and requirements   The CSI drivers are mentioned in only two specific locations of the SAS official documentation.     Source: dilbert.com (Scott Adams)  Limitations for Azure Files and EKS CSI drivers   While the SAS documentation does not provide a formal list of supported CSI drivers, there is a warning in this section against the usage of specific CSI drivers.   Two specific CSI drivers that do not meet the CAS and the SPRE pods requirement for a fully POSIX-compliant system, are listed:   The Microsoft AKS Container Storage Interface (CSI) driver for Azure Files. When this CSI driver is set to use the SMB protocol, it does not support the standard POSIX file system operations that SAS processes use to unlink or delete files. The Microsoft AKS Container Storage Interface (CSI) driver for Azure Files. The Amazon EKS Elastic File Service (EFS) CSI driver. This driver does not allow files to be owned by distinct UIDs and GIDs. Certain SAS processes would not be able to chown files that are stored in these volumes. In addition, it is recommend to avoid the use of Azure Files as the backing storage for the internal PostgreSQL instance because SAS testing with this configuration has found multiple issues.   These warnings come from real life experience where customers have tried to use these CSI drivers in the field and faced issues during file access operations with CAS, the Compute servers or the internal crunchy PostgreSQL. As an example read about two SAS colleagues' experience with the aws-efs-csi-driver…) in the extract below from a SAS Internal blog :   Using the aws-efs-csi-driver In our IAC for AWS, the nfs-subdir-external-provisioner is used when EFS is the desired persistent storage. However, when customers aren’t using our IAC, the aws provided efs csi driver is commonly used instead. Honestly, it makes sense logically, but there is of course a gotcha when this choice is made. While EFS claims to be POSIX compliant, it seems the provided csi driver didn’t get the memo. When using the EFS CSI driver, there is no way to alter ownership of files or directories in PVCs. The chown/chgrp command will fail as described here.  To add to the fun, the issue has been marked as closed and not planned… This often presents itself as:     rsync: chown …  failed: Operation not permitted (1)  For now, our best recommendation is to avoid use of the aws-efs-csi-driver and instead use the nfs-subdir-external-provisioner as the IAC does.   But as noted in the documentation, and explained in the post’s introduction: "This is not an exhaustive list of storage options that do not fulfill SAS requirements. Many storage options are available across all the supported cloud platforms and Kubernetes distributions. SAS cannot test with all of them."     Understanding the nuances…   Now we have to be careful…these known limitations, listed above, DO NOT MEAN that Azure Files or Amazon EFS storage can never be used with SAS Viya…   The devil is in the detail…and yes I know, it is complicated…   But for example:   Azure Files Premium, with the NFS protocol support can be used and could be a great fit for many Viya volumes that requires RWX access mode. Amazon EFS, is officially documented as the storage to use in HA deployment of SAS Viya (preferred over the Single VM NFS server option for the RWX Viya volumes), and is a supported option in the SAS provided IaC/DaC tools. However it is preferable to mount the EFS storage with a generic NFS provisioner (rather than the EFS CSI driver as it has some limitations).   So, when considering the storage options for a Viya environment, it is important to take several things into account :   The kind of Viya volume we deal with: Is it one of the Viya platform dynamically assigned during the deployment? or a static volume to store user’s data? are we looking for a shared storage (RWX) or would block storage (match RWO access mode request) be a better option for this volume? Full POSIX support (usually depends on the protocol used to access the file system). How is the storage mounted (specific CSI driver or generic NFS provisioner?), is there a minimum storage size for the volumes mounted with the CSI?     Required CSI Driver   The other location where the CSI driver is mentioned in the SAS Official documentation is in the "Cluster requirements for AWS" section.   Instead of a limitation, this time, there is a requirement to have the AWS Elastic Block Store (EBS) CSI driver installed in the cluster.   As explained in this blog post from Rob Collum, the reason of this requirement is that since EKS 1.23, without a specific installation of the EBS CSI driver inside the cluster, it is not possible to use the default AWS gp2 or gp3 storage classes.   These default storage classes are recommended for the SAS Viya volumes that require RWO access (such as OpenDistro, Redis, RabbitMQ, etc.…).     Recap of the most common Cloud storage provisioner and CSI   At this point, and to avoid further confusion, let's look at a small table listing the most common Cloud vendors CSI drivers.   Cloud Provider CSI Driver Storage Storage type Access mode AWS efs.csi.aws.com EFS (Elastic File storage) NFS managed service RWX AWS ebs.csi.aws.com EBS (Elastic Block Storage) Managed disk (Block storage) RWO Azure file.csi.azure.com Azure Files Premium NFS managed service (NFS support) RWX Azure disk.csi.azure.com Azure Disks Managed disk (Block storage) RWO Google filestore.csi.storage.gke.io Google Filestore NFS managed service (NFS support) RWX Google pd.csi.storage.gke.io Google Persistent Disks (standard-rwo) Managed disk (Block storage) RWO     The Azure Files CSI driver "trap"   While you can use the Azure Files CSI driver (assuming you use Azure Files premium with the NFS support), there is an additional gotcha that you’ll need to know if you want to use it for your SAS Viya platform dynamically provisioned volumes...   Back in 2023, a SAS colleague explored the use the Azure Files Premium CSI driver and provided guidance on how to implement it with the NFS protocol and the nconnect mount option. He wrote an internal blog that shows how you can use the Azure Files storage for your SAS Viya dynamically provisioned volumes during the deployment and provision them with the Azure Files CSI driver and that it all works perfectly.   …HOWEVER…   In this case, for each dynamically requested Viya volume, the use of the CSI driver triggers the provisioning a distinct new Azure File Share.   Now the problem is that, as of today the minimum size for Azure Premium files share is 100GB   The result is that the provisioned capacity in Azure is way higher than what is effectively needed…Some volumes only require 1GB (or even less), but a 100GB file share is provisioned in Azure for each of them.   The screenshot below compares the required capacity with the provisioned capacity.       Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   So while it is technically possible to use the CSI driver here, the cost is very high and a lot of space is wasted.  This situation was recently faced at a customer site by a SAS colleague in the AP region.   A first workaround, that the customer tried, was to set the shareName attribute in the Azure Files storage class. When this attribute is set, all volumes that use the azurefile-csi storage class consolidate their content in the same Azure Files share (so we don't need to create one file share for each PVC).   For example, if we have the following 9 PVCs:   cas-default-data cas-default-permstore sas-cas-backup-data sas-common-backup-data sas-commonfile sas-pyconfig sas-risk-cirrus-core-backup-data sas-risk-cirrus-search-data   …then the content of all the corresponding PV is created at the root level of a single file share, as below:     … which poses a risk of data contention and potential overwrites due to the lack of directory isolation.   This shareName attribute has been tested by the SAS R&D who confirmed that it was not suited for dynamic volume provisioning. Using this setting with Viya could lead to data corruption issues as the Viya components, which request storage through distinct PVCs, expect distinct volumes to be provisioned.   As a result of this experience and test, a new section Guidance When Using Microsoft Azure Files was added in the official documentation.     The official workaround, in order to use Azure Files for the dynamic provisioning of the RWX-access based Viya platform volumes, is to use an NFS provisioner. The Kubernetes NFS Subdir External Provisioner can be used to create a custom storage class based on an existing Azure Premium Files share. The NFS provisioner can create distinct subdirectories for each Viya volumes under the Azure file share. Note that (at the time of this post write up), this NFS subdir external provisioner is used for various NFS -based storage types by the Deployment as Code project (aka "viya4-deployment").     A better experience with the GKE Filestore CSI Driver   This "over-provisioning" issue when using the CSI driver for dynamic provisioning is not specific to Azure, the same problem has been seen in the past with the Google File Store.   However while the GKE Filestore CSI driver version 1.26 and earlier only allowed for 10 shares at 100GB minimum, things have been improved with the enterprise-multishare-rwx GKE storage class, where the minimum size has been reduced to 10 GB.   While there might be a bit of extra provisioned space, the overhead is minimal.     Conclusion   While using CSI drivers can cause issues, it is also something that is expected to be required by more and more customers…as it often provides a better integration with the managed storage services and extends the Kubernetes capabilities (encryption, snapshot, DR ,etc.).   But remember that there is no list of SAS officially supported CSI drivers...what we have in the documentation today is more a list of CSI drivers that SHOULD NOT be used for specific volumes. The cost efficiency also depends on the kind of volumes for which you would use the CSI driver. For example Azure Files CSI driver, which is supported with the premium SKU (NFS support) is probably not cost efficient for all the Viya platform dynamically created volumes, but it may be cost efficient for the SASDATA or CASDATA static volumes...   Each volume comes with its own specific requirements (access mode, capacity, latency, security, POSIX compliance, data protection, etc.…). Having multiple storage classes is a recommended practice as it allows to associate the right kind of storage technology and device to each Persistent volume.   The topic of storage in Kubernetes is complex and the technologies are evolving rapidly…Some of the issues or limitations discussed in this post may disappear in the future, but other issues could arise when untested scenarios are implemented…   Here are two examples :   It has been recently discovered that using Azure NetApp Ultra (NFS based storage) for the internal crunchy PostgreSQL was causing issues during the upgrade operations to the new PostgreSQL 16 version. The current recommendation is to use block storage for the internal crunchy PostgreSQL volumes. The Kubernetes sig-storage group recommends to move from the NFS subdir external- provisioner (the project is not maintained) to the NFS CSI driver. SAS is currently evaluating alternatives for the viya4-deployment GitHub project.   Finally while it would be impossible to test, support and list all possible CSI drivers implementation with the SAS Viya platform, it is assumed that the CSI drivers can be used for most of the SAS Viya volumes in general. However in case new problems or limits with specific CSI drivers are discovered in the field and reported, they can be reproduced and assessed by SAS and potentially documented to increase awareness across the SAS Technical community.   That’s it for today! Thanks for reading!!!   Find more articles from SAS Global Enablement and Learning here. ... View more

This post is a follow-up on a previous blog where we explained how recent changes in Control Group v2 management in Kubernetes have impacted the stability of the CAS server in the SAS Viya platform.   In the latest post, we discussed some workarounds, but with the latest versions of SAS Viya there is now a new feature in the SAS software to avoid this issue in most of the cases.   This new feature is called "Backing Store for CAS Memory Allocations" and this post explains how it works and how it can be setup to mitigate the CAS OOM (Out of Memory) issues impact in the Viya environment.   TL;DR   This post contains a lot of details and is fairly technical, so if you are not about to read the whole article, here is a little “TL;DR” summary 😊   A new CAS feature has been introduced from 2024.09 to limit the CAS memory allocation. If it is enabled, when a CAS action uses more memory than a defined limit, it is interrupted with a failure and a meaningful error message, but the CAS session remains active and the Linux OOM killer is not triggered (which avoids the situation where the OOM killer performs a “group” kill of every process in the CAS container, leading to CAS service interruption). It greatly improves the stability and reliability of the CAS Server service.   The technical mechanism used behind the scene to limit the CAS memory allocation (before the OOM kicks in) is to back the memory allocation onto a special Kubernetes volume (emptyDir of “memory” medium – RAM-backed filesystem) – for which we can set a SizeLimit value. When this value (usually set to 80% of the CAS container's memory limit) is exceeded, the CAS action is interrupted but the CAS session and services remain available for the end users.   The "Backing Store for CAS Memory Allocations" feature was first introduced with SAS Viya stable 2024.09 (which means that it is also included in the latest SAS Viya LTS 2024.09) and instructions where provided to patch or edit the CASDeployment Custom resource to enable it.   Then with the new stable 2024.11 version, new PatchTransformers were added, so it is now possible to manually enable the backing store for CAS memory as part of the initial deployment of SAS Viya (with the proper configuration in the kustomization.yaml file).   The current plan is make the backing store become the default in the next versions (in the first months of 2025).   CAS Memory allocation and current issue   We know that the CAS_DISK_CACHE can be used to cache CAS tables on disk (as SASHDAT files) using memory-mapped file. CAS can leverage the CAS_DISK_CACHE to quickly and efficiently "swap in", "swap out" memory blocks as needed and hence holding a volume of data that is higher than the amount of physical memory on the machine (Read these nice posts from Rob Colum and Nicolas Robert if you want to know more about how and when the CAS Disk Cache is used)   But in CAS, an action also allocates "resident" (physical) memory in order to process a table, maybe doing things like creating computed columns or views that required RSS memory to hold rows/columns.   For this part of the Analytics processing, the CAS_DISK_CACHE is not used; CAS allocates memory using the Threaded Kernel (TK) and Memory is backed using the "mmap" system call with the MAP_ANONYMOUS flag. With this flag, the pages are only backed by either real memory or the paging files (with MAP_ANONYMOUS the mapping is not backed by any file).   However, in most Kubernetes system, containers do not have a configured paging file, and CAS can only use real memory…It increases the risks of the CAS session process (running in containers) to be terminated by the OOM killer - as soon as the container's defined memory limit is exceeded.   Having session processes forcibly killed is not a great experience for the end users.   As an aggravating factor, we know (from my previous blog) that - from recent Kubernetes versions (1.28) – when a single CAS session process is killed by the OOM killer, all the other processes in the same cgroup(v2) are also killed, which lead the whole CAS server (SMP) or individual CAS worker(s) (MPP) to go down, impacting the CAS service.   While it is understandable that, from time to time a CAS action could fail (because the system is not equipped with enough memory to run it), it is hardly acceptable that the failure of a single individual CAS action causes the entire deployment of CAS to restart and interrupt the functionality of CAS...   That's the problem that is addressed by this new "Backing store for CAS Memory allocations" feature.   What is this new CAS backing store ?   It is now possible to enable a "backing store" to support CAS memory allocations and prevent the whole system being impacted by the fact that a single action requires more memory than what has been made available for the CAS container.   With this feature, the CAS Threaded Kernel (TK)  is informed that the files in a specified directory can be used to back up most of the memory allocations.   The TK_BACKING_STORE_DIR environment variable is set in the CAS container and points to a specific path where the Threaded Kernel stores the mapping files (ex: /cas/tkMemory ).   The path is mounted in the pod and mapped to a Kubernetes emptyDir  volume.   As noted in the official Kubernetes documentation:  "The emptyDir is created when a pod is assigned to a node and is initially empty. When a Pod is removed from a node for any reason, the data in the emptyDir is deleted permanently".   By default, the content of emptyDir volumes is stored on the node’s root disk (typically under /var/lib/kubelet ). However, it is possible to set the emptyDir.medium field to "Memory", the documentation explains that Kubernetes mounts a tmpfs filesystem (RAM-backed virtual filesystem) instead of using the disk.   A size limit can then be specified to limits the capacity of the emptyDir volume. That's how we can control/restrict the memory consumption before it is too late...   In the CASDeployment custom resource specification, it would look like that :     OK but how does that help ?   The backing store for CAS memory allocations relies on the SizeLimit parameter as THE way to set a control on the maximum amount of memory that a CAS action could use, without involving the OOM Killer.   With the backing store feature enabled: when a CAS action is submitted to CAS, the action starts to map the memory files in the emptyDir and if the size of the mmapped files exceeds the defined SizeLimit (for example 24GB in the previous example), then there will be an individual failure of the CAS action.   The error message shown to the end users (for example in SAS Studio or any other CAS client) will inform them that the CAS action has failed because it ran out of memory.   It will look like this:     Or like that:     With this new behavior, the problem is addressed before involving the OOM killer and seeing it terminate all the nearby CAS processes. It helps ensure that the CAS server and even the CAS session itself suffer less impact from large-memory tasks, allowing the end-users to continue to work with CAS.   The “fail fast” principle is used here to prevent the OOM killer issue and improve the CAS overall stability.   Finally, the release of the files in the backing store follows the same pattern as releasing memory back to the operating system without this feature enabled. When the CAS session process terminates, all of the files used to back the TK memory are released (which ensure there will be no leaked resources).   Additional considerations   Specific memory allocations that are not constrained   While most of the use cases benefit from the “Backing Store for CAS Memory Allocations” feature to prevent OOM kills, there are some exceptions. Not all memory consumption is constrained by the size of the backing store. These include:   Memory allocated by the main CAS processes on behalf of actions (but it should be a reasonably small amount.) Memory allocated by third party code (database access drivers provided by the database vendor or any open-source code that an action consumes). Memory in other programming languages - when an action spawns processes or threads in other programming languages (such as Python or Java) to implement the action's logic   So, for example, Python or Java code running inside the CAS session does not go through the CAS TK Memory allocation and as such will not be subject to the memory control via the TK backing store.   Requires a careful choice of SizeLimit   The SizeLimit value defines the maximum amount of memory that can be used by CAS actions before causing an "out of memory" failure.   For the CAS backing store to be efficient, it is important that this threshold is lower than the threshold that would trigger the OOM killer…the goal here is to ensure that the session fails and display an “out of memory” error message to the end-user running this action, before the CAS container memory limit is exceeded and all processes are being killed.   Testing, with the default CAS auto-resources configuration, has shown that setting the backing store to on the node seems to be effective in preventing OOM kills in most situations. In situations where the CAS resources request and limit are manually set (usually to places more than one CAS pod on a node), it is recommended to use a similar fraction (80%) of the container memory limit.   Coordination with the CAS resources management configuration.   In CAS, it is possible to implement resource management through policies that are created with the Viya CLI. You can create up to 5 five "priority-level" policies per CAS server that is used to place space quotas on table data. Each “priority-level-n” is associated to a group of users.  If you enable this kind of configuration, it is also possible to align the backing store configuration to define distinct TK Memory allocation limits for each policy.   Other ways to prevent the OOM “group-kill” issue   Finally, note that SAS R&D is also currently researching additional ways to prevent the OOM killer from terminating CAS processes. One alternative workaround is based on the use of a Kubernetes Daemonset to overwrite the cgroups configuration files created by Kubernetes. The issue with the OOM "group-kill" change in Kubernetes is not specific to CAS, other applications have been impacted and a pull request to allow configuration of the "group OOM kill" behavior should be included in Kubernetes 1.32.   How to implement it ?   When the "Backing Store for CAS Memory Allocations" capability was introduced for the first time in SAS Viya, the official way to enable it was to patch the CASDeployment Custom resource deployment or to directly edit the CASDeployment Custom resource.   But since version 2024.11, there is now an official Kustomize Patch Transformer to apply the changes. A new "Configure a Backing Store for Memory Allocations" paragraph has been added in the "Optional Customization" section of the official documentation.   There are actually 4 distinct transformers to apply depending on the CAS configuration that is in place.   The first one cas-enable-default-backing-store.yaml should be used when the CAS autoresourcing is enabled (which is the CAS default configuration) OR if you have manually set the CAS resources request and limits. You might have done that to allow several CAS pods to run on the same node, maybe because you need to optimize the infrastructure costs or because you are deploying SAS Viya in a Bare metal environment on a small set of large physical machines as discussed there) In these cases, there is no need to set the size for the backing store ( SizeLimit ) as it is automatically set by the CAS operator to 80% of the CAS container memory limit (when it is manually set). The second one , cas-enable-backing-store.yaml lets you specify the SizeLimit . It should be used when: The CAS container doesn't have a memory limit OR You don't want to use the 80% default value (A customer might decide something like "I want more memory dedicated to the file cache" or "I'm using the CAS Gateway and it uses lots of memory in Python, so, I want a smaller limit on CAS TK Memory"). Then when CAS resource management policies have been enabled with different priority groups, we can have separate backing stores for each resource management priority group and set different SizeLimit values for the priority groups that correspond to group of users. Two transformers are provided as examples (one with 5 priority groups and another with just one).   As usual, once you have copied and eventually adjusted the values in the transformer YAML file, don't forget to reference it in the "Transformers" section of your main kustomization.yaml file.   Note that the Kubernetes manifest, generated from the Kustomize build with the updated reference, must be re-applied and that the CAS Server should be restarted too to pick up the configuration change.   Here is an example of what you would see in the CAS pod specification when CAS auto-resource is enabled (no use of CAS Resource Management policies) and when the backing store is enabled with the default transformer ( cas-enable-default-backing-store.yml ).   The first screenshot shows the effect of the CAS auto-resource configuration on an 8 CPU/64GB machine. The CAS container requests and limits are automatically computed by the CAS operator.     Then you can see below the effect of enabling the Backing Store in the CASDeployment CR:     Conclusion   The previous posts on this topic talked about switching from cgroupsv2 to cgroupsv1 as a workaround to avoid the OOM Killer issue in Kubernetes…However the Kubernetes community has decided to move cgroupsv1 into maintenance mode for Kubernetes 1.31, so removing cgroups2 as a solution may be poorly received... that’s another reason to opt for this new "Backing Store for CAS Memory Allocations" feature instead.   Finally, nothing really prevents a specific CAS action from running out of memory at some point. It’s always a possibility… maybe because the code has not been optimized or the dataset size is too large, or simply because running this action on this amount of data requires more memory than physically available in the infrastructure. However, with this new "backing store" feature, this type of situation is reported as an "out of memory" condition and only affect the individual user's session, as opposed to all sessions running in a CAS pod being arbitrarily killed.   That’s why this change is really improving the overall reliability and availability of the CAS Server.   Video Bonus   Now, as a little reward to your perseverance in this post reading 😉, you can find below a short 3 minutes video to quickly demonstrate the benefit of using the "Backing Store for CAS Memory Allocations".       Thanks for reading !     Find more articles from SAS Global Enablement and Learning here. ... View more

With the July version of SAS Viya (stable 2024.07), a nice change has been made in the section of the SAS Viya Operations guide that covers the customizations of a deployment.   Before this version, the "Common Customizations" page just provided a long list of all the customizations (outside of product-specific customizations) that should be considered for a deployment.   But now this "Common Customizations" content has been divided in two parts :   Required customizations Optional customizations   While many customizations are optional, a few of them should be decided in advance (and they sometimes can’t be made without specific pre-requisites in place).   This documentation change helps to identify some of the key decisions that MUST be made (in a way that corresponds to the customer needs or requirements) before starting any new Viya deployment.   So, I thought it would be great opportunity to revisit and discuss these architecture decisions and deployment pre-requisites through the review of these six "Required customizations" and explain why they would form the basis of a Viya environment Architecture design document.   As we often say during our GEL workshops:   "The deployment of SAS Viya by itself is usually a matter of hours (or even minutes). However, the discussions, decisions and preparation leading to that moment can take days or even weeks..."    TLS Mode Host and port for Cluster Ingress CAS topology and workload strategy Internal or external PostgreSQL?  Internal or external OpenSearch? StorageClass to use for RWX PVC ? "Recap" table   Note: In this post we don't explain HOW these customizations are implemented as it is already nicely explained in our documentation or in the README files.     TLS mode ?   TLS (Transport Layer Security) allows to encrypt data in motion and is used in the SAS Viya platform to secure communications with internal and external components.   3 options are possible : "Full stack TLS", "Front door TLS", or "no TLS".   By default (using the kustomization.yaml example file provided in the documentation), your Viya environment will be automatically configured with "Full stack TLS". It means that all the microservices and web applications implement HTTPS in their communications. In this mode, all network connections from outside and even inside the Viya environment are secured.   With "Front door TLS", only the components that are intended to accept connections from outside the Kubernetes cluster (such as the Ingress controller and CAS server) are configured to only accept encrypted network connections.   In both cases, the customer can use the default certificate issuer (openSSL) or use another certificate issuer (cert-manager). The customer can also provide their own certificates if they prefer to secure the ingress.   The customer should always be advised to use TLS to secure the network traffic ("No TLS" should never be used as the traffic would be in clear text and easily intercepted by anyone). However, most of the time, customers are very sensitive about  security and opt for the "Full stack TLS option".   Reference :   SAS Doc : https://go.documentation.sas.com/doc/en/itopscdc/default/itopssr/n18dxcft030ccfn1ws2mujww1fav.htm#n0w1os9ufnn7xwn1an3qbhxjxze9     Host and port for Cluster Ingress As part of the deployment, you need to provide a small set of parameter’s values in your initial kustomization.yaml file.   In the officially documented example, they are represented as place holders under this form:  {{ PLACE_HOLDER }}. One of them is the {{ NAME-OF-INGRESS-HOST }}   How can you find this value ?   I wrote a couple of posts (part 1 and 2) to answer this question, here is an extract :   {{ NAME_OF_INGRESS_HOST }} corresponds to the URL base that the customer wants to expose to the SAS applications users. The {{ PORT }} parameter is optional, as the HTTP schema is usually determined by the start of the url (“http” or “https”). However, the important thing to understand here is that {{ NAME_OF_INGRESS_HOST }} also correspond to a DNS name that must be associated to the external IP address exposed by the Ingress controller.   When you install the Ingress controller in a Cloud environment, an external IP address is automatically provisioned by the Cloud provider (You can see this address in the EXTERNAL-IP column when you describe the ingress controller service).   A DNS name then needs to be mapped to the ingress public IP by the site’s IT team configuring their DNS services. The exact components of the FQDN will likely be shaped by the site’s DNS standards, for example: https://viya.your_dept.your_site.com/ and Viya accommodates that.   But there must be a DNS entry associated with this IP address. It is a key pre-requisites that should not be overlooked during the deployment planning process.     CAS topology and workload strategy   As the main analytics and data management engine of the SAS Viya platform, CAS is a cornerstone of the environment. A CAS Server is required at the start-up time of Viya : it is a pre-requisite to have at least one CAS Server defined otherwise the Viya platform will not work for any user (whether the users plan on using CAS or not).   Some choices must be made at a minimum for the CAS topology and the CAS workload strategy.   While, by default (using the kustomization.yaml example file provided in the documentation) the CAS server consists of a Single-node (SMP : for Symmetric Multi-Processing), it is possible to opt for a Multi-node CAS server (MPP: for Massively Parallel Processing) to spread the Analytics processing and data across multiple CAS nodes.   This choice is mostly driven by the amount of data to process (remember that with CAS, the data are loaded into memory to be processed by CAS, and a single machine may not have enough RAM to hold them) and an official sizing is a key input in this decision.   Note that more sophisticated CAS topologies are also possible such as:   Adding a secondary CAS controller into a CAS MPP deployment (to improve the CAS Server availability) Adding new CAS server(s), for example to isolate and specialize the different kind of workloads (ex: 1 CAS server for the reports, 1 other for the Visual forecasting, etc…)   Now, regarding the CAS workload strategies, several options are also possible. The SAS general recommendation is to choose the default "auto-resourcing" configuration. This approach ensures that each CAS instance (Controllers and workers) is running on a dedicated host and can leverage all its capacity (in a similar way as with a Viya 3.x deployment). While this is ideal for maximum performance, it might be overkill for dev/test environments.     Note that options are also available to have a differentiated workload classes for the CAS controller(s) and the workers. The goal is to optimize the costs by using distinct node pools and smaller machines for the CAS controller(s) that usually don’t require as much CPU and memory as the workers.   The CAS topology and workload management topics are a key part of the Architecture design and should be discussed with your site’s IT team to make sure the associated decisions meets your site’s needs and constraints.   Another exciting benefit of running Viya 4 as containerized workloads in a Kubernetes cluster on virtualized hardware is that changing your deployment topology can be effected simply by submitting configuration changes programmatically. This should give your site’s administrators confidence to try new workload configurations in pursuit of best performance, efficiency, and cost.   If these concepts are not clear to you, the  SAS Viya Architecture course has dedicated modules to cover them.   References :   CAS Server Topology Changes And CAS Table Balancing:  https://communities.sas.com/t5/SAS-Communities-Library/CAS-Server-Topology-Changes-And-CAS-Table-Balancing/ta-p/903549       Internal or external PostgreSQL?   The SAS Viya platform relies on an instance of PostgreSQL for its SAS Infrastructure Data Server. This instance can be either internal or external.   By default, SAS Viya is installed with the internal PostgreSQL instance that relies on Crunchy data server which is shipped with the SAS order. But the SAS Viya platform can also be configured to use an external instance of PostgreSQL such as the Azure PostgreSQL Flexible Server.   This post, from my colleague Edoardo Riva, lists some of the considerations associated to the choice of an internal versus external instance.   It is worth noting that many SAS Solutions also require an additional PostgreSQL server instance for the Common Data Store (aka CDS), which requires its own PostgreSQL instance. When deploying a SAS Solution, the location of the Common Data Store instance (internal or external) - as well as any additional potential instances, for Airflow for example - must also be discussed and addressed.     Internal or external OpenSearch?   The SAS Viya platform also relies on an instance of OpenSearch to create and store search indices (for example for the SAS Information Catalog and Visual Investigator).   Starting with 2023.03 it is possible to use an external instance of OpenSearch.   Once again, you can find a nice post from Edoardo, comparing the use of an internal versus an external instance of OpenSearch.   The decision between : letting the SAS Viya deployment install and manage OpenSearch (default) versus using an existing external instance, must be made before the deployment.     StorageClass to use for RWX PVC ?   The last item of the "Required customizations" covers another key decision : which kind of storage to use for the Viya component that requires a ReadWriteMany (aka RWX) storage class.   As a reminder, the RWX storage type can be seen as the equivalent of a shared file system in the bare OS world. It corresponds to a kind of storage that supports concurrent access by multiple pods running on different nodes.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Since the SAS Viya PersistentVolumeClaims don’t specify a StorageClassName, the default storage class specified by your Kubernetes configuration is used.   This  is convenient because most of the Managed Kubernetes platforms that we support in the Cloud come with a default storage class.  On the other hand, it’s not often the ideal choice because the usual default storage class is backed up by managed disks, which has only RWO access(standard-rwo in GCP, managed-csi in Azure, etc..).   While many of the SAS Viya platform volumes only need storage with RWO access, some of them require RWX storage…As a consequence, if you don’t do anything regarding the storage class when deploying for example in AKS, GKE or EKS, your deployment will fail.   So, for them we need to provide an alternative RWX storage class and make the associated configuration changes, as in the example below :     See how the annotationSelector is used to target the PVC that requires this RWX access.   So, the associated architecture decision is: "What storage class should I use for the Viya components that require RWX access ?"   Several options are possible, they depends on the Kubernetes platform. You can have a look at this table from the Kubernetes documentation to get a first list of RWX compatible volumes plug-ins, but options mostly depends on the Cloud provider.   Note that if you are using the viya4-iac-x (IaC) or  viya4-deployment (aka DaC) tools, you don’t have to tweak the Kustomize Transformer files : the DaC automatically configures a new RWX storage class and make it the default. In this case, the new storage class is backed up by an NFS Client provisioner pointing to either a Cloud Managed File services (such as Azure NetApp, GCP File store or EFS in AWS) or to an NFS server running on a Cloud VM (depending on your IaC storage choice).   The available DaC options may correspond to the customer’s needs and requirements. Otherwise, it is required to identify a RWX storage class and make the configuration changes so it is used for the Viya volumes that requires it. Note that you can also use distinct RWO and RWX storage classes for the different volumes required by the Viya platform.     Other key decisions   Finally, it is important to remember that there several  key decisions that are NOT reflected by a specific requirement listed in the new "Required Customization" section.   Some of them (Kubernetes platform, dedicated or shared cluster, etc…) are critical and early to decide, with downstream implications on the other aspects of the architecture and deployment (networking, storage, etc…).   While it is probably not an exhaustive list, the decisions listed below correspond to the customer’s specific requirements or constraints and must also be made during the design phase (before the deployment) :   Kubernetes Platform ? AKS? GKE? EKS ? OpenShift ? other ? Dedicated or shared Kubernetes cluster ? Cadence (LTS or stable) ? Storage : used for the Viya platform components and for things like input/output data, user’s home directory, etc…as an example, for production environments, block storage (instead of NFS based storage) is strongly recommended for specific SAS Viya components (OpenSearch, Redis and RabbitMQ) volumes ! Network : kubelet or CNI ? (Note that the question is not relevant for deployment of SAS Viya on Upstream Opensource Kubernetes cluster, where only the Calico CNI is supported.) Identity provider and authentication mechanism : various options are available for the integration with the Identity provider (LDAP or SCIM) and for the authentication.   We have added this decisions into our recap table below.     Recap and advice   The table below lists the key decisions that should be made before performing the initial deployment with their default value, pre-requisites and discuss whether they could be changed after the initial deployment.     Decision Default value Pre-requisites Can be changed  after the initial deployment? TLS mode “Full-stack” TLS Custom certificates (in .pem format) and the associated private key must be provided if the customer wants to use their own certificates for the ingress (instead of the default ones). Yes. The TLS mode can be changed afterwards, however it is recommended to agree on a TLS mode before the deployment to avoid any outage. Ingress host name and DNS None The customer needs to associate a DNS name to the Ingress controller service external IP , available Yes. However if the  INGRESS_HOST is wrong, nobody will be able to login to the deployed environment from the outside. CAS Topology and Resource Management strategy SMP, CAS-autoresourcing None Yes. The topology and workload placement strategy can be changed but it will impact the deployment and users (outage and need to reload the data). Internal or External PostgreSQL instance(s)? Internal None with the internal instance. For external: extensions, number of connection, pre-created SharedServices database, etc…see this page Yes but only in one way. SAS has created the PostgreSQL Data Transfer Tool for the SAS Viya Platform to move content from the internal instance of PostgreSQL to an external instance. But it is not possible to change from external to internal. Internal or External OpenSearch instance? Internal   No. Be careful, after the deployment has completed, you cannot modify your deployment to use a different OpenSearch instance. RWX Storage class to use for the SAS Viya Persistent Volumes ? There is usually no default RWX storage class available in the cluster. But the IaC tool for opensource Kubernetes and the DaC tool creates one for the SAS Viya volumes. A RWX storage class must be made available otherwise the deployment will fail. Maybe but not easily…while some volumes can be rebuilt from scratch, other may hold user’s data that needs to be migrated to new storage devices. Kubernetes platform ? None A supported Kubernetes platform should me made available with cluster-wide privileges No. If the Kubernetes platform is changed, SAS Viya must be redeployed (however a full-system migration could be done from one Cloud provider to another). Dedicated or shared Kubernetes cluster ? NA When multiple Viya versions are sharing the same cluster, there are some caveats to be aware of. Keeping all the Viya environments at the same version is strongly recommended. See above. LTS or stable cadence ? None To remain fully supported an update of the version N of SAS Viya must be made before the version N+4 is released. It means that customers on the stable cadence must update their Viya at least version every 3 to 4 months to avoid to be in limited support. Yes it is possible to change the cadence (under certain conditions). Storage strategy for the platform data and the user’s data NA Storage classes or PV types to use for the Viya platform volumes as well as input/output data must be made available Possible but requires tests, backup, outage and migration of the content. Network type ? kubelet ? CNI As of today, kubenet is usually the default for Managed Kubernetes Services in the cloud. In some cases (SAS with SingleStore, deployment with open-source Kubernetes) a CNI plugin is required. So always check the latest documentation version for requirements. Changing from one CNI implementation to another may be possible depending on the Cloud provider but requires tests, backup, outage and migration of the content. Refer to the specific cloud provider’s documentation. Identity provider and authentication mechanism NA The requirements depends on the type of identity provider and authentication, see the official documentation The identity provider cannot be changed for an “in-use” system. authentication mechanism can be changed in some cases but could require an outage and rebuild and re-apply the deployment manifests   My final advice would be to make sure that you have captured at minima the architecture decisions listed in this table (and prepared the associated pre-requisites) before starting your SAS Viya deployment process.   Also keep in mind that some post-deployment tasks are usually required as well before users can start to access the SAS Viya applications. They also depend on the offering and products that are deployed. See this page in the official documentation for a list of post-deployment steps.   Hope that helped ! Thanks for reading 😊     Find more articles from SAS Global Enablement and Learning here. ... View more