With the 2024.07 release, SAS Mirror Manager introduced the ability to verify the integrity of SAS Viya container images. Then in 2024.10, it was improved further.    This feature provides the ability to check the Viya images authenticity and integrity, which is a critical security process for establishing trust with our customers deploying SAS Viya.   As noted in the official documentation, “SAS provides container image signatures for all official product images. These signatures can be used with SAS Mirror Manager and other tools in order to verify that images originated from SAS.”   In this post we’ll discuss the details of the Viya container image verification and see some examples.     What has changed?   SAS Mirror Manager verifies image signatures   Starting with 2024.07, SAS Mirror Manager verifies image signatures at the time when images are downloaded from SAS container repositories.   As illustrated in the diagram below, it means that each time you run the SAS Mirror Manager tool to pull the SAS Viya images from the official SAS container registry (cr.sas.com), a check of the images signatures is automatically performed. The public key included in the order’s certificates archive is used.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     SAS Mirror Manager mirror the image signature   Since the version 2024.10 of SAS Viya, SAS Mirror Manager can also mirror the image signature to some destination container registries (such as Azure Container Registry, Elastic Container Registry, and so on).   In the situation where you have made your Viya container images available into a private container registry (such as Azure Container Registry or JFrog), then you can also manually verify the associated signatures.   You can use an opensource tool called “Cosign” to perform the verification.       Requirements for image verification in a private registry   OCI version 1.1 SAS relies on the OCI version 1.1 specification for the SAS Mirror Manager image signature implementation. Signatures can be mirrored into any Container registry implementation that conforms to this specification.   As of today, the following Container Registry implementations support this version of the specification:   Azure Container registry Amazon Elastic Container Registry Google Artifact Registry   The image signing public key The signing key was added to new orders after May 2024, so older orders that were simply renewed will not have it in the corresponding certs.zip files. In the short term, if you are using an older order and want to verify the signatures of the images mirrored into a private registry, then current solution would be to request a new order. If a signing key is associated to your order, you should see a file named cosign_release_2023.pub when you unzip the order's certificate archive ("-certs.zip" file).   The Cosign open-source verification utility The Cosign source code and binaries can be found on the Cosign GitHub project.  You can install the cosign utility on many operating systems.   For example, on a RedHat Linux based operating system (RedHat Enterprise Linux, CentOS, Rocky Linux), you could simply run the commands below to download and install the RPM package:       Example   The SAS documentation provides the steps to perform the signature verification, and if you visit the latest version of the SAS Viya: Deployment on Google Kubernetes Engine hands-on (soon to be published in learn.sas.com), you can also find an example on how to do it against SAS Viya images that have been mirrored into the Google Artifact Registry.   Here are the steps :   Create the OAuth token to access the Google Artifact Registry When we run the Cosign tool, we need to provide a username and password to authenticate against the remote registry. With the Google Artifact registry, you can run a gcloud command to generate a short-lived OAuth access token to authenticate.     Install Cosign Download and use the appropriate package depending on your operating system. In our example we are running Cosign on Rocky Linux 9, so we simply download and install the RPM package (see the example above). Extract the image signing public key: You can simply use the unzip command to extract the public key from the order’s -certs.zip file. Verify the image signature with Cosign: In the example below, we collect the sas-logon-app image name, with its exact path in the registry and the associated image tag. Then we run the Cosign command using the syntax that is provided in the SAS documentation.       Finally, when you run the verification command, you should see something like this:       It confirms that the container image signature has been verified with success!     Conclusion   That’s it for today. This approach to validate the source and integrity of SAS container images improves the security posture of our software. Sites deploying SAS Viya can reliably ensure that the software they're installing comes from SAS in its original form without modification or replacement by intermediaries.   Special thanks to Elliot Peele for his support in the write up of this post !     Find more articles from SAS Global Enablement and Learning here. ... View more

The Ingress Controller is a critical requirement and key component of the SAS Viya infrastructure. In the first part of the post (that you should read first 😊 ), I explained the concept of the Ingress Controller, made the distinction between "Ingress NGINX" and "NGINX Ingress", then discussed the version, how to install and configure ingress-nginx .   In this second and last part, I will focus on some changes that are needed to allow SAS Viya to work with the latest version of ingress-nginx (1.12 and later). We will also see that several CVEs (including a critical one) were recently fixed in the ingress-nginx project and what is the impact for SAS Viya.   Here is a little "ToC" table that lists the key points, for this second part of the post 😊   Ingress NGNIX 1.12 breaking changes Ingress Common Vulnerability Exposures (CVEs) Am I affected by this security issue? Remediation: Upgrade version Can I always upgrade to a patched ingress-nginx version to fix these security issues? The NSG rule deletion issue Before ingress-nginx 1.12 ingress-nginx 1.12 and after Solution Conclusion     Ingress NGNIX 1.12 breaking changes   Quite recently, a note has been added in the documentation in case you were using ingress-nginx 1.12.x and later for your Kubernetes cluster.   Some default values have been changed beginning with 1.12 and the new default values could cause problems and failures to deploy SAS Viya. (Remember that if you plan to deploy SAS Viya in Kubernetes 1.31, ingress-nginx 1.12.x or later is mandatory).   Some additional configuration of the ingress-nginx controller is required :   For all Viya versions, you’ll need to set the value for annotations-risk-level to Critical If you deploy SAS Viya LTS 2024.09, or SAS Viya Stable 2024.10 or 2024.11 the strict-validate-path-type parameter must be set to false in the ConfigMap for the ingress-nginx controller in your Kubernetes cluster (when ingress-nginx 1.12.x or later is used). From stable 2024.12, SAS Viya works with the strict path validation that is enabled by default (strict-validate-path-type set to true).   The table below provides the commands to patch the ConfigMap depending on the SAS Viya version:   Viya Version Commands Any stable of LTS version before stable 2024.12 kubectl patch cm ingress-nginx-controller -n name-of-namespace -p '{"data":{"annotations-risk-level":"Critical", "strict-validate-path-type":"false"}}'; From  2024.12 kubectl patch cm ingress-nginx-controller -n name-of-namespace -p '{"data":{"annotations-risk-level":"Critical"}}';   Note : If the strict-validate-path-type configuration was modified for a previous SAS Viya platform deployment running ingress-nginx 1.12.x, it can be reverted with:   kubectl patch cm ingress-nginx-controller -n name-of-namespace -p '{"data":{"strict-validate-path-type":"true"}}';     Ingress Common Vulnerability Exposures (CVEs)   Many versions of ingress-nginx are affected by several security vulnerabilities that could make it easy for attackers to take over your Kubernetes cluster.   See the Kubernetes announcement related to these vulnerabilities: Ingress-nginx CVE-2025-1974: What You Need to Know.   As a consequence, your SAS Viya platform deployment might be exposed to this vulnerability.   This security issue has the following associated CVEs:   CVE-2025-1974 (Critical) CVE-2025-1098 (High) CVE-2025-1097 (High) CVE-2025-24514 (High) CVE-2025-24513 (Medium)   The best and easiest remedy is to upgrade to the new patched release of ingress-nginx .     Am I affected by this security issue ?   You can run this command to get the ingress-nginx version that you are running in your cluster:   kubectl get pods -A -l app.kubernetes.io/name=ingress-nginx -o jsonpath="{.items[*].spec.containers[?(@.name=='controller')].image}"   You should see something like this :   registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32c   Look at the version after the "controller" word, in this example we are running ingress-nginx 1.12.1.     Remediation: Upgrade version   If, for your customer environment, you obtain a version listed below, SAS strongly recommends that you upgrade to a patched version of ingress-nginx (supported by the Kubernetes version on which your cluster is running).      The SAS Technical Support has published an article for the customers with all the required guidelines, SAS® Viya® platform guidance for Kubernetes Ingress-NGINX Controller CVEs.     Can I always upgrade to a patched ingress-nginx version to fix these security issues ?   Customers currently running Viya LTS 2023.03 cannot upgrade to an ingress-nginx version that includes the fixes for the CVEs. SAS Viya LTS 2023.03 supports K8s 1.23-1.25 but unfortunately the ingress-nginx maintainers did not provide a fix for the CVEs that is supported on Kubernetes versions older than K8s 1.26…This is outside of SAS's control.   SAS recommendation is to upgrade to a newer Viya LTS/Kubernetes version combination. SAS Viya LTS 2023.10, LTS 2024.03, LTS 2024.09 will all work with the CVE patched version of Ingress NGINX.   Note: Viya LTS 2023.03 is EOL in May 2025 (As soon as LTS 2025.03 will be released - which is currently planned for the 15th of May 2025), so the update to a newer version of  SAS Viya is inevitable to remain in standard support.   In the latest DaC tool (viya4-deployment) version (8.2.0), the default version of ingress-nginx delivered has been changed to one that contains the fix for the CVEs.     The NSG rule deletion issue   Finally, there is something else that changed with ingress-nginx 1.12. It affects sites running SAS Viya in Azure where firewall rules must be added to allow additional inbound access to the Ingress external address (for example to allow access from Microsoft Entra ID when SCIM is configured).     Before ingress-nginx 1.12   Before 1.12, if you had to allow both inbound access to the Ingress Controller:   from Azure Entra ID (for the SCIM Client), AND from a range of IP address (corresponding to your clients IP addresses),   The proven practice was to :   Set the azure-allowed-service-tags annotation in your ingress-nginx controller definition (as shown below), for the inbound access from Azure Entra ID.     Then create your own Azure NSGs (Network Security Groups) - in the Azure portal or with the az CLI - to allow access from a specific range of client IP addresses…instead of using the LoadBalancerSourceRange specification in your ingress-nginx controller definition.   In the example below, you see the "manually created" NSG inbound security rule with a priority of 100 : it allows the access on port 80 and 443 from a range of IP addresses that correspond to our end-user’s IP addresses. But you also see the a rule generated (with a random name) from our azure-allowed-service-tag Ingress NGINX annotation which allow Azure Entra ID ("AzureActiveDirectory") to connect to our SAS Viya environment (for the SCIM push).     These guidelines are coming from this post written by Stuart Rogers.   In the post, Stuart explains in details, the reason to use "manually created" NSGs instead of the Ingress NGINX  controller.service.loadBalancerSourceRanges specification …basically loadBalancerSourceRanges uses IPTABLES to block any connection from outside of the defined range – hence preventing the access from Azure Entra ID !   This kind of setup has worked for several years. However it was recently discovered that this setup would not work any longer with ingress-nginx 1.12 !     ingress-nginx 1.12 and after   What happens now, if you implement this setup with ingress-nginx 1.12 or later is that our "manually created" NSG rules is silently removed by Azure after some time  ! (1 to 3 days).   One of our colleague, at SAS, discovered the issue and, after some investigation, found out (from the ingress-nginx-controller logs) that Azure/AKS was periodically "reconciling" the ingress controller firewall rules with the NSG, with the changes being executed by the AKS service principal :   Warning ConflictConfiguration 3m15s azure-cloud-provider Please use annotation service.beta.kubernetes.io/azure-allowed-ip-ranges instead of spec.loadBalancerSourceRanges while using service.beta.kubernetes.io/azure-allowed-service-tags annotation at the same time.     Solution   As suggested in the log message, the solution is to configure the ingress-nginx-controller service to use the   azure-allowed-ip-ranges annotation instead of creating the NSG rules directly in Azure.   This change allows Azure to properly create the required inbound NSG rules for both Entra/SCIM AND end-user source ranges.   Our colleague provided an example on how to patch a "DaC" deployed environment to implement the solution :   kubectl -n ingress-nginx describe svc ingress-nginx-controller # add the "azure-allowed-service-tags" annotation to allow Azure Entra ID inbound access (SCIM) kubectl -n ingress-nginx patch svc ingress-nginx-controller --type='json' -p='[{"op": "add", "path": "/metadata/annotations/service.beta.kubernetes.io~1azure-allowed-service-tags", "value":"AzureActiveDirectory"}]' # remove the loadBalancerSourceRanges specification kubectl -n ingress-nginx patch svc ingress-nginx-controller --type=json -p='[{"op": "remove", "path": "/spec/loadBalancerSourceRanges"}]' # add the azure-allowed-ip-ranges annotation to allow a range of IP adresses to kubectl -n ingress-nginx patch svc ingress-nginx-controller --type='json' \ -p='[ { "op": "add", "path": "/metadata/annotations/service.beta.kubernetes.io~1azure-allowed-ip-ranges", "value": "109.xxx.xxx.xxx/27,149.xxx.0.0/16,194.xxx.xx.xxx/28,10.0.0.0/16,13.xx.xxx.xxx/32,192.xx.xx.0/24,48.xxx.xxx.xxx/32,10.xx.xx.xx/16" } ]'   But if you are not using the "DaC" tool (viya4-deployment) and want to install ingress-nginx on your own with Helm, you can use the script below (provided by Stuart Rogers) as a reference:   helm install ingress-nginx --namespace ingress-nginx \ --set controller.service.externalTrafficPolicy=Local \ --set controller.service.sessionAffinity=None \ --set controller.config.use-forwarded-headers="true" \ --set controller.autoscaling.enabled=true \ --set controller.autoscaling.minReplicas=2 \ --set controller.autoscaling.maxReplicas=5 \ --set controller.resources.requests.cpu=100m \ --set controller.resources.requests.memory=500Mi \ --set controller.autoscaling.targetCPUUtilizationPercentage=90 \ --set controller.autoscaling.targetMemoryUtilizationPercentage=90 \ --set-string controller.service.annotations."service\.beta\.kubernetes\.io/azure-allowed-service-tags"="AzureActiveDirectory" \ --set-string controller.service.annotations."service\.beta\.kubernetes\.io/azure-allowed-ip-ranges"="10.244.0.0/16, 149.xxx.xxx." \ --version 4.12.1 \ ingress-nginx/ingress-nginx kubectl wait -n ingress-nginx --for=condition=available --all deployments --timeout=60s # Changed according to https://documentation.sas.com/doc/en/itopscdc/v_060/dplyml0phy0dkr/n02w9l89l6b6ttn1wncv617wndp3.htm kubectl patch cm ingress-nginx-controller -n ingress-nginx -p '{"data":{"allow-snippet-annotations":"true", "annotation-value-word-blocklist":"load_module,lua_package,_by_lua,location,root,proxy_pass,serviceaccount,{,},\\"}}'; # Changes in 2025.02 if version_ge "$NGINX_Version" "4.12.0"; then # Changed according to https://documentation.sas.com/doc/en/itopscdc/default/dplyml0phy0dkr/p1deewvd9xtqiyn168adq6fu9msv.htm if version_ge "$CADENCE_VERSION" "2024.12"; then kubectl patch cm ingress-nginx-controller -n ingress-nginx -p '{"data":{"annotations-risk-level":"Critical"}}'; else kubectl patch cm ingress-nginx-controller -n ingress-nginx -p '{"data":{"annotations-risk-level":"Critical", "strict-validate-path-type":"false"}}'; fi fi   In this case, the NSGs for both Entra ID and the range of allowed client IPs would be automatically generated in Azure as shown below:       Conclusion   That’s it for today…If you have read the whole post (part 1 and part 2) you should now be ready to move to the new ingress-nginx version 1.11.5 or 1.12.1 or later ! 😊 Thanks to Heather Hewlett, Tony Gianni and Stuart Rogers for sharing their knowledge and experience with the ingress-nginx setup and configuration.     Find more articles from SAS Global Enablement and Learning here. ... View more

The Ingress Controller is a critical requirement and key component of the SAS Viya infrastructure. This component can be seen as the "front door" or "entry point" of your SAS Viya deployment. SAS currently (as of April 2025) only supports the Ingress NGINX Controller (aka ingress-nginx) - with the exception of OpenShift routes if you are deploying SAS Viya in the Red Hat OpenShift platform.   There were some recent changes and announcements related to the Ingress NGINX Controller, so I thought it could be a good time to list and explain them.   As there was a lot to discuss I have split this post in two parts.   Here is a little "ToC" table for this first part to help you to navigate through this post 😊   What is the Ingress Controller? Ingress NGINX <> NGINX Ingress …A little clarification on the Terminology Which ingress NGINX version should I use? How do I install ingress-nginx? Vulnerability Mitigation for the "allow-snippet-annotation"   What is the Ingress Controller ?   In a Kubernetes cluster, an "Ingress" is the object that makes your web (HTTP or HTTPS) service available from the outside. It lets you map traffic to different backends based on rules that you define in the Ingress definition.   But in order for the Ingress resources to work, the cluster must have an Ingress Controller running (usually inside its own Kubernetes namespace). There are many implementations of the Ingress Controller component (Cloud plugins or generic, based on different technologies such as HAProxy, Istio, Envoy, etc.).   The Ingress Controller acts as a reverse proxy and load balancer. Its main role is to read the rules defined in Ingress resources and route incoming traffic to the appropriate services.   Ingress NGINX <> NGINX Ingress…A little clarification on the Terminology   There are two types of nginx ingress controllers with VERY VERY similar product names ☹:   One from the community (Ingress NGINX) The second from F5 (NGINX Ingress)   They are really different and the only one we support is the one from the community.     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   SAS supports the Ingress NGINX Controller which is commonly called ingress-nginx.   So here is the link to the project: ingress-nginx  to avoid any further confusion!   It is important, because if you install the wrong one (NGINX Ingress), then the annotations used in the SAS Viya ingresses definitions will not work…   Which version should I install?   The SAS Viya official documentation has a dedicated section to answer this question.   If you look at the latest version of the documentation (2025.04 at the time of this write up), you can see that the SAS Viya platform supports ingress-nginx 1.9.6 and later. Another important requirement is that ingress-nginx snippet annotations setting must be enabled.   However we can also read : "Make sure that the version of ingress-nginx that you are using is compatible with your cloud provider, and check for updates."   So aside the minimum version requirement and the need to allow snippet annotations, what really drives the ingress-nginx version is the Kubernetes version.   The ingress-nginx GitHub page has a reference table that tells you which Kubernetes versions are supported for each ingress-nginx versions.   See below an extract from this table:     It means that, for example, if you are running Viya 2025.03 on Azure AKS 1.31, you need to deploy at least ingress-nginx with v1.12. The corresponding Helm chart version is also included, which will come in handy in the next section.   How do I install ingress-nginx ?   Since ingress-nginx is a system requirement for SAS Viya, it should be the responsibility of the customer to deploy it.   However, the customer may ask for your help or for a test environment you may have to deploy it by yourself. The instructions to deploy ingress-nginx are provided in the "SAS® Viya® System Requirements" course available on learn.sas.com (see the extract below).     Using Helm is the most common way to install ingress-nginx. You need to provide the Helm chart version as shown in the example above.   Another point of reference, if you are looking for the commands to install ingress-nginx is the viya4-deployment GitHub project (aka "DaC" – for Deployment As Code).   The DAC playbook has a "baseline" role that includes the setup of ingress-nginx. If you look at the corresponding codebase , you’ll notice that several rules are now used to determine the correct version to install and implement specific configuration (CVE mitigation, annotations, strict path validation,…) depending on the Kubernetes version, Viya version and Cloud provider…   So, even if you don’t use it to install SAS Viya itself, it could be a good idea to use only the "baseline" feature of the DaC tool to ensure that you install the correct ingress-nginx version with the required configurations.   Vulnerability Mitigation for the "allow-snippet-annotation"   Back in 2021, a security issue was discovered in ingress-nginx where a user, who can create or update ingress objects, can use the custom snippets feature to obtain all secrets in the cluster! (see CVE-2021-25742 for more details)   However, the SAS Viya platform requires custom snippets to function properly.   So for each supported LTS and stable Viya version there is a set of instructions provided in the SAS documentation to explain how to mitigate the security risk by using an annotation-value-word-blocklist annotation. The  annotation-value-word-blocklist contains the characters/words that are well known of being used to abuse Ingress configuration and must be blocked. When an annotation is detected with a value that matches one of the blocked bad words, the whole Ingress won't be configured (source).   Here is an example on how to patch the ingress-nginx configMap object to implement this mitigation :   kubectl patch cm ingress-nginx-controller -n ingress-nginx -p '{"data":{"allow-snippet-annotations":"true", "annotation-value-word-blocklist":"load_module,lua_package,_by_lua,location,root,proxy_pass,serviceaccount,{,},\\"}}';   SAS strongly recommends to implement this mitigation to limit the risk of attackers exploiting this vulnerability. That's all for this first part of the post !   Now that you know what is the Ingress NGINX controller, which version to install and how to install it, you are ready for the next part where we will discuss the changes brought by the new ingress-nginx version (1.12) as well as some new security patches recently released to address several CVEs (Common Vulnerabilities and Exposures).     Find more articles from SAS Global Enablement and Learning here. ... View more

Last month, our team was contacted, by a colleague in the field, to help with a customer request.  The customer was interested in discussing the PSS (Pod Security Standards) settings in relation to a default SAS Viya installation.   More specifically, they were asking for guidance regarding several perceived security violations in their environment after the deployment of SAS Viya 2025.01.   Looking into the details of the "policies violation" report and with the help of colleagues experienced in this area (Alexander Koller, Carus Kyle) we were able to provide some guidance.   We've now had a number of queries about Kubernetes Pod Security Standards and other kind of Pod’s policies in a Kubernetes cluster, so we thought it may be the right time to write a little blog to discuss this topic through a customer use case. 🙂   First, a little reminder about PSS   As a Kubernetes administrator in charge of the Cluster security, you should implement Security Standards for your Pods. Until version 1.25, "Pod Security Policies" was the mechanism provided in Kubernetes, to implement these standards. However, while they allowed a very granular control of the pods, there were complaints about the complexity of this "fine grain" policies.   As a result, from Kubernetes 1.25 the "Pod Security Policies" were removed, and administrators now have the choice between:   Pod Security Admission : a built-in Kubernetes mechanism to enforce Pod Security Standards with 3 levels of isolation: privileged, baseline, and restricted. a 3rd party admission plugin, that they can deploy and configure themselves.   So, the security rules that exist in a Kubernetes Cluster (and that can prevent a SAS Viya deployment) are either coming from the Kubernetes PSA configuration or third-party plugins (such as GateKeeper/OPA, Kyverno, etc…).   One major difference between the deprecated "Pod Security Policies" and the new "Pod Security Admission with the Pod Security Standards" is that, once a PSS profile (baseline or restricted) is chosen, it is not possible to define exceptions.   List of policy violations   Now let’s come back to our customer’s case, so we can illustrate the PSS concept.   Here is the “policies violations” report as provided by the customer:   # Name Total Violations Description 1 container-must-have-limits 1 violations - enforcementAction: deny group: ''kind: Podmessage: >-container <sas-cas-server> memory limit <28Gi> is higher than the maximum allowed of <8Gi> name: sas-cas-server-default-controller namespace: sasviya version: v1 2 psp-pods-allowed-user-ranges   879 violations: - enforcementAction: deny group: '' kind: Pod message: >- Container hydrator is attempting to run without a required securityContext/runAsGroup. Allowed runAsGroup: {"ranges": [{"max": 1337, "min": 0}], "rule": "MustRunAs"} name: sas-annotations-6644dc6f99-cmz97 namespace: sasviya version: v1- enforcementAction: deny group: '' kind: Pod message: >- Container hydrator is attempting to run without a required securityContext/runAsUser name: sas-annotations-6644dc6f99-cmz97 namespace: sasviya version: v1 3 psp-allow-privilege-escalation-container   1 violations: - enforcementAction: deny group: '' kind: Pod message: 'Privilege escalation container is not allowed: sas-opendistro-sysctl' name: sas-opendistro-default-0 namespace: sasviya version: v1 4 psp-host-filesystem   1 violations: - enforcementAction: deny group: '' kind: Pod message: >- HostPath volume {"hostPath": {"path": "/mnt/nvme-disks", "type": ""}, "name": "cas-disk-cache"} is not allowed, pod: sas-cas-server-default-controller. Allowed path: [{"pathPrefix": "/foo", "readOnly": true}] name: sas-cas-server-default-controller namespace: sasviya version: v1 5 psp-privileged-container   1 violations: - enforcementAction: deny group: '' kind: Pod message: >- Privileged container is not allowed: sas-opendistro-sysctl, securityContext: {"allowPrivilegeEscalation": true, "capabilities": {"drop": ["ALL"]}, "privileged": true, "readOnlyRootFilesystem": true, "runAsNonRoot": false} name: sas-opendistro-default-0 namespace: sasviya version: v1   Let’s review each type of violation and try to understand what it means.   #1 container-must-have-limits   As the name suggests, this policy seems to enforce restrictions on the resource limits that can be set for a container.   The description (or violation’s message) is pretty clear here: the resource limit for the memory is set to 28Gi in our sas-cas-server-default-controller container. But it is higher than the maximum allowed of 8Gi.   The CAS server has unique attributes and requirements. Because it has been engineered to use all available resources to complete each request in the shortest amount of time, CAS is usually configured to run on dedicated nodes, using most of the available resources.   By default, CAS operates with auto-resourcing enabled, allowing the server to dynamically adjust resource usage. In such case, the CAS Operator scans node specifications and allocates around 80% of the available memory to CAS worker pods. It is likely the reason why the memory is set to 28Gi, which does not meet in this case, this custom and general policy of not exceeding the 8Gi for any container.   #2 psp-pods-allowed-user-ranges   According to its name and the associated description, this policy is related to the user account used to run the container.   When looking at the description, it looks, at first, like there is a specific issue with the sas-annotations pod and its hydrator container …But in hindsight, it is likely that this violation has been actually observed for most of the SAS Viya containers, because there are 879 violations of the policy 😊 (So it probably also includes the init containers referenced in the Viya pods)   According to the violation’s message, the broken rules here are that neither runAsGroup nor runAsUser specification are set in the Security Context of the containers.   In Kubernetes, a Security Context "defines privilege and access control settings for a Pod or Container". It can be used to force all the processes inside a container to run under the same user, add or remove Linux capabilities for a Container and many other things in terms of security controls...Refer to the Kubernetes documentation for more details on Security Contexts.   When looking at the securityContext specification of the hydrator container in the sas-annotation pod, we can, indeed, confirm that the absence of runAsGroup and runAsUser specifications:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   I'm using k9s in this example, but you could use a kubectl command to get the same information.     You can even use the script below to run a report across all the SecurityContext that are set for the pods and containers in a given namespace :     The Security Context can be set at both the Pod level and Container level. The report tells you, for each pod, if a PodSecurityContext is set and for each container inside the pod if a SecurityContext is set. When set, the securityContext specification is provided.   Most of the time for all the SAS Viya pods, you'll see that the runAsGroup or runAsUser attributes are not set in the SecurityContext:     The only pods (in this 2025.02 Viya deployment) with these attributes set, are the CAS and sas-pyconfig pods.   Example:     Here, the runAsUser field specifies that for any Containers in the CAS Controller Pod, all processes run with user ID 1001.   Although the runAsUser is not set for (almost) all the SAS Viya pods, if you run the ps -ef command inside a SAS Viya container, you would see that all the processes run under the sas account. It’s because in Kubernetes, when  runAsUser is not set inside the SecurityContext, the main process of the container runs with the user ID specified in the container image (USER instruction in the container’s image Dockerfile).   Additionally, while the runAsGroup or runAsUser attributes are not set, what we see in the report is that most of the SAS Viya pods have the runAsNonRoot attribute set to truein their PodSecurity context (which requires all the containers in the pod to be run as non-root users).   Interestingly, setting this runAsNonRoot attribute to true meets one of the policy requirements of the restricted isolation level in the Kubernetes Pod Security Standard definition…) - as you can see below...     #3 psp-allow-privilege-escalation-container   This policy is related to the privilege escalation (such as via set-user-ID or set-group-ID file mode) that, may be sometimes required for some of the SAS Viya components or for specific Viya customizations…   The message tells us that the “Privilege escalation container is not allowed” for the sas-opendistro-sysctl container inside the sas-opendistro-default-0 pod.   You can confirm it if you look at the container’s Security Context:     It is a known and documented issue with the deployment of OpenSearch inside the SAS Viya platform:     The offending container is created by the sysctl-transformer.yaml that is set by default in the "transformers:" section of the kustomization.yaml file :   sas-bases/overlays/internal-elasticsearch/sysctl-transformer.yaml   As documented there in the SAS Viya Platform Operation guide, the purpose of the transformer is to set the vm.max_map_count kernel parameter on the OpenSearch underlying host to ensure there is adequate virtual memory available for accessing the search indices.   The SAS Documentation explains that, instead of using the transformer, it is possible to set the vm.max_map_count in advance on the hosts where the OpenSearch pods will be running (stateful nodes).   While you could do it manually on the nodes using the commands provided in the SAS documentation (basically run the sysctl -w vm.max_map_count=262144 command as root on the host), you can plan it in the infrastructure provisioning, using a custom script (or "user data") to be run when the VM is provisioned (see examples in Azure  Google Cloud or AWS).   Also note that a PR (Pull Request) has been submitted in the IAC for Azure GitHub repository to enable Terraform to implement this setting when the Stateful nodes are provisioned.   #4 psp-host-filesystem   Once again, the policy violation is pretty clear: HostPath volume {"hostPath": {"path": "/mnt/nvme-disks", "type": ""}, "name": "cas-disk-cache"} is not allowed.   While it’s often the most performant option, the usage of the hostPath volume type is generally seen as a security vulnerability in Kubernetes.   According to the violation description, hostPath volume type has been configured for the CAS Disk Cache location.   A possible workaround is to: not mount the hostPath location directly to the pod, but instead, create a PersistentVolumeClaim pointing to a PersistentVolume using the local-storage or manual Storage classes (Alexander Koller also provides nice examples there).   If you are deploying SAS Viya in opensource Kubernetes, a local-storage should already exist for you.   If you are deploying in the Cloud, you can also explore different options for the CAS Disk Cache, such as the Kubernetes Generic Ephemeral Volumes. If SAS Viya is deployed in Azure, you could also use the emptyDir volume type with ephemeral OS disk in Azure (as demonstrated in this opened Pull Request that has been submitted in the IAC for Azure GitHub repository).   But I’ll stop here, as the topic of "avoiding the use of hostPath and NFS volumes in a SAS Viya deployment to comply with security standards" would deserve its own post 😊.   #5 psp-privileged-container   Once again, the issue here is with the OpenSearch sysctl container.   According to the message, what triggers the violations here are the privileged: true, and runAsNonRoot: false specifications in the container’s securityContext specifications. Note that, in the Kubernetes built-in PSSs, these settings are respectively forbidden with the baseline and restricted profiles.   As explained above, this violation is the result of the sysctl-transformer.yaml transformer that is included by default in the kustomization.yaml for the OpenSearch container and the same solutions/workarounds apply.   Summary and answers for the security violations   So, let’s summarize the violations and solutions or workarounds in a table (as they were provided to our colleague working with the customer).   This information will help the customer to understand the reason why some constraints were not observed during the Viya deployment and how to fix the issue.   # Name Description Notes/Solution 1 container-must-have-limits message: >- container <sas-cas-server> memory limit <28Gi> is higher than the maximum allowed of <8Gi> Policy Source: Gatekeeper custom policy.   The CAS CPU/memory resources settings (limits/request) can be changed, and the memory limit can be set below 8GB. However, it will likely impact the CAS topology, workload placement, performance and potentially limit the amount of data to be loaded.   To be clear: yes, we can tune the CAS container, so its memory limit meets the 8GB maximum allowed but does it fit with the Architecture design / requirements agreed with the customer regarding CAS? (Most of the time, CAS is the component in SAS Viya that is expected to fully utilize host RAM in most scenarios.)   If it does not, the customer could consider adding an exception to the policy for this specific pod.   2 psp-pods-allowed-user-ranges   message: >- Container hydrator is attempting to run without a required securityContext/runAsGroup. Allowed runAsGroup: {"ranges": [{"max": 1337, "min": 0}], "rule": "MustRunAs"} name: sas-annotations-6644dc6f99-cmz97message: >- Container hydrator is attempting to run without a required securityContext/runAsUser Policy Source: Gatekeeper custom policy.   This policy controls the user and group IDs of the container (more specifically runAsUser, runAsGroup, supplementalGroups, fsGroup specs...).   In SAS Viya, except for CAS and Compute pods, the runAsUser, runAsGroup attributes are not set.   As a workaround you can write a patchTransformer that adds the runAsUser attribute to the pod definition with the correct user ID 1001. Since most of the SAS Viya pods have the runAsNonRoot parameter set to true, asking the customer for an exception is another option to consider.   3 psp-allow-privilege-escalation-container   message: 'Privilege escalation container is not allowed: sas-opendistro-sysctl' Policy Source: Kubernetes PSS, restricted isolation level.   See the OpenSearch section in the SAS Viya operations guide.   4 psp-host-filesystem   message: >- HostPath volume {"hostPath": {"path": "/mnt/nvme-disks", "type": ""}, "name": "cas-disk-cache"} is not allowed, pod: sas-cas-server-default-controller. Policy Source: Kubernetes PSS, baseline isolation level.   If hostPath is not allowed, then the customer needs to find another Kubernetes volume type for CAS disk cache. Using the Generic Ephemeral Volume with an AWS managed disk may be a good solution.   Performance may be degraded compared to using local NVME drives though.   5 psp-privileged-container   message: >- Privileged container is not allowed: sas-opendistro-sysctl, securityContext: {"allowPrivilegeEscalation": true, "capabilities": {"drop": ["ALL"]}, "privileged": true, "readOnlyRootFilesystem": true, "runAsNonRoot": false} Policy Source: Kubernetes PSS, baseline and restricted isolation level.   See the OpenSearch section in the SAS Viya operations guide.   Conclusion   More and more customers, investing in Kubernetes, are now implementing cluster security policies, using the Kubernetes built-in Pod Security Standards and/or 3 rd party tools.   In this particular case, the customer was deploying SAS Viya in AWS.  They had a mix of PSSs, with the restricted isolation level and also specific Gatekeeper constraints.   Note that we have seen some customers cases where additional security rules were also enforced by turning on the Cloud vendor’s built-in policies engine, such as the Azure AKS Built-in policies…   On the other hand, the SAS Viya has very specific requirements, support a wide range of customization that may not meet the generic policies implemented in the cluster by the customer.   That’s the reason why it is important to understand exactly what the constraints are, in order to determine workarounds or required configuration changes in the SAS Viya platform (when available).   Sometimes, the violations of the policies in place are indicative of a real problem. Other times, like here, they simply need an explanation and sometimes an exception to the policies can be implemented.   While it is possible, in general,  for SAS Viya to run in a cluster where the Kubernetes built-in restricted PSS is enforced, it often requires to find alternatives to commonly used implementations (ex: CAS and Compute access to data through NFS or hostPath, enable host account on CAS) and some specific customizations may require workarounds or may even not be possible (ex: enable SAS Watchdog to monitor SAS sessions, executing python code in MAS, etc.).   That's it for today, I hope it was helpful !   References   SAS Viya Platform Operations documentation: Pod Security Admissions SecurityContext : https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ Pod Security Standards, profile details: https://kubernetes.io/docs/concepts/security/pod-security-standards/#profile-details     Find more articles from SAS Global Enablement and Learning here. ... View more

As a Deployment engineer - about to start the installation of a new Viya environment in Kubernetes - or as a Technical Architect - working on the storage design for a Viya project - you may hear, or be questioned by your customer about something called the "CSI drivers"...   When searching in the SAS Official documentation, you could be surprised to see no list of officially supported CSI drivers for Kubernetes! It is because SAS can unfortunately not test each and every technology associated to Kubernetes or Containers in every scenario of a SAS Viya deployment.   As my colleague, Rob Collum recently said in one of our team meetings : "Some things are tested and supported by SAS and some other things… can possibly work"…his remark was spot-on : a lot of technologies used in the Cloud or in Kubernetes platforms can be used and are likely to work with SAS Viya, but it does not mean that they have been officially tested by SAS for every kind of  SAS Viya deployment…     However, what we can do, as SAS professionals is to share the experience and the lessons learned in the field where customers have implemented proven technologies, even when they were sometimes not explicitly listed as fully tested and supported in the SAS Documentation.   There are several examples of that: Customers deploying Kubernetes platform under limited support (Tanzu, Rancher, etc.), customers using new network or storage container integration technologies, etc.   In this particular post, we discuss and share some feedback on the usage of the Container Storage Interface (CSI) drivers.   What are the CSI drivers?   In order to expose external storage to the containers running in the pods, Kubernetes initially came up with an "in-tree" volume plugin system (including things like awsElasticBlockStore, azureDisk or cinder).   But the code was part of the Kubernetes system which caused friction over time…it forced storage vendors to align with the Kubernetes release system, and caused reliability and security issues in the core Kubernetes binaries (with a code that was difficult to test and maintain for the Kubernetes maintainers).   To address this challenge, the CSI (Container Storage Interface) was developed as "a standard for exposing arbitrary block and file storage systems to containerized workloads on Container Orchestration Systems (COs) like Kubernetes".        Or simply put, a CSI driver is a standard that allows Kubernetes to interact with storage systems.   Nowadays, most of the original volume plugins (awsElasticBlockStore, azureDisk or cinder) are deprecated and the official Kubernetes documentation recommends to use the "out-of-tree" volume plugins implementing the CSI.   Once installed in the cluster, new available storage classes appear in Kubernetes and can be referenced by the SAS Viya Persistent Volume claims, volumeClaimTemplates or Custom resources.   In the Cloud Managed Kubernetes environments (AKS, EKS, GKE, etc.) the Cloud CSI drivers maybe be pre-installed and the associated storage classes are already available.   Why do we need them?   Cloud providers provide CSI drivers for a better integration with their storage managed services. In the "on-premise" world, Storage systems vendors provide CSI drivers for a better integration with their SAN or NAS storage systems.   The usage of the NFS subdir provisioner is generally more flexible than the CSI drivers and its usage is currently encouraged both in our official documentation and in the SAS "Deployment as Code" GitHub project (aka DAC).   However, we see more and more customers expressing concern about using "open-source community provided" software in their production environments and require the use of official Cloud supported CSI drivers instead.   In addition of the Cloud vendor support, there can be other benefits such as TLS encryption in-transit or at-rest, automatic reprovisioning when additional space is required, advanced integration with the Cloud file services (snapshots, backups, etc.).     CSI Drivers Known limitations and requirements   The CSI drivers are mentioned in only two specific locations of the SAS official documentation.     Source: dilbert.com (Scott Adams)  Limitations for Azure Files and EKS CSI drivers   While the SAS documentation does not provide a formal list of supported CSI drivers, there is a warning in this section against the usage of specific CSI drivers.   Two specific CSI drivers that do not meet the CAS and the SPRE pods requirement for a fully POSIX-compliant system, are listed:   The Microsoft AKS Container Storage Interface (CSI) driver for Azure Files. When this CSI driver is set to use the SMB protocol, it does not support the standard POSIX file system operations that SAS processes use to unlink or delete files. The Microsoft AKS Container Storage Interface (CSI) driver for Azure Files. The Amazon EKS Elastic File Service (EFS) CSI driver. This driver does not allow files to be owned by distinct UIDs and GIDs. Certain SAS processes would not be able to chown files that are stored in these volumes. In addition, it is recommend to avoid the use of Azure Files as the backing storage for the internal PostgreSQL instance because SAS testing with this configuration has found multiple issues.   These warnings come from real life experience where customers have tried to use these CSI drivers in the field and faced issues during file access operations with CAS, the Compute servers or the internal crunchy PostgreSQL. As an example read about two SAS colleagues' experience with the aws-efs-csi-driver…) in the extract below from a SAS Internal blog :   Using the aws-efs-csi-driver In our IAC for AWS, the nfs-subdir-external-provisioner is used when EFS is the desired persistent storage. However, when customers aren’t using our IAC, the aws provided efs csi driver is commonly used instead. Honestly, it makes sense logically, but there is of course a gotcha when this choice is made. While EFS claims to be POSIX compliant, it seems the provided csi driver didn’t get the memo. When using the EFS CSI driver, there is no way to alter ownership of files or directories in PVCs. The chown/chgrp command will fail as described here.  To add to the fun, the issue has been marked as closed and not planned… This often presents itself as:     rsync: chown …  failed: Operation not permitted (1)  For now, our best recommendation is to avoid use of the aws-efs-csi-driver and instead use the nfs-subdir-external-provisioner as the IAC does.   But as noted in the documentation, and explained in the post’s introduction: "This is not an exhaustive list of storage options that do not fulfill SAS requirements. Many storage options are available across all the supported cloud platforms and Kubernetes distributions. SAS cannot test with all of them."     Understanding the nuances…   Now we have to be careful…these known limitations, listed above, DO NOT MEAN that Azure Files or Amazon EFS storage can never be used with SAS Viya…   The devil is in the detail…and yes I know, it is complicated…   But for example:   Azure Files Premium, with the NFS protocol support can be used and could be a great fit for many Viya volumes that requires RWX access mode. Amazon EFS, is officially documented as the storage to use in HA deployment of SAS Viya (preferred over the Single VM NFS server option for the RWX Viya volumes), and is a supported option in the SAS provided IaC/DaC tools. However it is preferable to mount the EFS storage with a generic NFS provisioner (rather than the EFS CSI driver as it has some limitations).   So, when considering the storage options for a Viya environment, it is important to take several things into account :   The kind of Viya volume we deal with: Is it one of the Viya platform dynamically assigned during the deployment? or a static volume to store user’s data? are we looking for a shared storage (RWX) or would block storage (match RWO access mode request) be a better option for this volume? Full POSIX support (usually depends on the protocol used to access the file system). How is the storage mounted (specific CSI driver or generic NFS provisioner?), is there a minimum storage size for the volumes mounted with the CSI?     Required CSI Driver   The other location where the CSI driver is mentioned in the SAS Official documentation is in the "Cluster requirements for AWS" section.   Instead of a limitation, this time, there is a requirement to have the AWS Elastic Block Store (EBS) CSI driver installed in the cluster.   As explained in this blog post from Rob Collum, the reason of this requirement is that since EKS 1.23, without a specific installation of the EBS CSI driver inside the cluster, it is not possible to use the default AWS gp2 or gp3 storage classes.   These default storage classes are recommended for the SAS Viya volumes that require RWO access (such as OpenDistro, Redis, RabbitMQ, etc.…).     Recap of the most common Cloud storage provisioner and CSI   At this point, and to avoid further confusion, let's look at a small table listing the most common Cloud vendors CSI drivers.   Cloud Provider CSI Driver Storage Storage type Access mode AWS efs.csi.aws.com EFS (Elastic File storage) NFS managed service RWX AWS ebs.csi.aws.com EBS (Elastic Block Storage) Managed disk (Block storage) RWO Azure file.csi.azure.com Azure Files Premium NFS managed service (NFS support) RWX Azure disk.csi.azure.com Azure Disks Managed disk (Block storage) RWO Google filestore.csi.storage.gke.io Google Filestore NFS managed service (NFS support) RWX Google pd.csi.storage.gke.io Google Persistent Disks (standard-rwo) Managed disk (Block storage) RWO     The Azure Files CSI driver "trap"   While you can use the Azure Files CSI driver (assuming you use Azure Files premium with the NFS support), there is an additional gotcha that you’ll need to know if you want to use it for your SAS Viya platform dynamically provisioned volumes...   Back in 2023, a SAS colleague explored the use the Azure Files Premium CSI driver and provided guidance on how to implement it with the NFS protocol and the nconnect mount option. He wrote an internal blog that shows how you can use the Azure Files storage for your SAS Viya dynamically provisioned volumes during the deployment and provision them with the Azure Files CSI driver and that it all works perfectly.   …HOWEVER…   In this case, for each dynamically requested Viya volume, the use of the CSI driver triggers the provisioning a distinct new Azure File Share.   Now the problem is that, as of today the minimum size for Azure Premium files share is 100GB   The result is that the provisioned capacity in Azure is way higher than what is effectively needed…Some volumes only require 1GB (or even less), but a 100GB file share is provisioned in Azure for each of them.   The screenshot below compares the required capacity with the provisioned capacity.       Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   So while it is technically possible to use the CSI driver here, the cost is very high and a lot of space is wasted.  This situation was recently faced at a customer site by a SAS colleague in the AP region.   A first workaround, that the customer tried, was to set the shareName attribute in the Azure Files storage class. When this attribute is set, all volumes that use the azurefile-csi storage class consolidate their content in the same Azure Files share (so we don't need to create one file share for each PVC).   For example, if we have the following 9 PVCs:   cas-default-data cas-default-permstore sas-cas-backup-data sas-common-backup-data sas-commonfile sas-pyconfig sas-risk-cirrus-core-backup-data sas-risk-cirrus-search-data   …then the content of all the corresponding PV is created at the root level of a single file share, as below:     … which poses a risk of data contention and potential overwrites due to the lack of directory isolation.   This shareName attribute has been tested by the SAS R&D who confirmed that it was not suited for dynamic volume provisioning. Using this setting with Viya could lead to data corruption issues as the Viya components, which request storage through distinct PVCs, expect distinct volumes to be provisioned.   As a result of this experience and test, a new section Guidance When Using Microsoft Azure Files was added in the official documentation.     The official workaround, in order to use Azure Files for the dynamic provisioning of the RWX-access based Viya platform volumes, is to use an NFS provisioner. The Kubernetes NFS Subdir External Provisioner can be used to create a custom storage class based on an existing Azure Premium Files share. The NFS provisioner can create distinct subdirectories for each Viya volumes under the Azure file share. Note that (at the time of this post write up), this NFS subdir external provisioner is used for various NFS -based storage types by the Deployment as Code project (aka "viya4-deployment").     A better experience with the GKE Filestore CSI Driver   This "over-provisioning" issue when using the CSI driver for dynamic provisioning is not specific to Azure, the same problem has been seen in the past with the Google File Store.   However while the GKE Filestore CSI driver version 1.26 and earlier only allowed for 10 shares at 100GB minimum, things have been improved with the enterprise-multishare-rwx GKE storage class, where the minimum size has been reduced to 10 GB.   While there might be a bit of extra provisioned space, the overhead is minimal.     Conclusion   While using CSI drivers can cause issues, it is also something that is expected to be required by more and more customers…as it often provides a better integration with the managed storage services and extends the Kubernetes capabilities (encryption, snapshot, DR ,etc.).   But remember that there is no list of SAS officially supported CSI drivers...what we have in the documentation today is more a list of CSI drivers that SHOULD NOT be used for specific volumes. The cost efficiency also depends on the kind of volumes for which you would use the CSI driver. For example Azure Files CSI driver, which is supported with the premium SKU (NFS support) is probably not cost efficient for all the Viya platform dynamically created volumes, but it may be cost efficient for the SASDATA or CASDATA static volumes...   Each volume comes with its own specific requirements (access mode, capacity, latency, security, POSIX compliance, data protection, etc.…). Having multiple storage classes is a recommended practice as it allows to associate the right kind of storage technology and device to each Persistent volume.   The topic of storage in Kubernetes is complex and the technologies are evolving rapidly…Some of the issues or limitations discussed in this post may disappear in the future, but other issues could arise when untested scenarios are implemented…   Here are two examples :   It has been recently discovered that using Azure NetApp Ultra (NFS based storage) for the internal crunchy PostgreSQL was causing issues during the upgrade operations to the new PostgreSQL 16 version. The current recommendation is to use block storage for the internal crunchy PostgreSQL volumes. The Kubernetes sig-storage group recommends to move from the NFS subdir external- provisioner (the project is not maintained) to the NFS CSI driver. SAS is currently evaluating alternatives for the viya4-deployment GitHub project.   Finally while it would be impossible to test, support and list all possible CSI drivers implementation with the SAS Viya platform, it is assumed that the CSI drivers can be used for most of the SAS Viya volumes in general. However in case new problems or limits with specific CSI drivers are discovered in the field and reported, they can be reproduced and assessed by SAS and potentially documented to increase awareness across the SAS Technical community.   That’s it for today! Thanks for reading!!!   Find more articles from SAS Global Enablement and Learning here. ... View more

This post is a follow-up on a previous blog where we explained how recent changes in Control Group v2 management in Kubernetes have impacted the stability of the CAS server in the SAS Viya platform.   In the latest post, we discussed some workarounds, but with the latest versions of SAS Viya there is now a new feature in the SAS software to avoid this issue in most of the cases.   This new feature is called "Backing Store for CAS Memory Allocations" and this post explains how it works and how it can be setup to mitigate the CAS OOM (Out of Memory) issues impact in the Viya environment.   TL;DR   This post contains a lot of details and is fairly technical, so if you are not about to read the whole article, here is a little “TL;DR” summary 😊   A new CAS feature has been introduced from 2024.09 to limit the CAS memory allocation. If it is enabled, when a CAS action uses more memory than a defined limit, it is interrupted with a failure and a meaningful error message, but the CAS session remains active and the Linux OOM killer is not triggered (which avoids the situation where the OOM killer performs a “group” kill of every process in the CAS container, leading to CAS service interruption). It greatly improves the stability and reliability of the CAS Server service.   The technical mechanism used behind the scene to limit the CAS memory allocation (before the OOM kicks in) is to back the memory allocation onto a special Kubernetes volume (emptyDir of “memory” medium – RAM-backed filesystem) – for which we can set a SizeLimit value. When this value (usually set to 80% of the CAS container's memory limit) is exceeded, the CAS action is interrupted but the CAS session and services remain available for the end users.   The "Backing Store for CAS Memory Allocations" feature was first introduced with SAS Viya stable 2024.09 (which means that it is also included in the latest SAS Viya LTS 2024.09) and instructions where provided to patch or edit the CASDeployment Custom resource to enable it.   Then with the new stable 2024.11 version, new PatchTransformers were added, so it is now possible to manually enable the backing store for CAS memory as part of the initial deployment of SAS Viya (with the proper configuration in the kustomization.yaml file).   The current plan is make the backing store become the default in the next versions (in the first months of 2025).   CAS Memory allocation and current issue   We know that the CAS_DISK_CACHE can be used to cache CAS tables on disk (as SASHDAT files) using memory-mapped file. CAS can leverage the CAS_DISK_CACHE to quickly and efficiently "swap in", "swap out" memory blocks as needed and hence holding a volume of data that is higher than the amount of physical memory on the machine (Read these nice posts from Rob Colum and Nicolas Robert if you want to know more about how and when the CAS Disk Cache is used)   But in CAS, an action also allocates "resident" (physical) memory in order to process a table, maybe doing things like creating computed columns or views that required RSS memory to hold rows/columns.   For this part of the Analytics processing, the CAS_DISK_CACHE is not used; CAS allocates memory using the Threaded Kernel (TK) and Memory is backed using the "mmap" system call with the MAP_ANONYMOUS flag. With this flag, the pages are only backed by either real memory or the paging files (with MAP_ANONYMOUS the mapping is not backed by any file).   However, in most Kubernetes system, containers do not have a configured paging file, and CAS can only use real memory…It increases the risks of the CAS session process (running in containers) to be terminated by the OOM killer - as soon as the container's defined memory limit is exceeded.   Having session processes forcibly killed is not a great experience for the end users.   As an aggravating factor, we know (from my previous blog) that - from recent Kubernetes versions (1.28) – when a single CAS session process is killed by the OOM killer, all the other processes in the same cgroup(v2) are also killed, which lead the whole CAS server (SMP) or individual CAS worker(s) (MPP) to go down, impacting the CAS service.   While it is understandable that, from time to time a CAS action could fail (because the system is not equipped with enough memory to run it), it is hardly acceptable that the failure of a single individual CAS action causes the entire deployment of CAS to restart and interrupt the functionality of CAS...   That's the problem that is addressed by this new "Backing store for CAS Memory allocations" feature.   What is this new CAS backing store ?   It is now possible to enable a "backing store" to support CAS memory allocations and prevent the whole system being impacted by the fact that a single action requires more memory than what has been made available for the CAS container.   With this feature, the CAS Threaded Kernel (TK)  is informed that the files in a specified directory can be used to back up most of the memory allocations.   The TK_BACKING_STORE_DIR environment variable is set in the CAS container and points to a specific path where the Threaded Kernel stores the mapping files (ex: /cas/tkMemory ).   The path is mounted in the pod and mapped to a Kubernetes emptyDir  volume.   As noted in the official Kubernetes documentation:  "The emptyDir is created when a pod is assigned to a node and is initially empty. When a Pod is removed from a node for any reason, the data in the emptyDir is deleted permanently".   By default, the content of emptyDir volumes is stored on the node’s root disk (typically under /var/lib/kubelet ). However, it is possible to set the emptyDir.medium field to "Memory", the documentation explains that Kubernetes mounts a tmpfs filesystem (RAM-backed virtual filesystem) instead of using the disk.   A size limit can then be specified to limits the capacity of the emptyDir volume. That's how we can control/restrict the memory consumption before it is too late...   In the CASDeployment custom resource specification, it would look like that :     OK but how does that help ?   The backing store for CAS memory allocations relies on the SizeLimit parameter as THE way to set a control on the maximum amount of memory that a CAS action could use, without involving the OOM Killer.   With the backing store feature enabled: when a CAS action is submitted to CAS, the action starts to map the memory files in the emptyDir and if the size of the mmapped files exceeds the defined SizeLimit (for example 24GB in the previous example), then there will be an individual failure of the CAS action.   The error message shown to the end users (for example in SAS Studio or any other CAS client) will inform them that the CAS action has failed because it ran out of memory.   It will look like this:     Or like that:     With this new behavior, the problem is addressed before involving the OOM killer and seeing it terminate all the nearby CAS processes. It helps ensure that the CAS server and even the CAS session itself suffer less impact from large-memory tasks, allowing the end-users to continue to work with CAS.   The “fail fast” principle is used here to prevent the OOM killer issue and improve the CAS overall stability.   Finally, the release of the files in the backing store follows the same pattern as releasing memory back to the operating system without this feature enabled. When the CAS session process terminates, all of the files used to back the TK memory are released (which ensure there will be no leaked resources).   Additional considerations   Specific memory allocations that are not constrained   While most of the use cases benefit from the “Backing Store for CAS Memory Allocations” feature to prevent OOM kills, there are some exceptions. Not all memory consumption is constrained by the size of the backing store. These include:   Memory allocated by the main CAS processes on behalf of actions (but it should be a reasonably small amount.) Memory allocated by third party code (database access drivers provided by the database vendor or any open-source code that an action consumes). Memory in other programming languages - when an action spawns processes or threads in other programming languages (such as Python or Java) to implement the action's logic   So, for example, Python or Java code running inside the CAS session does not go through the CAS TK Memory allocation and as such will not be subject to the memory control via the TK backing store.   Requires a careful choice of SizeLimit   The SizeLimit value defines the maximum amount of memory that can be used by CAS actions before causing an "out of memory" failure.   For the CAS backing store to be efficient, it is important that this threshold is lower than the threshold that would trigger the OOM killer…the goal here is to ensure that the session fails and display an “out of memory” error message to the end-user running this action, before the CAS container memory limit is exceeded and all processes are being killed.   Testing, with the default CAS auto-resources configuration, has shown that setting the backing store to on the node seems to be effective in preventing OOM kills in most situations. In situations where the CAS resources request and limit are manually set (usually to places more than one CAS pod on a node), it is recommended to use a similar fraction (80%) of the container memory limit.   Coordination with the CAS resources management configuration.   In CAS, it is possible to implement resource management through policies that are created with the Viya CLI. You can create up to 5 five "priority-level" policies per CAS server that is used to place space quotas on table data. Each “priority-level-n” is associated to a group of users.  If you enable this kind of configuration, it is also possible to align the backing store configuration to define distinct TK Memory allocation limits for each policy.   Other ways to prevent the OOM “group-kill” issue   Finally, note that SAS R&D is also currently researching additional ways to prevent the OOM killer from terminating CAS processes. One alternative workaround is based on the use of a Kubernetes Daemonset to overwrite the cgroups configuration files created by Kubernetes. The issue with the OOM "group-kill" change in Kubernetes is not specific to CAS, other applications have been impacted and a pull request to allow configuration of the "group OOM kill" behavior should be included in Kubernetes 1.32.   How to implement it ?   When the "Backing Store for CAS Memory Allocations" capability was introduced for the first time in SAS Viya, the official way to enable it was to patch the CASDeployment Custom resource deployment or to directly edit the CASDeployment Custom resource.   But since version 2024.11, there is now an official Kustomize Patch Transformer to apply the changes. A new "Configure a Backing Store for Memory Allocations" paragraph has been added in the "Optional Customization" section of the official documentation.   There are actually 4 distinct transformers to apply depending on the CAS configuration that is in place.   The first one cas-enable-default-backing-store.yaml should be used when the CAS autoresourcing is enabled (which is the CAS default configuration) OR if you have manually set the CAS resources request and limits. You might have done that to allow several CAS pods to run on the same node, maybe because you need to optimize the infrastructure costs or because you are deploying SAS Viya in a Bare metal environment on a small set of large physical machines as discussed there) In these cases, there is no need to set the size for the backing store ( SizeLimit ) as it is automatically set by the CAS operator to 80% of the CAS container memory limit (when it is manually set). The second one , cas-enable-backing-store.yaml lets you specify the SizeLimit . It should be used when: The CAS container doesn't have a memory limit OR You don't want to use the 80% default value (A customer might decide something like "I want more memory dedicated to the file cache" or "I'm using the CAS Gateway and it uses lots of memory in Python, so, I want a smaller limit on CAS TK Memory"). Then when CAS resource management policies have been enabled with different priority groups, we can have separate backing stores for each resource management priority group and set different SizeLimit values for the priority groups that correspond to group of users. Two transformers are provided as examples (one with 5 priority groups and another with just one).   As usual, once you have copied and eventually adjusted the values in the transformer YAML file, don't forget to reference it in the "Transformers" section of your main kustomization.yaml file.   Note that the Kubernetes manifest, generated from the Kustomize build with the updated reference, must be re-applied and that the CAS Server should be restarted too to pick up the configuration change.   Here is an example of what you would see in the CAS pod specification when CAS auto-resource is enabled (no use of CAS Resource Management policies) and when the backing store is enabled with the default transformer ( cas-enable-default-backing-store.yml ).   The first screenshot shows the effect of the CAS auto-resource configuration on an 8 CPU/64GB machine. The CAS container requests and limits are automatically computed by the CAS operator.     Then you can see below the effect of enabling the Backing Store in the CASDeployment CR:     Conclusion   The previous posts on this topic talked about switching from cgroupsv2 to cgroupsv1 as a workaround to avoid the OOM Killer issue in Kubernetes…However the Kubernetes community has decided to move cgroupsv1 into maintenance mode for Kubernetes 1.31, so removing cgroups2 as a solution may be poorly received... that’s another reason to opt for this new "Backing Store for CAS Memory Allocations" feature instead.   Finally, nothing really prevents a specific CAS action from running out of memory at some point. It’s always a possibility… maybe because the code has not been optimized or the dataset size is too large, or simply because running this action on this amount of data requires more memory than physically available in the infrastructure. However, with this new "backing store" feature, this type of situation is reported as an "out of memory" condition and only affect the individual user's session, as opposed to all sessions running in a CAS pod being arbitrarily killed.   That’s why this change is really improving the overall reliability and availability of the CAS Server.   Video Bonus   Now, as a little reward to your perseverance in this post reading 😉, you can find below a short 3 minutes video to quickly demonstrate the benefit of using the "Backing Store for CAS Memory Allocations".       Thanks for reading !     Find more articles from SAS Global Enablement and Learning here. ... View more

With the July version of SAS Viya (stable 2024.07), a nice change has been made in the section of the SAS Viya Operations guide that covers the customizations of a deployment.   Before this version, the "Common Customizations" page just provided a long list of all the customizations (outside of product-specific customizations) that should be considered for a deployment.   But now this "Common Customizations" content has been divided in two parts :   Required customizations Optional customizations   While many customizations are optional, a few of them should be decided in advance (and they sometimes can’t be made without specific pre-requisites in place).   This documentation change helps to identify some of the key decisions that MUST be made (in a way that corresponds to the customer needs or requirements) before starting any new Viya deployment.   So, I thought it would be great opportunity to revisit and discuss these architecture decisions and deployment pre-requisites through the review of these six "Required customizations" and explain why they would form the basis of a Viya environment Architecture design document.   As we often say during our GEL workshops:   "The deployment of SAS Viya by itself is usually a matter of hours (or even minutes). However, the discussions, decisions and preparation leading to that moment can take days or even weeks..."    TLS Mode Host and port for Cluster Ingress CAS topology and workload strategy Internal or external PostgreSQL?  Internal or external OpenSearch? StorageClass to use for RWX PVC ? "Recap" table   Note: In this post we don't explain HOW these customizations are implemented as it is already nicely explained in our documentation or in the README files.     TLS mode ?   TLS (Transport Layer Security) allows to encrypt data in motion and is used in the SAS Viya platform to secure communications with internal and external components.   3 options are possible : "Full stack TLS", "Front door TLS", or "no TLS".   By default (using the kustomization.yaml example file provided in the documentation), your Viya environment will be automatically configured with "Full stack TLS". It means that all the microservices and web applications implement HTTPS in their communications. In this mode, all network connections from outside and even inside the Viya environment are secured.   With "Front door TLS", only the components that are intended to accept connections from outside the Kubernetes cluster (such as the Ingress controller and CAS server) are configured to only accept encrypted network connections.   In both cases, the customer can use the default certificate issuer (openSSL) or use another certificate issuer (cert-manager). The customer can also provide their own certificates if they prefer to secure the ingress.   The customer should always be advised to use TLS to secure the network traffic ("No TLS" should never be used as the traffic would be in clear text and easily intercepted by anyone). However, most of the time, customers are very sensitive about  security and opt for the "Full stack TLS option".   Reference :   SAS Doc : https://go.documentation.sas.com/doc/en/itopscdc/default/itopssr/n18dxcft030ccfn1ws2mujww1fav.htm#n0w1os9ufnn7xwn1an3qbhxjxze9     Host and port for Cluster Ingress As part of the deployment, you need to provide a small set of parameter’s values in your initial kustomization.yaml file.   In the officially documented example, they are represented as place holders under this form:  {{ PLACE_HOLDER }}. One of them is the {{ NAME-OF-INGRESS-HOST }}   How can you find this value ?   I wrote a couple of posts (part 1 and 2) to answer this question, here is an extract :   {{ NAME_OF_INGRESS_HOST }} corresponds to the URL base that the customer wants to expose to the SAS applications users. The {{ PORT }} parameter is optional, as the HTTP schema is usually determined by the start of the url (“http” or “https”). However, the important thing to understand here is that {{ NAME_OF_INGRESS_HOST }} also correspond to a DNS name that must be associated to the external IP address exposed by the Ingress controller.   When you install the Ingress controller in a Cloud environment, an external IP address is automatically provisioned by the Cloud provider (You can see this address in the EXTERNAL-IP column when you describe the ingress controller service).   A DNS name then needs to be mapped to the ingress public IP by the site’s IT team configuring their DNS services. The exact components of the FQDN will likely be shaped by the site’s DNS standards, for example: https://viya.your_dept.your_site.com/ and Viya accommodates that.   But there must be a DNS entry associated with this IP address. It is a key pre-requisites that should not be overlooked during the deployment planning process.     CAS topology and workload strategy   As the main analytics and data management engine of the SAS Viya platform, CAS is a cornerstone of the environment. A CAS Server is required at the start-up time of Viya : it is a pre-requisite to have at least one CAS Server defined otherwise the Viya platform will not work for any user (whether the users plan on using CAS or not).   Some choices must be made at a minimum for the CAS topology and the CAS workload strategy.   While, by default (using the kustomization.yaml example file provided in the documentation) the CAS server consists of a Single-node (SMP : for Symmetric Multi-Processing), it is possible to opt for a Multi-node CAS server (MPP: for Massively Parallel Processing) to spread the Analytics processing and data across multiple CAS nodes.   This choice is mostly driven by the amount of data to process (remember that with CAS, the data are loaded into memory to be processed by CAS, and a single machine may not have enough RAM to hold them) and an official sizing is a key input in this decision.   Note that more sophisticated CAS topologies are also possible such as:   Adding a secondary CAS controller into a CAS MPP deployment (to improve the CAS Server availability) Adding new CAS server(s), for example to isolate and specialize the different kind of workloads (ex: 1 CAS server for the reports, 1 other for the Visual forecasting, etc…)   Now, regarding the CAS workload strategies, several options are also possible. The SAS general recommendation is to choose the default "auto-resourcing" configuration. This approach ensures that each CAS instance (Controllers and workers) is running on a dedicated host and can leverage all its capacity (in a similar way as with a Viya 3.x deployment). While this is ideal for maximum performance, it might be overkill for dev/test environments.     Note that options are also available to have a differentiated workload classes for the CAS controller(s) and the workers. The goal is to optimize the costs by using distinct node pools and smaller machines for the CAS controller(s) that usually don’t require as much CPU and memory as the workers.   The CAS topology and workload management topics are a key part of the Architecture design and should be discussed with your site’s IT team to make sure the associated decisions meets your site’s needs and constraints.   Another exciting benefit of running Viya 4 as containerized workloads in a Kubernetes cluster on virtualized hardware is that changing your deployment topology can be effected simply by submitting configuration changes programmatically. This should give your site’s administrators confidence to try new workload configurations in pursuit of best performance, efficiency, and cost.   If these concepts are not clear to you, the  SAS Viya Architecture course has dedicated modules to cover them.   References :   CAS Server Topology Changes And CAS Table Balancing:  https://communities.sas.com/t5/SAS-Communities-Library/CAS-Server-Topology-Changes-And-CAS-Table-Balancing/ta-p/903549       Internal or external PostgreSQL?   The SAS Viya platform relies on an instance of PostgreSQL for its SAS Infrastructure Data Server. This instance can be either internal or external.   By default, SAS Viya is installed with the internal PostgreSQL instance that relies on Crunchy data server which is shipped with the SAS order. But the SAS Viya platform can also be configured to use an external instance of PostgreSQL such as the Azure PostgreSQL Flexible Server.   This post, from my colleague Edoardo Riva, lists some of the considerations associated to the choice of an internal versus external instance.   It is worth noting that many SAS Solutions also require an additional PostgreSQL server instance for the Common Data Store (aka CDS), which requires its own PostgreSQL instance. When deploying a SAS Solution, the location of the Common Data Store instance (internal or external) - as well as any additional potential instances, for Airflow for example - must also be discussed and addressed.     Internal or external OpenSearch?   The SAS Viya platform also relies on an instance of OpenSearch to create and store search indices (for example for the SAS Information Catalog and Visual Investigator).   Starting with 2023.03 it is possible to use an external instance of OpenSearch.   Once again, you can find a nice post from Edoardo, comparing the use of an internal versus an external instance of OpenSearch.   The decision between : letting the SAS Viya deployment install and manage OpenSearch (default) versus using an existing external instance, must be made before the deployment.     StorageClass to use for RWX PVC ?   The last item of the "Required customizations" covers another key decision : which kind of storage to use for the Viya component that requires a ReadWriteMany (aka RWX) storage class.   As a reminder, the RWX storage type can be seen as the equivalent of a shared file system in the bare OS world. It corresponds to a kind of storage that supports concurrent access by multiple pods running on different nodes.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Since the SAS Viya PersistentVolumeClaims don’t specify a StorageClassName, the default storage class specified by your Kubernetes configuration is used.   This  is convenient because most of the Managed Kubernetes platforms that we support in the Cloud come with a default storage class.  On the other hand, it’s not often the ideal choice because the usual default storage class is backed up by managed disks, which has only RWO access(standard-rwo in GCP, managed-csi in Azure, etc..).   While many of the SAS Viya platform volumes only need storage with RWO access, some of them require RWX storage…As a consequence, if you don’t do anything regarding the storage class when deploying for example in AKS, GKE or EKS, your deployment will fail.   So, for them we need to provide an alternative RWX storage class and make the associated configuration changes, as in the example below :     See how the annotationSelector is used to target the PVC that requires this RWX access.   So, the associated architecture decision is: "What storage class should I use for the Viya components that require RWX access ?"   Several options are possible, they depends on the Kubernetes platform. You can have a look at this table from the Kubernetes documentation to get a first list of RWX compatible volumes plug-ins, but options mostly depends on the Cloud provider.   Note that if you are using the viya4-iac-x (IaC) or  viya4-deployment (aka DaC) tools, you don’t have to tweak the Kustomize Transformer files : the DaC automatically configures a new RWX storage class and make it the default. In this case, the new storage class is backed up by an NFS Client provisioner pointing to either a Cloud Managed File services (such as Azure NetApp, GCP File store or EFS in AWS) or to an NFS server running on a Cloud VM (depending on your IaC storage choice).   The available DaC options may correspond to the customer’s needs and requirements. Otherwise, it is required to identify a RWX storage class and make the configuration changes so it is used for the Viya volumes that requires it. Note that you can also use distinct RWO and RWX storage classes for the different volumes required by the Viya platform.     Other key decisions   Finally, it is important to remember that there several  key decisions that are NOT reflected by a specific requirement listed in the new "Required Customization" section.   Some of them (Kubernetes platform, dedicated or shared cluster, etc…) are critical and early to decide, with downstream implications on the other aspects of the architecture and deployment (networking, storage, etc…).   While it is probably not an exhaustive list, the decisions listed below correspond to the customer’s specific requirements or constraints and must also be made during the design phase (before the deployment) :   Kubernetes Platform ? AKS? GKE? EKS ? OpenShift ? other ? Dedicated or shared Kubernetes cluster ? Cadence (LTS or stable) ? Storage : used for the Viya platform components and for things like input/output data, user’s home directory, etc…as an example, for production environments, block storage (instead of NFS based storage) is strongly recommended for specific SAS Viya components (OpenSearch, Redis and RabbitMQ) volumes ! Network : kubelet or CNI ? (Note that the question is not relevant for deployment of SAS Viya on Upstream Opensource Kubernetes cluster, where only the Calico CNI is supported.) Identity provider and authentication mechanism : various options are available for the integration with the Identity provider (LDAP or SCIM) and for the authentication.   We have added this decisions into our recap table below.     Recap and advice   The table below lists the key decisions that should be made before performing the initial deployment with their default value, pre-requisites and discuss whether they could be changed after the initial deployment.     Decision Default value Pre-requisites Can be changed  after the initial deployment? TLS mode “Full-stack” TLS Custom certificates (in .pem format) and the associated private key must be provided if the customer wants to use their own certificates for the ingress (instead of the default ones). Yes. The TLS mode can be changed afterwards, however it is recommended to agree on a TLS mode before the deployment to avoid any outage. Ingress host name and DNS None The customer needs to associate a DNS name to the Ingress controller service external IP , available Yes. However if the  INGRESS_HOST is wrong, nobody will be able to login to the deployed environment from the outside. CAS Topology and Resource Management strategy SMP, CAS-autoresourcing None Yes. The topology and workload placement strategy can be changed but it will impact the deployment and users (outage and need to reload the data). Internal or External PostgreSQL instance(s)? Internal None with the internal instance. For external: extensions, number of connection, pre-created SharedServices database, etc…see this page Yes but only in one way. SAS has created the PostgreSQL Data Transfer Tool for the SAS Viya Platform to move content from the internal instance of PostgreSQL to an external instance. But it is not possible to change from external to internal. Internal or External OpenSearch instance? Internal   No. Be careful, after the deployment has completed, you cannot modify your deployment to use a different OpenSearch instance. RWX Storage class to use for the SAS Viya Persistent Volumes ? There is usually no default RWX storage class available in the cluster. But the IaC tool for opensource Kubernetes and the DaC tool creates one for the SAS Viya volumes. A RWX storage class must be made available otherwise the deployment will fail. Maybe but not easily…while some volumes can be rebuilt from scratch, other may hold user’s data that needs to be migrated to new storage devices. Kubernetes platform ? None A supported Kubernetes platform should me made available with cluster-wide privileges No. If the Kubernetes platform is changed, SAS Viya must be redeployed (however a full-system migration could be done from one Cloud provider to another). Dedicated or shared Kubernetes cluster ? NA When multiple Viya versions are sharing the same cluster, there are some caveats to be aware of. Keeping all the Viya environments at the same version is strongly recommended. See above. LTS or stable cadence ? None To remain fully supported an update of the version N of SAS Viya must be made before the version N+4 is released. It means that customers on the stable cadence must update their Viya at least version every 3 to 4 months to avoid to be in limited support. Yes it is possible to change the cadence (under certain conditions). Storage strategy for the platform data and the user’s data NA Storage classes or PV types to use for the Viya platform volumes as well as input/output data must be made available Possible but requires tests, backup, outage and migration of the content. Network type ? kubelet ? CNI As of today, kubenet is usually the default for Managed Kubernetes Services in the cloud. In some cases (SAS with SingleStore, deployment with open-source Kubernetes) a CNI plugin is required. So always check the latest documentation version for requirements. Changing from one CNI implementation to another may be possible depending on the Cloud provider but requires tests, backup, outage and migration of the content. Refer to the specific cloud provider’s documentation. Identity provider and authentication mechanism NA The requirements depends on the type of identity provider and authentication, see the official documentation The identity provider cannot be changed for an “in-use” system. authentication mechanism can be changed in some cases but could require an outage and rebuild and re-apply the deployment manifests   My final advice would be to make sure that you have captured at minima the architecture decisions listed in this table (and prepared the associated pre-requisites) before starting your SAS Viya deployment process.   Also keep in mind that some post-deployment tasks are usually required as well before users can start to access the SAS Viya applications. They also depend on the offering and products that are deployed. See this page in the official documentation for a list of post-deployment steps.   Hope that helped ! Thanks for reading 😊     Find more articles from SAS Global Enablement and Learning here. ... View more

As a follow-up of my previous post "Manage your SAS Viya platform configuration in a Git Repository", I wondered if it would be possible to store and manage the SAS Viya configuration in a Cloud Provider’s Git repository service (instead of using something like GitLab or GitHub).   Git is the most widely used source-code management tool and Git repositories have become a key component of modern DevOps platforms. Azure, AWS and Google Cloud Platform offer "git compatible" Code Source repository services, such as Azure Repos, AWS CodeCommit or Google Cloud Source Repositories.   But can the SAS Viya platform play nicely with them ?   Here is a diagram of the generic deployment system that we’ll aim to put in place.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Assuming we use the "sas-orchestration command" deployment method and already have a Cloud Source Repository in place (created from the Cloud portal or with the CLI), our goal is :   To push an initial SAS Viya configuration (a folder containing the kustomization.yaml file and the site-config folder) to the Cloud Git Repository. Then to run the sas-orchestration command to deploy SAS Viya using the configuration files stored in the Cloud Git Repository.   The purpose of this post is to demonstrate, in a few steps, how to do that with the Google Cloud Source Repositories service.   Introducing the Google Cloud Source Repositories Git local authentication setup  Cloud Source Repository setup Prepare the initial SAS Viya configuration in the local Git repository Push the initial SAS Viya configuration to the Cloud Git Repository Edit your code in the Google CSR Deploying SAS Viya Conclusion   Introducing the Google Cloud Source Repositories   Google Cloud Source Repositories (or "Google CSR" to keep it short) is the Git Repository service available in the Google Cloud Platform market place*.   It has several interesting features (as you can see on the screenshot) and is a "fully-managed" Git repository.                     Google CSR currently supports 3 types of local authentication  :   SSH : Consists of a private key that resides on your local system and a public key that you register with Google Cloud. When using this mode, you’ll have to use your private SSH key to authenticate yourself against the Git repository. Google Cloud CLI : Cloud Source Repositories lets you authenticate by using the google cloud CLI (gcloud). You need to run the gcloud init command to authenticate in your GCP project. Manually generated credentials, a script is generated in the Google portal to create the .gitconfig and .gitcookies files (using the Netscape/Mozilla cookie file format), which is passed in the in the Git HTTP session, when the git command is used.   Google also documents 3 distinct kind of URLs to use in order to "git clone" the repository, corresponding to the authentication systems :   Cloning using SSH:   git clone ssh://[EMAIL]@source.developers.google.com:2022/p/[PROJECT_ID]/r/[REPO_NAME]   Cloning using the gcloud command:   gcloud source repos clone [REPO_NAME] --project=[PROJECT_NAME]   Cloning using the manually generated credentials:         When I started my research, I was not too sure which kind of local authentication to choose for my need to make the "sas-orchestration" tool read the SAS Viya configuration from the Google CSR.   However, with some hindsight, knowing that the sas-orchestration container relies on the "go-getter" library and requires that the "go-getter" protocol must be “Git over http(s)”, I could have been able to determine which authentication method to use…   (*) Note : Important: Effective June 17, 2024, Cloud Source Repositories isn't available to new customers. Organizations that have used Cloud Source Repositories prior to June 17, 2024 are not affected by this change. Git local authentication setup ....   End of the suspense 😊   The authentication method to choose here is the "Manually generated credentials".   It is a choice by elimination: the “Git over SSH” protocol is not supported for the DaaS tools go-getter URLs and the gcloud CLI is not included in the sas-orchestration image. When you opt for this method, you need to click on a link, which (after authentication with your Google account), generates a page as below: This page gives you the snippets of code to run to configure your local git client (for Windows and Linux). When you run this code locally, it creates an http.cookiefile and references it in the Git configuration (~/.gitconfig file). In a Linux operating system, it looks like this: This is how Git can authenticate you over http(s) to the source.developers.google.com domain. Once it is in place, you can run your git clone command to pull the content from your Google CSR.   Cloud Source Repository setup It is very easy to create your first CSR repository in Google, you can use the web interface (go to https://source.cloud.google.com/repos and click on "Add repository") or the gcloud CLI with a command like:   Prepare the initial SAS Viya configuration in the local Git repository.   gcloud source repos create raphpoumar-source-repo   Now, imagine that you have your initial set of SAS Viya configuration files, locally available on your jumphost/bastion linux machine, and you want to move them to your brand-new Google Cloud Source Repo. The following actions need to happen:   Clone the empty CSR repository with git Example:     It creates a local folder on your machine with the same name as your CSR repository. This new folder contains the git database (under the .git folder) is your local Git Repository.   Setup the local Git configuration (~/.gitconfig). Example:   #Setup git default identity git config --global user.email $EMAIL git config --global user.name $STUDENT git config --global credential.helper store git config --global http.sslVerify false   Create a local sub directory inside the local git directory and place your SAS Viya configuration files there. It would look like this:   Now add and commit the SAS Viya configuration files in the local git repository. Note that the commit message appears in the Google CSR version history after the push. At this point, the Git repository has been initiated but it remains local.   Push the initial SAS Viya configuration to the Cloud Git Repository.   It is time to register the remote Git repository (which is in the Google CSR) and to push our local changes there.   First, we add our local repository as a remote:   STUDENT=$(cat ~/student.txt) PROJECT_NAME=sas-gelsandbox git remote add googlehttps://source.developers.google.com/p/$PROJECT_NAME/r/$REPO_NAME   And then we can push from your local Git repository to the Google Cloud repository But then…you might see this kind of message: With a worrying line at the end saying that there was an error to push our files 🙁   Now, if we run the "git describe" command provided in this message, we see that the culprit is the file site-config/postgres/ServiceAccountKey.json. What the message actually means, is that our push has been rejected because the service has detected that we were trying to upload what looks like a private key (the "ServiceAccountKey.json" file) ! In a standard usage of the Git repository, it is indeed a bad idea. We usually don't want to place any credential information in a code repository. However, our case is a bit specific as we are not keeping source code but configuration files. In addition, we need this key to configure PostgreSQL and our Code Source repository is secured and private, so we can override this default security setting with the nokeycheck option. So, let’s try again   git push -o nokeycheck --all google Now we should see something like that: The problem is fixed 😊 If we open the Cloud Source Repositories web page, we can confirm that our configuration files have been transferred !   Edit your configuration files in the Google CSR   OK so now my kustomization.yaml and site-config folder files are in the Google source repository, but wait…how do I update them if I want to change my SAS Viya configuration? There are several ways. You can make the changes in your local git repository (from the local Linux machine used to do the first push) and add/commit/push them into the Google CSR remote repository. But it is also possible to edit directly your configuration files from the Cloud Source Repositories web UI. You’ll first need to give additional IAM permissions in GCP (like "Source Repository Writer"), but then you can click on the "Edit code" button and  a new Cloud shell instance will be started and the editor directly opens your configuration files (as shown below). Finally, even easier, if you have Visual Studio Code (with the Git plugin) installed on your local Windows workstation, you can also use it to modify your SAS Viya configuration files, commit and push the changes (as long as you have updated your Windows Git configuration to use the manually created credentials). So, now that our configuration files are in place and can be edited in the Google CSR, let’s see how to pull them when we deploy the software 😊   Deploying SAS Viya The final step is to set up the go-getter URL in the sas-orchestration command and run it to deploy SAS Viya! The parameter that tells the sas-orchestration container where to find the SAS Viya Configuration file is "user-content". Unlike with the GitLab example shown in my previous post, we cannot use a Git private token for the authentication (this authentication method was not available with the Google CSR). Instead, we need a way to make our credentials available for the sas-orchestration container. Here is how the command looks like:     When we activate the verbose mode of the sas-orchestration tool we can see that the git clone command is executed inside the container to clone the remote repository. So, as shown in the command example, the trick is to mount our .gitcookies and .gitconfig files inside the sas-orchestration container and to help Git to pick up the .gitconfig file by setting the $HOME environment variable. In this way the sas-orchestration tool has all the required credentials to perform the “git over http(s)” request to pull the viyaorch content from the Google repository.   Conclusion   Finally, we have seen that it is possible to run a Viya deployment from a configuration stored in the Google Cloud Source Repository. If you want to test it by yourself, a hands-on called "Deploy Viya 4 from Google Cloud Source Repositories" is available in the "course exercises" notes of the “SAS Viya 4: Deployment on Google Kubernetes Engine” GEL Lab: we’ll drive you through all the configuration steps described in this post. We could then imagine to use the Google CSR as the central location for all our SAS Viya environments configurations. As we've seen, the configuration files could be updated there in various ways and scenarios:   Directly edited from the Google CSR UI Spin up an ad hoc VM that "git clone" the repository, update a given configuration, then push the changes. Then another instance or process can run the deployments. Synchronized from GitHub or Bitbucket repositories The initial diagram was updated to reflect all these new opportunities 😊 Thanks for reading !   Find more articles from SAS Global Enablement and Learning here. ... View more

If you are about to deploy the latest SAS Viya version (2024.06 or later) and are using an external PostgreSQL server, there is a new system requirement.   If you miss it, your deployment may fail….This post explains the rationale and provides some guidance on how to meet this new requirement. We'll look at the "What?", "Why?" and "How?".   What?   From the "What’s new page" for SAS Viya stable 2024.06:   “Starting with 2024.06, the SharedServices database for an external PostgreSQL server is no longer created automatically during the initial deployment of the SAS Viya platform. Instead, you must manually create it before you start the SAS Viya platform deployment. This database supports the required SAS Infrastructure Data Server component."   In a SAS Viya deployment, the SAS Data Server Operator pod (aka "DSO") is the component responsible for creating the SharedServices database in the SAS Infrastructure Data Server (aka "IDS") instance.   From the stable 2024.06 version, the DSO does not create the SharedServices database if the PostgreSQL implementation is external.  The DSO still creates the SharedServices database if the Internal Crunchy Data Server is used.   Symptoms if you miss it   It is an important change.   If we forget about this requirement and deploy as we did prior to Stable 2024.06, we would probably see that our deployment fails with most pods either in "error" or stuck in the "Init: 0/x " status with a message like the one below from the sas-start-sequencer container’s log:   {"level":"info","version":1,"source":"start-sequencer","messageKey":"DB Check sleeping","messageParameters":{"elapsed time":"21m:29s:301ms","name":"main"},"properties": {"caller":"sdsci/main.go:405   In k9s, the pod’s view would look like that:     And after some troubleshooting, we would ultimately identify the root cause by looking in the SAS Data Server Operator pod’s log (the pod’s name is something like "sas-data-server-operator-c97c5c796-nxmnc") :   {"level":"error","version":1,"source":"sas-data-server-operator-c97c5c796-nxmnc", "messageParameters": {"p1":"failed to connect to `host=frarpo-r-0294-default- flexpsql.postgres.database.azure.com user=pgadmin database=SharedServices`: server error (FATAL: database \"SharedServices\" does not exist (SQLSTATE 3D000))"}, "messageKey":"failed to initialize database, got error%v", "properties":{"logger":"commons/storage/gorm", "file":"/go/pkg/mod/gopkgs.sas.com/sonder/v2@v2.42.2/storage/gorm/gorm.go:406", "caller":"gorm/gorm.go:607"},"timeStamp":"2024-06-26T09:43:46.992640+00:00","message":"failed to initialize database, got error %v"} … {"level":"error","version":1,"source":"sas-data-server-operator-c97c5c796-nxmnc","messageKey": "Reconciler error", "properties":{"error":"database server is external and cannot connect to the SAS database", "caller":"logr/logr.go:49"}, "attributes":{"controllerKind":"DataServer","DataServer":{"name":"sas-platform-postgres","namespace":"test"},"namespace":"test","name":"sas-platform-postgres", "reconcileID":"f7de99f8-f6b0-4a96-9e6f-53a9a8570ab5", "controller":"dataserver","controllerGroup":"webinfdsvr.sas.com"}, "timeStamp":"2024-06-26T09:43:46. 992706+00:00","message":"Reconciler error"}   An important detail…   This change, in the way the DSO handles an external PostgreSQL, relies on the presence of a specific annotation on the DataServer Custom Resource Definition which is only added if the new version of the PatchTransformer template delivered from version 2024.06 is applied.   When the DSO detects this annotation, it does NOT create the SharedServices Database in case it is not there. Instead, it only logs the fact that the SharedServices database is not there, with an "error" message (as shown above).   Here is an extract of the new PatchTransformer file with Stable 2024.06, usually called platform-postgres-dataserver-transformer.yaml or cds-postgres-dataserver-transformer.yaml (per the README instructions).     So if this sas.com/database-server-location annotation is NOT set for the DataServer Custom Resource, then there is no change and the SharedServices database is created by the DSO, in the external PostgreSQL server, just as before.   Here are some examples of cases in which the new annotation is not added:   As of today, if you deploy SAS Viya with the DaC tool - viya4-deployment GitHub project which has its own copy of the xxx-dataserver-transformer.yaml (without the annotation). or if you reference a static version of this configuration file in your Kustomization.yaml file from a previous SAS Viya version in your deployment of 2024.06.   In either of these situations, you won't see any change of behavior – the SharedServices database is automatically created, as before, by the DSO, even for an external PostgreSQL server.       Why?   The reasons leading to this DSO change in 2024.06 are related to security concerns :     Here is the specific requirement that could be problematic for security conscious database administrators:   The user that is the database owner must also have CONNECT privileges on the system database named postgres to enable SAS Viya platform services to start up.   The long-term goal is to remove the need to have such privileges associated to the PostgreSQL account used in the SAS Viya configuration.   The 2024.06 change removes the need for the Data Server Operator (DSO) to require this privilege (If a customer uses external Postgres and correctly pre-creates the SharedServices database, then the DSO won't connect to the "postgres" system database in the first place).   However…if you look at the external PostgreSQL requirement, you can see that we still have this requirement ☹:     It is because, outside of Data Server Operator (DSO), the Java services of the Viya platform still require a user with the CONNECT privileges to the system database (named postgres). Work is in progress to move to a newer Java framework and once all Java services will be using it, then there would be no need for a PostgreSQL account with to the system database for an external DB server.   With 2024.06, we are taking a first step toward the goal to reduce the permissions requirements on the Postgres user account used by Viya to interact with the external PostgreSQL server.     How?   Now…How can we avoid our deployment with the external postgres failing when we deploy stable 2024.06 ?   Pre-create the SharedServices Database   From the 2024.06 version of SAS Viya, if you are using external Postgres then you need to create the SharedServices database before starting the deployment process. Some guidelines are provided in the official documentation for this new requirement:     But while there is a link to the official PostgreSQL documentation, there are no command examples.   To create the SharedServices database, you should be able to run the code below with the user that you were passing into DSO.   CREATE DATABASE "SharedServices" WITH TEMPLATE template0 LC_COLLATE 'C' LC_CTYPE 'C'; REVOKE ALL on DATABASE "SharedServices" FROM public;   This is the SQL command that the DSO was running until 2024.06 (and keeps running when the PostgreSQL instance is internal).   That seems pretty easy, right?   Yes it is…As long as you are connected to the PostgreSQL database with an account with enough privileges to do so.   In customers environments, the database administrator should be able to connect to the external PostgreSQL server and run this command with the appropriate account.   Otherwise, here is a little trick to do it with a few commands.   Pro-tip: Create the Shared Service from an ad-hoc pod   The idea is to start an ad-hoc temporary pod, in the Kubernetes cluster, to run a psql command to create the Database.   In the example below, we set our information in environment variable and then we instantiate a pod in which we can run the command to create the database.   # Set required postgres variables PG_ADMIN_USER=pgadmin POSTGRES_PASSWORD="<my-super-password>" PG_SERVER=frarpo-r-0294-default-flexpsql.postgres.database.azure.com PG_PORT=5432 NS="test" # Create the SharedServices database kubectl run postgresql-dev-client --rm --tty -i --restart='Never' --namespace $NS --image docker.io/bitnami/postgresql:13 \  --env="PGPASSWORD=$POSTGRES_PASSWORD" \ --command -- psql -U $PG_ADMIN_USER --host $PG_SERVER  -d postgres -p $PG_PORT   This command opens a psql prompt:     Then you can type the SQL commands noted above to create and secure the SharedServices database:     The SharedServices database should be created before deploying SAS Viya.   However, if this requirement has been forgotten, it is still possible to do it even after the SAS Viya deployment has failed (due to the missing SharedServices database). But you must delete the DSO pod (sas-data-server-operator), after you run the SQL command.  When a new DSO pod is started, it detects the SharedServices database and, thanks to the Kubernetes auto-healing capabilities, all of the SAS Viya pods should recover to a running/ready state (this scenario was tested in our GEL lab).   Specific case: If you use the IaC tool for Upstream opensource Kubernetes   Finally, there is another particular case where you would NOT be impacted by this change.   If you are deploying SAS Viya in upstream open-source Kubernetes and if you decide to use the "SAS Viya 4 Infrastructure as Code (IaC) for Open Source Kubernetes" to automatically install the PostgreSQL server, there is already an ansible task that creates the SharedServices database and grant all the privileges to the configured PostgreSQL administrator account.     In this case, the DSO change with 2024.06 will be transparent for you.   Conclusion   There is always a tradeoff to make between simplicity and security.   This change adds a requirement, with an additional task (pre-create the SharedServices database), either for the customer’s database administrator or for the person in charge of the Viya deployment. Despite the additional work, the expectation is that the reliance on a PostgreSQL super user account (with permissions on the postgres system database) can be completely removed in later SAS Viya versions.   That would allow SAS Viya to support the Zero Trust principle of "least privileges".       Find more articles from SAS Global Enablement and Learning here. ... View more