The SAS Viya platform includes by default (and for free!) SAS/CONNECT with all its capabilities. Previous articles, including Moving to the cloud with SAS/CONNECT, have shown SAS/CONNECT evolution to adapt to cloud environments and support our customers in their journey to modern architectures. Innovations keep coming! Starting with SAS Viya version 2023.10, SASCMD sign-ons (a.k.a. MP Connect sign-ons) always start new SAS sessions in new pods by default.   Let's start by stepping back a moment to see what MP Connect is, and why this is a welcome change.   Distributed computing, with a single machine?   Before cloud computing, before any kind of multi-machine distributed computing, SAS customers had no choice but to run their code on a single machine (yes, you can tell I’ve been at SAS for quite some time).   SAS capabilities have always been disruptive, even in those limited environments. Multi-process CONNECT (or MP Connect) was conceived to divide time-consuming tasks into multiple units of work and to execute these units of work in parallel. It did that – and still does – by providing a framework that lets you start and coordinate multiple child SAS processes from a controlling parent SAS session. This way you can parallelize code that otherwise runs sequentially, for the purpose of reducing the total elapsed time necessary to execute a particular application.    An example of a process split into multiple parallel subprocess.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   From machine to pod   MP Connect capabilities are automatically available in SAS Viya platforms, because they are provided by the SAS/CONNECT product, which is included by default in every SAS Viya environment. What happens when you move your existing MP Connect code to SAS Viya? Apparently, it's business as usual: child processes are spawned from the parent session to execute part of your code in parallel. But, under the hood, the execution environment is completely different. What used to be running on a physical machine, now runs in Kubernetes, inside a pod.   MP Connect spawning child sessions in a compute pod.   Maybe your original machine had 8 cores and 64GB of RAM (not too much, in modern terms) and your code is written to take full advantage of that computing power by spawning 7 additional sessions that use, in total, almost 100% of all CPUs. Kubernetes is built to control the execution environment and prevent a single pod from exhausting the resources of the node where it's running. As soon as your code starts requesting all that CPU power, Kubernetes will throttle down your pod - in a default configuration, down to 2 CPUs maximum. As a result, your existing code will run 4 times slower! It's obvious that this setup does not scale as expected. You could argue that the issue can be easily fixed by configuring the Kubernetes cluster to give more resources to SAS compute and connect pods. Yes, this solution could work, but it suffers of two problems:   It simply moves the bar a bit further. Today you want back your 8 CPUs, what will happen when you'll have more data and you will need 16, 32, or more CPUs? A single pod can only scale up to the size of the node where it's running. Scaling up an individual gigantic pod is an anti-pattern in the cloud. Cloud environments are designed to scale out by launching multiple, smaller pods. The Kubernetes cluster hosting your SAS Viya platform is probably designed to support multiple users with multiple applications. You cannot hog all resources for yourself!   What could be a better solution? Obviously, embracing a cloud-native design and scaling out to multiple pods!   Scaling out to multiple pods   If you've been following so far, you can now understand how the new default behavior is welcome in cloud environments. Starting with SAS Viya 2023.10, every new MP Connect sign-on always starts a child SAS process in its own dedicated pod, embracing cloud-native scalability and elasticity. Kubernetes can spread out the pods on multiple nodes, and, if the cluster is configured for auto-scaling, your limit is the sky... or better, your budget!    MP Connect launching child sessions in dedicated connect pods.   It's a matter of choices   Every time a new default is introduced in existing environments, it's important to give SAS Administrators to option to embrace this new capability, or to reset the SAS Viya platform to behave just like before. In this case, you can use a new environment variable, SAS_LOCAL_MPCONNECT. When set to true, it re-enables local MP Connect Sign-Ons, i.e. the original functionality of spawning a child session in the same pod where the parent process is running.   A SAS Administrator can use SAS Environment Manager to set the SAS_LOCAL_MPCONNECT environment variable to true in sas.compute.server: startup_commands and sas.connect.server: startup_commands configuration instances. In this case, the setting is configured for every SAS compute and connect server running in the environment.    Setting the SAS_LOCAL_MPCONNECT option for all compute server sessions.   A more limited scope could be achieved by setting the option case-by-case, as needed. As an end-user, you can add the following line in your code just before submitting the SIGNON statement:   options set=SAS_LOCAL_MPCONNECT=true;   In this case, only the current execution reverts back to the previous functionality.   Why would I revert back?   This new capability seems the obvious choice when you are writing code that uses MP Connect in SAS Viya. So why would you revert back to the previous functionality? The most obvious answer is when you are migrating existing code that could be broken by the new behavior. It's easy to understand that if the existing code uses local resources to share data between MP Connect sessions, it cannot work as-is when these sessions are launched in different pods. Here are a couple of examples:   The code saves datasets in the SASWORK library of one session to retrieve them from another session. SASWORK locations can be shared between SAS processes with the INHERITLIB= option of the SIGNON statement. You are using SASESOCK libraries to stream data between sessions through local pipes.   Those issues can be solved by re-architecting your application - for example, by sharing data through Kubernetes volumes mounted on all pod sessions, instead of using local directories. Yet, reverting back to the previous behavior can be an interim step to keep the code running during your migration.   Conclusion   I am always excited when I see how SAS Viya keeps evolving by embracing existing functionality and integrating it into modern cloud-native architectures. We have seen in this post how traditional MP connect code can be used without sacrificing Kubernetes capabilities to manage resource utilization, provide scalability when required, and ensure user-level process isolation.     Find more articles from SAS Global Enablement and Learning here. ... View more

The SAS/CONNECT Programming with SAS Viya: SIGNON article presents multiple ways to use the SIGNON statement in a SAS client to start and connect to a remote server session executing in the SAS Viya platform. One of the options covers the use case of external clients, with a brief list of preliminary configuration steps required to enable the capability. This article dives into the detailed steps behind that summary list, including a step-by-step recorded demo.   What are we trying to accomplish?   In a default deployment, the SAS/CONNECT spawner service is only reachable from internal clients. An example of an internal client is a SAS Studio session that submits the SIGNON command.    An internal client connecting to the SAS/CONNECT spawner to launch a server. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   External clients need additional configuration to reach the SAS/CONNECT spawner from outside the Kubernetes cluster. You have to create either a LoadBalancer or NodePort service. Examples of external clients are a SAS 9.4 Display Manager session executing on a laptop, or a SAS Studio session running in a different SAS Viya cluster or namespace.    External clients connecting to the SAS/CONNECT spawner via a Kubernetes LoadBalancer or Nodeport to launch server sessions.   The configuration steps Preliminary: verify that internal connections work as expected   Before even starting any configuration change, it’s a good practice to test that the default functionality is working as expected. You can verify that internal SAS/CONNECT connections are OK, in your SAS Viya environment, by logging into SAS Studio and submitting this simple test code – no changes required:   /* launch a SAS/CONNECT session on the same cluster */ %let session1=sas-connect-spawner 17551; /* authenticate re-using the already logged-in credentials */ SIGNON session1 user="_OAUTH_"; %put The client session is running on host: &SYSHOSTNAME; rsubmit; %put The server session is running on host: &SYSHOSTNAME; endrsubmit; SIGNOFF;   This code will do the following: Sign on to the SAS Connect Spawner using the Kubernetes service name sas-connect-spawner , which is only available inside the namespace, re-using the same credentials you entered when logging into SAS Studio Locally output the hostname of the pod where the client SAS session is running Use rsubmit to return the hostname of the pod where the SAS/CONNECT session is running Sign off from the remote session If this all works, you are ready to perform the steps to enable external access. 1.      Enable external access to the SAS/CONNECT spawner.   You can follow the instructions from the README file at <deploy>/sas-bases/docs/configure_sasconnect_spawner_in_the_sas_viya_platform.htm , in the section “Provide External Access to sas-connect-spawner via a Load Balancer”.   As you can read there, SAS provides a sample file ( sas-bases/examples/sas-connect-spawner/enable-external-access/sas-connect-spawner-enable-loadbalancer.yaml ) that you can copy in your sas-config directory, customize as needed, and include in the kustomization.yaml used for the original deployment. Then it’s just a matter of re-applying it using the same method as the original deployment (manual kubectl commands, sas-orchestration CLI, etc.)   An effective customization I always like to do is to reuse the load balancer frontend, which was already provisioned for the SAS Viya Ingress. This can bring a couple of benefits: Lower infrastructure costs, by avoiding the creation or provisioning of another load balancer. May simplify the TLS configuration, as detailed in the next step, by re-using the same external IP address and DNS alias as the Ingress. If you want to do it, you have to find the IP of the ingress load balancer with a command such as:   NS=ingress-nginx kubectl -n ${NS} get service ingress-nginx-controller -o json | jq -r ".spec.loadBalancerIP"   Then add it to the sas-connect-spawner-enable-loadbalancer.yaml as the last line, as in this example:    Example configuration of sas-connect-spawner-enable-loadbalancer.yaml   After the new configuration has been applied and submitted to Kubernetes, it's always a good practice to verify that the load balancer service gets created as expected, listening on an IP address external to the cluster:   NS=sasviya kubectl -n ${NS} get service sas-connect-spawner-loadbalancer Sample result: NAME                               TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)           AGE sas-connect-spawner-loadbalancer   LoadBalancer   10.42.230.223   10.XXX.XX.X    17551:31883/TCP   15m   2.      Verify the DNS name presented by the SAS/CONNECT spawner TLS certificate.   In the previous step, we added the new Kubernetes LoadBalancer service to an existing external load balancer that had already been configured in the corporate DNS and whose DNS alias was already specified in SAS Viya kustomization.yaml. For that reason, the certificates generated by SAS Viya already include that DNS name. The simplest way to verify that is to find the DNS name used for the ingress, and confirm that it is included in the SAN field of the SAS/CONNECT spawner TLS certificate.   To find the DNS name of the ingress, you can look into the kustomization.yaml file used to deploy SAS Viya:   grep INGRESS_HOST kustomization.yaml   Sample result:   - INGRESS_HOST=sasviya.gelenable.sas.com   Now list all the aliases in the SAN field of the SAS/CONNECT spawner TLS certificate:   NS=sasviya CONNECT_SPAWNER_SECRET=$(kubectl -n ${NS} get secrets | grep sas-connect-spawner | awk '{print $1}') kubectl -n ${NS} get secret ${CONNECT_SPAWNER_SECRET} -o jsonpath="{.data['tls\.crt']}" | base64 -d | openssl x509 -text -noout | grep -A 1 "Subject Alternative Name"   With these commands, we first get the name of the Kubernetes secret used by the SAS/CONNECT spawner. Then, we extract the certificate from tls.crt field of that secret, base-64 decode it, and pass the result to an OpenSSL command that prints the details of the certificate in a human-readable format. Finally, we filter the results to only show the "Subject Alternative Name" details.   The result should be similar to the following. Note that it includes the full DNS name found in the previous step.   X509v3 Subject Alternative Name:     DNS:localhost, DNS:sas-connect-spawner, DNS:sas-connect-spawner-8b9cd797b-5wswp, DNS:sasnode07, DNS: sasviya.gelenable.sas.com, IP Address:10.42.126.166, IP Address:10.43.17.98, IP Address:10.XXX.XX.X, IP Address:127.0.0.1   If you did not reuse the existing load balancer, and you created a new load balancer with either a new frontend IP address or a different DNS alias for the SAS/CONNECT spawner, then you’ll have to follow the additional steps described in the Encryption in SAS® Viya®: Data in Motion guide in the section Add the External Host Name or IP Address to the SAN in the Signed Certificate.   3.      Configure the client to trust the SAS Viya root CA certificate.   It is quite common to deploy the SAS Viya platform using a self-signed SAS Viya root CA certificate. This CA certificate is unknown to clients external to the cluster, so they cannot trust certificates presented by any SAS Viya server - including SAS/CONNECT. You can tell you are in this situation when, the first time you log into a web application, such as SAS Studio, your browser presents you with a security warning that you have to accept before you can get in. In this case, to be able to use SAS/CONNECT from an external Windows client, you have to obtain the SAS Viya root CA certificate, and then import it into the Windows client truststore. After that, the client will trust the certificates presented by SAS Viya servers. The official documentation lists a simple command to save the root certificate into a temporary file:   NS=sasviya kubectl -n ${NS} get secret sas-viya-ca-certificate-secret -o go-template='{{(index .data "ca.crt")}}' | base64 -d > /tmp/SASViya_ca.crt   Then you have to copy the resulting SASViya_ca.crt file to your Windows client (you can email, ftp, network copy… ), and import it in the Windows truststore, for example following the instructions in the Encryption in SAS® Viya®: Data in Motion guide in the section Import CA Certificates into the Windows Trusted Root Certificate Authorities Store.   Finally: Connect!   After you get this far, it’s time to enjoy the fruit of your work. From the client machine, you can start a SAS session and submit some code to verify if all is fine and SAS/CONNECT is configured for external connections. You can use code like the following, after substituting the DNS alias or IP address of the load balancer in the session1 macro variable:   /* launch a SAS/CONNECT session from outside the cluster */ /* set TLS connection options */ OPTIONS NETENCRYPTALGORITHM=SSL; /* connect using a fully-qualifies hostname or DNS alias */ %let session1=sasviya.gelenable.sas.com 17551; /* authenticate with an authinfo file */ SIGNON session1 password="_AUTHINFO_"; rsubmit; %put &SYSHOSTNAME; endrsubmit; SIGNOFF;   The resulting submission should print, in the SAS log, the name of the SAS/CONNECT server pod that gets started in the SAS Viya cluster, as in the following screenshot:       Recorded Demo   A recording of the configuration of a demo environment, showing all the steps listed above, is available on the SAS YouTube channel: SAS Demo | Configuring SAS/CONNECT for external access to SAS Viya Summary   When you have an external SAS client that you want to connect to a SAS Viya environment, you have to perform some configuration steps before you can connect them. Enable the SAS/CONNECT spawner to support external clients. Add the external host name or IP address to the SAN in the SAS/CONNECT spawner certificate. Obtain the SAS Viya root CA certificate and add it to the truststore of the external client. Links Official documentation to configure SAS/CONNECT for external access: The README file available in your deployment directory at <deploy>/sas-bases/docs/configure_sasconnect_spawner_in_the_sas_viya_platform.htm , in the section "Provide External Access to sas-connect-spawner via a Load Balancer" The "External Client Sign-On to TLS-Enabled SAS Viya SAS/CONNECT Spawner" section of the SAS® Viya® Platform Administration documentation  Sample SAS/CONNECT code to sign-on from internal or external Clients Ensure the TLS certificate presented by SAS/CONNECT spawner includes the external hostname used to connect to the server and any desired DNS alias Ensure the client accepts as valid the TSL certificate presented by the SAS/CONNECT spawner: Obtain the SAS Viya Platform Generated Root CA Certificate and Import CA Certificates into the Windows Trusted Root Certificate Authorities Store SAS 9.4 and SAS Viya 3.x client requirements   Find more articles from SAS Global Enablement and Learning here. ... View more

Starting with the 2023.03 SAS Viya release, the SAS Viya platform supports using an external instance of OpenSearch to provide searching and indexing capabilities. What are the requirements, and how to enable this new capability? If you are interested, read on!   OpenSearch OpenSearch provides search and indexing features for software that runs on the SAS Viya platform. Starting with the 2020.1.3 Stable release, the SAS Viya platform includes an Apache 2.0-licensed distribution of OpenSearch with enhanced security and a SAS-provided Kubernetes operator to handle its operation and management.   Starting with the 2023.03 release, new configuration options are added to the platform to support either: an internal instance of OpenSearch automatically included in the deployment (just as before) an external OpenSearch instance that you administer and maintain OpenSearch is required by every SAS Viya offering except SAS Viya Programming.   Why? The OpenSearch instance deployed by default inside the SAS Viya platform includes opinionated deployment settings to configure it according to the platform requirements. For example, the pods must run under a particular user ID, and require specific kernel settings on the nodes underlying the Kubernetes cluster. Some customers may have security policies that prevent these settings in their Kubernetes clusters. And, let’s be honest, a production instance of OpenSearch does not fit typical cloud-native environments: for example, every update to the SAS Viya platform requires a full stop and restart of the OpenSearch cluster.   An external OpenSearch instance is managed outside of SAS Viya and can alleviate these considerations. Supporting an external OpenSearch removes some requirements from the Kubernetes cluster dedicated to the SAS Viya platform. As an added benefit, an external OpenSearch reduces the footprint of the SAS Viya platform deployment.   In the end, adding support for an external option provides you with more flexibility in your architectural choices!   Internal VS External A simple table can easily show the main comparison points between using an internal OpenSearch instance or an external one. Internal External Dedicated instance deployed by the same process as the rest of the SAS Viya platform OpenSearch instance deployed by you before starting the SAS Viya platform deployment Deployed in the same namespace as the SAS Viya platform Deployed outside the SAS Viya platform namespace SAS Viya platform cluster sizing and design must account for OpenSearch (including kernel requirements, and storage requirements) OpenSearch requirements and sizing do not impact the SAS Viya platform cluster Contains all required plugins (including enhanced security) Must include required plugins Automatically configured as required by the SAS Viya platform Must be configured according to SAS specifications Managed by a SAS-provided Kubernetes operator Managed as you choose to Supported by SAS Supported by you   Requirements Whether you decide to use an internal or an external OpenSearch, once your deployment is done, there is no turning back. It is not possible to reconfigure an existing SAS Viya platform to use a different OpenSearch. If you change your mind, you will have to un-deploy and re-deploy.   The official requirements for an external deployment are described in the readme file at <$deploy>/sas-bases/examples/configure-Elasticsearch/external/README.md . Here is a summarized listing, valid at the time of writing (always refer to the official documentation to have up-to-date requirements): You can only use the opensource OpenSearch edition, not Elasticsearch. The supported OpenSearch version is 2.5. It must include a specific list of plugins used by the SAS Viya platform. The OpenSearch server should be set up and running before you start the SAS Viya platform deployment. Managed cloud subscriptions to Elasticsearch and OpenSearch are not supported; only instances that you deploy and configure. TLS between the SAS Viya platform and OpenSearch is strongly recommended, but not required. SAS Visual Investigator requires additional configuration settings. You can find the listing of SAS Visual Investigator specific requirements in the README file at <$deploy>/sas-bases/examples/configure-Elasticsearch/external/config/README.md , and a sample config.yaml is provided in the same directory.   How to deploy the SAS Viya platform with an external OpenSearch The deployment instructions are provided in the same README file as the requirements.   To use an external OpenSearch instance, you need to gather the following parameters from the OpenSearch administrator: Username and Password of an account with administrative privileges for your external OpenSearch instance URL (and port) of your external OpenSearch instance NOTE: If the connection uses TLS and OpenSearch is using custom-signed certificates, you also need the server CA certificate so that you can import it into SAS Viya. This step is the same as with any other external CA certificate, whether it is used to protect a connection to OpenSearch, a database, an LDAP server, etc. You can follow the official instruction in the document Incorporate CA certificates into SAS Viya   After you have those connection details, follow these steps: Copy the provided external-opensearch-transformer.yaml , secret.yaml ,and  client-config-tls.yaml  files from: $deploy/sas-bases/examples/configure-elasticsearch/external/ to $deploy/site-config/external-opensearch/ . No need to change anything in its content. Adjust the username, password, and URL values in the latter two files as instructed by the in-line comments. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     Edit the base kustomization.yaml to reference these files: add external-opensearch-transformer.yaml to the transformers block transformers: ... - site-config/external-opensearch/external-opensearch-transformer.yaml add the secret.yaml and client-config-tls.yaml to the resources block resources: ... - site-config/external-opensearch/client-config-tls.yaml - site-config/external-opensearch/secret.yaml Start the SAS Viya platform deployment with your preferred method. And finally, a reminder: if you copied your base kustomization.yaml from an environment that uses an internal OpenSearch instance, be sure to remove any existing lines referencing the internal instance, otherwise you will get an unsuccessful deployment!   Conclusion   In this article we have described the benefits and considerations of the new option that you have when deploying the SAS Viya platform starting with the 2023.03 release, the support of an external instance of OpenSearch to provide searching and indexing capabilities. We have also listed the steps required to perform a deployment connecting to an external OpenSearch instance.   Find more articles from SAS Global Enablement and Learning here. ... View more

Starting with the 2023.02 SAS Viya release, there are small but significant changes to the process required to configure SAS temporary storage (SASWORK). In this post, we will delve into the implementation details and the steps you must follow to keep your configuration current.   What has not changed   Let’s start by listing what has not changed. The performance requirements for SASWORK have not changed. The architectural considerations are still the same.   In short, the two main reasons to use a separate, high-I/O throughput volume for SASWORK are still:   To improve the performance of SAS compute processing. To protect other volumes in your deployment from being filled up if users fill the disk used by SASWORK.   There are plenty of articles on these topics, see for example:   Some SASWORK storage options for SAS Viya on Kubernetes Using generic ephemeral volumes for SASWORK storage on Azure managed Kubernetes (AKS) SAS Viya temporary files (SASWORK and CAS Disk Cache) in Azure   How to apply the customization   The high-level steps needed to perform this customization have not changed as well. You can find official instructions in the README file $deploy/sas-bases/docs/ sas_programming_environment_storage_tasks.htm .   In summary:   Copy the change-viya-volume-storage-class.yaml  example from sas-bases to site-config Replace the {{ VOLUME-STORAGE-CLASS }} placeholder with the desired storage class (i.e., hostPath) Add a reference to this file into the base kustomization.yaml Apply the new configuration to your cluster using the same method originally used to deploy SAS Viya   No need to restart any pod. The next time a compute server will start, it will use the new definition.   What has changed, then? Digging a bit, you’ll find that the differences from previous releases lie in the content of the sample file change-viya-volume-storage-class.yaml and how to reference it in the base kustomization.yaml (i.e., points 1) and 3)).   Let’s see both in detail.   The patch file   Here is a comparison of the content of the sample file change-viya-volume-storage-class.yaml, before and after the change:   Up to SAS Viya 2023.01: Kustomize Patch Starting with SAS Viya 2023.02: Kustomize Transformer Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   As you may notice, although the syntax is quite different, both the old and the new methods perform the same 2 actions:   Delete the current definition of the viya volume Define a new viya volume using the storage class you will place in the placeholder {{ VOLUME-STORAGE-CLASS }}   Kustomization.yaml   The second change is how to reference that custom file in the base kustomization.yaml. Here is a partial printout to show a comparison of the old and new syntax:   Up to SAS Viya 2023.01: add a new entry in the patch section Starting with SAS Viya 2023.02: add a reference to the custom file in the transformers section     Notice that the old method placed the "target" lines with a PodTemplate selector in the patch section of kustomization.yaml, while, with the new syntax, the same goes in the change-viya-volume-storage-class.yaml file (twice, once per Transformer)   Also, note that the new instructions require placing the line referencing the custom change-viya-volume-storage-class.yaml before the existing line ' - sas-bases/overlays/required/transformers.yaml ', as shown above.   Why this change?   Given that the end result is still the same as with previous releases, you may wonder what's the reason for this change. Well, it all has to do with a change in the supported version of the tool used to parse the configuration directory, Kustomize.   Since the release of SAS Viya 2020, we’ve always been using the same tried and true Kustomize 3.7.0. The time for an upgrade was so due, that we did it twice in a row. SAS Viya stable 2023.02 requires Kustomize 4.5.7, while SAS Viya 2023.03 and later releases (both stable and LTS) require Kustomize 5.0.0. (Note: this is accurate as of the time of writing. Always refer to the official requirements for the correct version for your release)   Kustomize dropped support for the old syntax a few versions ago, so we had to change to avoid parsing errors.   Am I affected by this change?   Let’s start with the easy answer. If you have an existing environment where you accepted the default configuration, nothing has changed there, so you are not affected.   If you are going to deploy or change an environment for the first time, then you have to use the syntax following the instructions included in your release. If this is SAS Viya stable 2023.02 or later, that means the new instructions: no need to care about what earlier SAS Viya releases required.   The only case where you should care is if you have an existing environment, deployed with a release older than 2023.02 (either stable or LTS) where you applied the old-style patch to configure a custom location for SASWORK. Even in this case, the runtime environment will not be affected if you leave it "as is": users can still use their applications with no errors. But, as soon as you try to upgrade the installation to the 2023.02 release or later, the upgrade process will fail if you leave the previous file and configuration in place. You have to change the custom change-viya-volume-storage-class.yaml that was placed under your site-config directory, then change the base kustomization.yaml, as shown above. If you do not, you may initially not get any error, but the resulting manifest (site.yaml) will be invalid. The old custom files, used with kustomize 4 or later, will output PodTemplates totally missing any SASWORK definition:       Comparing a “default” manifest (right) with one created with the old-style patch and kustomize 5.0.0 (left)   As soon as you try to apply that incorrect manifest, the operation will fail with errors such as:   Error from server (Invalid): error when applying patch: ... Resource: "/v1, Resource=podtemplates", GroupVersionKind: "/v1, Kind=PodTemplate" Name: "sas-batch-cmd-pod-template", Namespace: "viyaorch" for: "site.yaml": PodTemplate "sas-batch-cmd-pod-template" is invalid: template.spec.containers[0].volumeMounts[1].name: Not found: "viya" Conclusion   Starting with the 2023.02 SAS Viya release, there have been small but significant changes to the steps required to configure SAS temporary storage (SASWORK). While the high-level steps have not changed, there are differences from previous releases in the content of the sample file change-viya-volume-storage-class.yaml and how to reference it in the base kustomization.yaml. The changes are required by the newer version of Kustomize used with recent SAS Viya releases, and the end result is still the same as with previous releases. Be sure to check and update your existing customizations to avoid any errors!   Find more articles from SAS Global Enablement and Learning here. ... View more

In the first article of the series, we have seen how to leverage the OpenShift Local Volume Operator to make Azure VM ephemeral disks available to the pods running in the OpenShift Container Platform cluster. Now that this is done, the next step is to use them for SAS Viya temporary storage.   In this article we will focus on CAS: how can you configure CAS to use the local storage backed by the Azure ephemeral disk as your CAS DISK CACHE?   A quick summary   Here is the TL;DR recap of where we left in the previous article:   We are working in a test environment based on OpenShift deployed on Azure virtual machines The virtual machines we selected provide a fast local SSD disk that we want to use for temporary storage We configured those disks and the cluster so that the OpenShift Local Storage Operator creates a PersistentVolume (PV) on each node to reference them, through a custom Storage Class called `sastmp` .  A PersistentVolume referencing an Azure ephemeral disk as configured in the last article.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The process   Just as with every endeavor, let’s plan our steps before starting the journey. Configuring CAS DISK CACHE is described in the official documentation, so it will be quite simple:   Verify the current (default) configuration Configure CAS DISK CACHE Resolve issues Verify the new configuration   Since we are in a test environment, we use a “crash test method”: perform the changes, apply, and, if something does not work, fix the issues. In a production environment, steps 2 and 3 should be inverted: you should find and fix issues in your dev/test environments so that you can apply the right sequence in production.   1. Verify the current (default) configuration   From the official documentation we know that, by default, CAS servers are configured to use a directory that is named `/cas/cache` on each controller and worker pod. This directory is provisioned as a Kubernetes emptyDir and uses disk space from the root volume of the Kubernetes node. With the default CAS DISK CACHE configuration, a CAS pod uses an empyDir that points to the node root disk.   We can run some code in SAS Studio to verify this setup so that we will be able to confirm the changes after applying the new configuration. No need to reinvent the wheel, we can take an example from @Rob Collum’s article A New Tool to Monitor CAS_DISK_CACHE Usage. Here is the code: cas sess1; proc cas; session sess1; accessControl.assumeRole / adminRole="superuser"; builtins.getCacheInfo result=results; print results.diskCacheInfo; run; quit; cas sess1 clear; Sample code in SAS Studio to show the default CAS configuration.    The Results page confirms our knowledge: The CAS disk cache is located at the path `/cas/cache` (This is the path as seen by the CAS process inside the pods). On all nodes of our test environment (controller, workers), that path resides on a disk with ~127GB of capacity, on a device called `sda4` or `sdb4` . As we have verified in the previous article, these correspond to the OS root disks. This indirectly confirms that we are using a default `emptyDir` . 2. Configure CAS DISK CACHE   We will follow and adapt the instructions from the official documentation to configure CAS so that the CAS Disk Cache will use the local temporary storage we configured in the previous article.   The example in the "Tune CAS_DISK_CACHE" section shows how to mount a volume of type `hostPath` , but you can choose the type that best suits your needs.   If your environment is based on Kubernetes 1.23 or later, you can leverage the generic ephemeral volumes that became GA in that release. This combines the advantages of using a PersistentVolumeClaim with the advantages of ephemeral storage: You can specify, in the pod definition, a template volume claim that mimics a statically provisioned PVC (such as the one we used in the previous article to test the local storage) OpenShift will take care of managing the PVC lifecycle: it creates a PVC bound to the pod when the pod starts and deletes the PVC when the pod is terminated. This is perfect for temporary storage, such as CAS DISK CACHE!   Here is a sample yaml transformer, based on the one provided in the official documentation, that defines a generic ephemeral volume that references our `sastmp` Storage Class. # # this defines the volume and volumemount for CAS DISK CACHE location --- apiVersion: builtin kind: PatchTransformer metadata: name: cas-cache-custom patch: |- - op: add path: /spec/controllerTemplate/spec/volumes/- value: name: cas-cache-sastmp ephemeral: volumeClaimTemplate: spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "sastmp" resources: requests: storage: 100Gi - op: add path: /spec/controllerTemplate/spec/containers/0/volumeMounts/- value: name: cas-cache-sastmp mountPath: /cas/cache-sastmp # # mountPath is the path inside the pod that CAS will reference - op: add path: /spec/controllerTemplate/spec/containers/0/env/- value: name: CASENV_CAS_DISK_CACHE value: "/cas/cache-sastmp" # # This has to match the value that is inside the pod target: version: v1alpha1 group: viya.sas.com kind: CASDeployment # # Target filtering: chose/uncomment one of these option: # # To target only the default CAS server (cas-shared-default) : labelSelector: "sas.com/cas-server-default" # # To target only a single CAS server (e.g. MyCAS) other than default: # name: {{MyCAS}} # # To target all CAS Servers # name: .*   After creating the cas-cache-custom transformer, the process to apply it is the same as any other customization. Since it depends on the method originally used to deploy SAS Viya, please see the SAS Viya official documentation referenced above for the exact steps.   The next step now is to stop and restart the CAS server to pick up the new configuration. It is enough to terminate the CAS server pods: the CAS Operator will take care of restarting them. You can use a quick command such as:   oc -n delete pods -l app.kubernetes.io/managed-by=sas-cas-operator 3. Resolve issues   If you followed this far, you are now expecting to have a working environment with CAS using the locally attached disk as its temporary storage. And that could be true if we were using any other Kubernetes platform.   Yet, as of February 2023, we may still need a crucial step to get there on OpenShift.   After applying the changes above and terminating CAS pods, you may find that the CAS operator is unable to restart your CAS server. The sas-cas-operator pod log may contain errors such as: "sas-cas-server-default-controller" is forbidden: unable to validate against any security context constraint .   What’s going on?   Simply put, OpenShift Is doing one of its jobs: enforcing a strict security policy.   Before installing SAS Viya, as a preliminary step, the cluster administrator created some security context constraints as instructed by the Preparing for OpenShift documentation page. One of these describes the permissions granted to CAS, including a list of permitted volume types: anything else gets automatically denied.   Assuming you used the sas-cas-server security context constraint, you can check what volumes it permits with the following code:   oc get scc sas-cas-server -o jsonpath='{.volumes}'   By default, you should get a listing like the following: notice that the "ephemeral" volume we tried to use is not there.   ["configMap","downwardAPI","emptyDir","nfs","persistentVolumeClaim","projected","secret"] In summary, the missing step is to add ephemeral volumes to the list of permitted entries in the CAS security context constraint. Here is a simple command to do it:   oc -n get scc sas-cas-server -o json | jq '.volumes += ["ephemeral"]' | oc -n apply -f -   This one-liner uses the OpenShift CLI to read the current definition of the sas-cas-server security context constraint and feed it into the `jq` tool which appends the entry "ephemeral" to the array of permitted volumes. Then, `jq` feeds the result back into another OpenShift CLI that re-loads the updated definition into the cluster.   This should be all. At this point, OpenShift should successfully validate the CAS controller request to allocate ephemeral volumes for the CAS pods, and the CAS server should be automatically started.     The ephemeral disk is now used by the CAS pod 4. Verify the new configuration   As a last step, we just have to verify that the new setting is actually working (trust but verify!).   We can use the same code in SAS Studio as we did initially. The new result should give us the information we are looking for: The CAS disk cache is now located at the new path `/cas/cache-sastmp` , as we specified. This is the path as seen by the CAS process inside the pods. On all nodes (controller, workers), that path resides on a disk with ~299GB of capacity, on a different device than the previous run. We can recognize that these correspond to the Azure temporary SDD disks. Bigger, faster. SUCCESS!   Sample code in SAS Studio to show the updated CAS configuration. Conclusion   In this series of articles, we are discussing how to leverage locally attached ephemeral disks for pods running in an OpenShift Container Platform cluster. They are a convenient choice for SAS Viya temporary storage and, in this article, we have seen how to configure CAS_DISK_CACHE to use them as Kubernetes generic ephemeral volumes.     ... View more

SAS compute sessions and CAS servers require temporary storage for multiple purposes. Choosing a suitable location for SAS Viya temporary storage can be a key tuning step for optimal production deployments. In today’s cloud environments, this means addressing multiple layers of abstraction, from the underlying infrastructure, to the Kubernetes cluster, up to the applications running inside it.   In this series of articles, we explore how to provision local temporary storage to a Red Hat OpenShift Container Platform cluster deployed on Azure, and how to leverage it in SAS Viya.   Introduction   Storage design can differ considerably between infrastructure providers and from cluster to cluster. To successfully deliver a working environment, by default, SAS Viya compute servers are configured to use a minimum common denominator for temporary storage that is guaranteed to be always available. Both SAS compute sessions and CAS servers leverage a directory provisioned as a Kubernetes emptyDir, which uses disk space from the root volume of the Kubernetes node where they are running.   This default configuration is acceptable for testing and evaluation, but not for production workloads. If disk space in the root volume of the node becomes low, then Kubernetes begins evicting pods, and, in extreme circumstances, the node itself can crash.   SAS recommends using fast storage for temporary locations. Often, the best performance is provided by using disks that are locally attached to the node, such as NVMe or SSD disks. Since this is temporary storage, the disks that are used do not need to persist beyond the duration of each pod, nor follow the pod if it is moved to a different node. For this reason, ephemeral storage is ideal.   Most cloud providers offer virtual machines that include a temporary disk for ephemeral storage. In this article, we will walk through the steps we used to provision Azure ephemeral disks as a temporary location for the pods in our OpenShift Container Platform cluster. A note of caution   These articles are the result of testing within the environments we use for the “SAS Viya 4 - Deployment on Red Hat OpenShift Container Platform” workshop. For our convenience, we deploy on-demand OpenShift on Azure clusters; although that works fine for our objectives, this infrastructure is not included in the official SAS Viya system requirements. So, why discussing this in a public post? Because it can still provide value for multiple situations:   although SAS Viya on OpenShift is currently fully supported only on VMware vSphere, it is possible to deploy it on different infrastructure providers, such as Azure, under the “SAS Support for Alternative Kubernetes Distributions” policy. the high-level process and steps are similar across different infrastructures. If your server has an available fast disk, you can use it as described here, independently from how the disk itself was provisioned. Some details may need to be adjusted from the example commands shown here, but it should all be manageable by a Kubernetes admin. finally, maybe you end up reading this post and decide it can be useful to you for a different use case on a similar infrastructure.   In summary, we are not stating that your SAS Viya cluster should be designed as described here, nor endorsing this architecture as fully supported.   Provisioning temporary storage   Let’s restart from the beginning. What is the objective? To provide via OpenShift some dedicated, fast storage that can be used for SAS and CAS working directories (SASWORK and CAS DISK CACHE).   Verify the current disk status   In our environment, we are using the Azure cloud as the underlying infrastructure. @Hans-Joachim Edert discusses a similar environment in his article Some SASWORK storage options for SAS Viya on Kubernetes, listing in the “hostPath” section some Microsoft virtual machine types that can provide the storage that fits our needs. All the virtual machines used by our cluster fall in this category and, according to Microsoft’s documentation, they have an additional internal SSD drive that could be used to host temporary storage. But there is a catch: Azure VMs automatically mount this drive to their OS at the /mnt path and, if we were using AKS, this would be a mount point which could be directly used as Kubernetes storage. With OpenShift things are slightly different.   OpenShift uses a dedicated operating system – Red Hat Enterprise Linux CoreOS (RHCOS) – which, by default, does not automatically mount these disks. We know from the documentation that the disks are there, but they are almost invisible. OpenShift provides a great tool for cluster administrators to quickly interact with the underlying RHCOS: the oc debug command. Using this tool, we can operate in a privileged session, just as if we connected to the node via ssh as the root user.  Here is an example of how to use it to verify the disks available on the first worker node using the lsblk command:   # Find the 1st worker node's name WORKER=$(oc get nodes -l "node-role.kubernetes.io/worker" -o name | head -1) echo ${WORKER} # Query disk status using the lsblk command in a debug session oc debug ${WORKER} # now we are in the debug container chroot /host lsblk -o NAME,SIZE,FSTYPE,LABEL,MOUNTPOINT # exit the debug container exit exit   As an example, here is the output in our test environment:   $ # Find the 1st worker node's name $ WORKER=$(oc get nodes -l "node-role.kubernetes.io/worker" -o name | head -1) $ echo ${WORKER} node/itaedr-r-0097-worker-1 $ # Query disk status using the lsblk command in a debug session on the node $ oc debug ${WORKER} Starting pod/itaedr-r-0097-worker-1-debug ... To use host binaries, run `chroot /host` Pod IP: 10.255.1.8 If you don't see a command prompt, try pressing enter. sh-4.4#  # now we are in the debug container sh-4.4#  chroot /host sh-4.4#  lsblk -o NAME,SIZE,FSTYPE,LABEL,MOUNTPOINT NAME     SIZE FSTYPE LABEL             MOUNTPOINT sda      128G |-sda1     1M |-sda2   127M vfat   EFI-SYSTEM |-sda3   384M ext4   boot              /boot `-sda4 127.5G xfs    root              /sysroot sdb      300G `-sdb1   300G ntfs   Temporary Storage sdc        1G ext4                     /var/lib/kubelet/pods/08cef6f4-8032-449d-bcea-309355cb383c/volumes/kubernetes.io~azure-disk/pvc-fa675cfb-cd0d-49e6-843e-44adec80241fsr0 sh-4.4#  # exit the debug container ...   Let’s try to interpret this output. We should be able to understand what disks are available to this node, and to the pods running there.   The first disk is sda . This has some partitions mounted on the node OS. In the above output, sda4 is the OS root, that is seen by the debug pod as /sysroot. The disk size is 128GB, which corresponds to what we requested when we created this Azure virtual machine. The second disk is sdb and it does not show any mountpoint. It contains a single partition sdb1 ; from its label Temporary Storage we can understand that this is the temporary disk that Azure provides. Its size depends on the virtual machine type used to instantiate this node (Not all Azure machine types include a temporary disk!). The FSTYPE column show that this disk, although currently not used, contains a partition of type ntfs, which is a Windows filesystem. There can be additional disks; these are dynamically created by OpenShift as requested by running pods PVCs. In our environment the default storage class is managed-premium (kubernetes.io/azure-disk), so, for each PVC, OpenShift creates a dedicated Azure Disk. In the example above, sdc is a 1GB disk mounted on the pod with id 08cef6f4-8032-449d-bcea-309355cb383c as a volume named pvc-fa675cfb-cd0d-49e6-843e-44adec80241f   👉 on each worker node, the names sda and sdb may be switched, because linux does not guarantee the order in which disks are configured at boot time.     A diagram of the disks available on a node in our test environment Prepare the temporary disk   As we have seen above, the temporary disk, although currently not used, contains a partition of type ntfs, which is a Windows file system. In the next steps, we are going to use tools that expect/support Linux file systems. Since this is a temporary empty partition, we can overwrite it and create a new empty partition with a Linux xfs file system.   Again, we can use an OpenShift debug container, this time on all worker nodes, to re-format the temporary partition with the desired xfs file system:   # fix the partition on the temporary disk on all worker nodes AZURE_TMP_DEV=/host/dev/disk/azure/resource for _NODE in $(oc get nodes -l node-role.kubernetes.io/worker -o name); do   # run inside the debug privileged container   oc debug -q ${_NODE} -- bash -c "\     parted ${AZURE_TMP_DEV} --script -- \       mklabel gpt \       mkpart xfs 1MiB -2048s ;\     sleep 15;\     lsblk ${AZURE_TMP_DEV};\     mkfs -t xfs -f ${AZURE_TMP_DEV}-part1 \   " done   Within the previous commands we are using a trick to overcome the issue that, on some nodes, the temporary disk is sdb , while on other nodes it is sda : the node OS automatically creates a link called /dev/disk/azure/resource that points to the correct disk, and another link called /dev/disk/azure/resource-part1 that points to the first partition inside that disk. To use these links from inside the OpenShift debug container, we have to prefix /host to their paths.   Deploy the Local Storage Operator   To leverage the Azure temporary disks, it is possible to manually create local Volumes using standard Kubernetes practices, but that would require the cluster administrator to fully manage lower-level storage lifecycle operations.   OpenShift simplifies local storage management thanks to the Local Storage Operator.   This operator is not installed by default. We can install the Local Storage Operator following RedHat instructions using the OperatorHub from the web console.     Provision local storage using the Local Storage Operator   We can now use the Local Storage Operator to provision local storage for SAS Viya pods. The operator can create Persistent Volumes by looking for available file systems at the paths specified in a Local Volume custom resource. Here are a couple of yaml examples, that leverage the xfs file system previously created at /dev/disk/azure/resource-part1 :   Create a Local Volume, available on all worker nodes (OpenShift controller nodes are excluded by default). apiVersion: "local.storage.openshift.io/v1" kind: "LocalVolume" metadata:   name: "lv-sastmp"   namespace: "openshift-local-storage" spec:   storageClassDevices:     - storageClassName: "sastmp"       volumeMode: Filesystem       fsType: xfs       devicePaths:         - /dev/disk/azure/resource-part1 Create a Local Volume, that only uses a subset of nodes, i.e. only the SAS CAS nodes. In this case, the yaml definition should include a nodeSelector. As an example, you could filter on the workload.sas.com/class=cas label: apiVersion: "local.storage.openshift.io/v1" kind: "LocalVolume" metadata: name: "lv-sastmp-cas" namespace: "openshift-local-storage" spec: nodeSelector: nodeSelectorTerms: - matchExpressions: - key: workload.sas.com/class operator: In values: - cas storageClassDevices: - storageClassName: "sastmp" volumeMode: Filesystem fsType: xfs devicePaths: - /dev/disk/azure/resource-part1   You can create the Local Volume resource in the OpenShift cluster by pasting the yaml code in the web console, in the Local Storage operator page, or by saving the content in a file and then applying it: oc apply -f localVolume.yaml   The operator should automatically create the storage class referenced in the Local Volume, if it did not already exist. Then, it should start a pod to manage the storage on each node. Finally, it should create a Persistent Volume for each matching disk discovered on each node.     The disk view after defining a Local Volume that references the ephemeral disk   Test the newly referenced local storage   It's always a good practice to test your infrastructure before using it in SAS Viya – or in any production software!   Local volumes are accessed by pods through Persistent Volume Claims (PVCs).   Let's create a PVC that references our local volumes, and a pod to test the storage, with the following two yaml definitions:   kind: PersistentVolumeClaim apiVersion: v1 metadata:   name: localpvc-sastmp spec:   accessModes:   - ReadWriteOnce   volumeMode: Filesystem   resources:     requests:       storage: 100Gi   storageClassName: sastmp kind: Pod apiVersion: v1 metadata:   name: gel-test-pod spec:   containers:   - name: gel-test-pod     image: gcr.io/google_containers/busybox:1.27     command:       - "/bin/sh"     args:       - "-c"       - "df -h | grep sd && touch /sastmp/SUCCESS && ls -l /sastmp/SUCCESS && exit 0 || exit 1"     volumeMounts:       - name: sastmp         mountPath: "/sastmp"   restartPolicy: "Never"   volumes:     - name: sastmp       persistentVolumeClaim:         claimName: localpvc-sastmp   You can notice in the pod definition that we reference the PVC just created (localpvc-sastmp) and mount it at the /sastmp path inside the container, so that we can use some commands to test reading and writing from that location.   The final step is to create these resources in the OpenShift cluster. The PVCs should be in the same namespace as the pods that will use them; we can specify the SAS Viya namespace:   oc apply -f localPVC.yaml -n gel-viya oc apply -f localTestPod.yaml -n gel-viya   To check the successful allocation of the disk, we can read the log of the test pod:   oc logs gel-test-pod -n gel-viya   The test commands that we used in the pod definition should output something similar to the following:   /dev/sdb1               299.9G      2.1G    297.7G   1% /sastmp /dev/sda4               127.5G      9.2G    118.3G   7% /etc/hosts /dev/sda4               127.5G      9.2G    118.3G   7% /dev/termination-log -rw-r--r--    1 root     root             0 Jan 13 23:31 /sastmp/SUCCESS   This shows that the ~300GB temp disk has been mounted at the /sastmp path as requested, and the pod was able to successfully write there a test file called "SUCCESS".     A final view showing the ephemeral disk used in the test pod   Conclusion   In this first article, we have seen how to leverage OpenShift Local Volume Operator to make Azure VM ephemeral disks available to the pods running in the OpenShift Container Platform cluster. Now that this is done, the next step will be to use them for SAS Viya temporary storage. You can read it in the next article: SAS Viya Temporary Storage on Red Hat OpenShift – Part 2: CAS DISK CACHE    Find more articles from SAS Global Enablement and Learning here. ... View more

Starting with the 2022.10 release, SAS Viya uses Redis to provide a distributed cache technology. Redis Server and the Redis Operator replace Apache Geode and SAS Cache Server. What are these components? And how do they impact SAS Viya architecture?   A brief history   As the SAS 9 platform evolved into a clustered, multi-tier architecture, some services deployed on the middle-tier and server-tier required a distributed cache to share run-time information between the multiple web application server instances. This was implemented by embedding some java components, provided by the Apache Geode project, into SAS Web Applications, then running an instance of the Cache Locator on each machine. The Cache Locator was used by the applications to locate other members and form a distributed data cache/data store.   With the evolution from monolithic SAS 9 applications to SAS Viya 3 stateless micro-services, it became clear that the distributed cache could not be embedded anymore in these services and had to be externalized. The SAS Cache Server was born, still based on the Apache Geode java objects, and the Cache Locator is still there to support it. Together, SAS Cache Locator and SAS Cache Server provide a distributed cache technology to microservices in SAS Viya 3.   Finally, with the advent of the cloud-native SAS Viya platform deployed in Kubernetes, these components have been adapted to run in the new environments, but they were not designed to be cloud-native. The new platform exposed some deficiencies, including susceptibility to network instability, the lack of support for new services written in the Go language, and the lack of a Kubernetes operator to manage their lifecycle.     Enter Redis   Until the 2022.09 release, SAS Viya includes two Kubernetes statefulsets, one for the SAS Cache Server and one for the SAS Cache Locator, running two pods each to ensure High Availability.   Starting with the 2022.10 release, these pods are deployed but not started. Finally, with the 2022.11 release, SAS Cache Server and SAS Cache Locator are completely removed from SAS Viya.   Starting with the 2022.10 release, SAS Viya uses Redis to provide a distributed cache technology: Redis Server and the Redis Operator replace Apache Geode and SAS Cache Server.   SAS Redis server, based on the open-source Redis project, provides a distributed cache to SAS Viya microservices. SAS Redis Operator is a Kubernetes operator, developed by SAS, that manages the life cycle of the SAS Redis Server.   The default deployment creates a Redis Cluster with 3 Kubernetes Statefulsets, each one including a controller and a replica node. The cluster is managed by the operator, which runs in a stateless pod.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Redis pods as seen in Lens   The cached data is sharded (distributed according to a hash algorithm) between the 3 controllers, then replicated. In case of a controller failure, after a timeout, the corresponding replica is automatically promoted to controller and the cluster can still service its clients.     The current implementation does not support scalability: if you change the number of nodes, it does not reallocate the cached data, and the clients cannot find it anymore. For this reason, the number of controller nodes is currently hard-coded to 3 in the definition of the distributedredisclusters custom resource:   $ kubectl get crd distributedredisclusters.redis.kun -o json | jq .spec.versions[].schema.openAPIV3Schema.properties.spec.properties.masterSize { "default": 3, "format": "int32", "maximum": 3, "minimum": 3, "type": "integer" }   Although it is possible to scale up the number of replicas, no official performance testing has been conducted so far; the default implementation should be sufficient for most deployments.     Conclusion   In this article, we have introduced the new SAS Redis Server and SAS Redis Operator available with SAS Viya 2022.10 and later. For more information about the management of SAS Redis Server and SAS Redis Operator, you can consult the updated SAS Viya Administration: Infrastructure Servers page.   Find more articles from SAS Global Enablement and Learning here. ... View more

A previous article describes how to configure SAS Viya to leverage an external instance of PostgreSQL for the SAS Infrastructure Data Server. After reading it, you may still wonder: “Why would I choose to leverage an external PostgreSQL instead of going with the default internal one provided by SAS?”. Well, I’m glad you asked: here you can find some pros and cons we collected during multiple architecture design workshops.   One of the design principles of SAS Viya is to give customers more choices and control over their desired architecture. This starts with supporting many Kubernetes cluster providers: you can run your cluster on local machines or in a cloud platform, including supported versions of Microsoft Azure, Amazon Web Services, Google Cloud Platform, and Red Hat OpenShift. Deployments with open-source Kubernetes are also supported. We can easily say this is the modern incarnation of the decades-old Multivendor Architecture!   As soon as the first customers started deploying SAS Viya on their Azure clouds (that was the first provider SAS supported back in 2020), they asked about the possibility to leverage their existing native cloud instances for some of the services that SAS Viya requires. Supporting external instances of PostgreSQL answers that requirement, and, in the future, additional services will be able to leverage native cloud implementations. Examples may include Azure Service Bus instead of the SAS Viya-provided RabbitMQ for message queues, or external instances of OpenSearch.   This already gives us the first, possible answer to the original question: using an external instance of PostgreSQL can be a good choice if you already have one! You should still consider all of the system requirements, to make sure that SAS Viya can leverage what you are already using.   Apart from this obvious case, what other considerations should you weigh when deciding to use a PostgreSQL instance not provided by SAS?   Here is a non-comprehensive list gathered from discussions with customers and colleagues during multiple architecture design workshops.   Do you have the required competency to properly manage a PostgreSQL database? In other words, who is responsible for maintenance? With an external instance, this can be the cloud provider, in case of managed instances, or it can be your IT department for self-hosted installations. With an internal instance, it’s all provided by SAS and managed by the SAS administrator. SAS documentation provides instructions for general management of the included server and services, with a specific section for PostgreSQL administration. Can the instance be configured as required by SAS Viya? Obviously, the chosen PostgreSQL instance should be included in the list of supported versions and distributions, but that’s only the first check. SAS Viya has specific requirements including user permissions, available database extensions, configuration settings (a very important one is the maximum number of concurrent connections), and so on. Make sure that these requirements do not conflict with the settings required by other applications that may be using the same external database instance. Can it handle the increase or decrease in your workload? The SAS-provided internal instance supports tuning both at the infrastructure level (number of database replicas, CPU, and memory allocated per pod) and at the database level, allowing you to tune the server to meet expected workloads. For the complete list of available PostgreSQL configuration parameters, see https://www.postgresql.org/docs/12/config-setting.html Does it support High Availability, Non-Disruptive-Updates? Sometimes the devil is in the details. Some external PostgreSQL distributions claim High-Availability support, but still leverage a single PostgreSQL replica. That means the underlying infrastructure (VM, storage, network, ...) has guaranteed SLAs, but, when the single instance is shut down for any reason – planned or unplanned – all client applications (including SAS Viya) are forced to stop. The internal instance is deployed by default with multiple replicas; if the primary server dies or is stopped for maintenance, one of the replicas is promoted to become the new primary, and service is resumed – it all happens automatically in a few minutes, with no forced stop or restarts. How much does it cost? If you are already running an external instance of PostgreSQL, you may be tempted to answer “almost nothing”; did you remember to consider any additional HW/storage that needs to be allocated for the increased demand? If instead, you have to deploy a new dedicated instance, it can cost you thousands of dollars per month in your cloud bill. The internal instance is co-located on the SAS Viya Kubernetes cluster, so it shares the costs with the rest of the infrastructure. Does your SAS Viya environment include any solution that currently does not support an external PostgreSQL database? If you are using any of the solutions listed on this system requirements page, you have no choice but to use the SAS-provided PostgreSQL: those offerings do not support an external PostgreSQL database instance. Support statements may change with newer releases, so be sure to always check that page before any new deployment to get the most current information. ​ Conclusion SAS Viya gives you many choices and control over your desired architecture. In this article, you can find some of the considerations gathered from discussions with customers and colleagues to evaluate whether to use an internal or external instance of PostgreSQL for your SAS Viya deployment. Do you have any additional points that we may have missed? We’re always open to discussion and eager to learn, so share them in the comments!   Find more articles from SAS Global Enablement and Learning here. ... View more

The SAS Infrastructure Data Server stores SAS Viya user content, such as reports, authorization rules, selected source definitions, attachments, and user preferences. With Viya 3 it was necessarily an internal component of the platform; starting with SAS Viya 2020, this component can also be replaced by an external service. The server requirements and configuration settings required to use an external instance have changed over time; in this article, we describe the most current ones as of release Stable 2023.04.   Two Available Options   The SAS Infrastructure Data Server requires an instance of PostgreSQL. By default, the deployment installs and configures its own opensource PostgreSQL database inside the SAS Viya Kubernetes cluster; this is referred as an internal instance. SAS deploys a comprehensive PostgreSQL container operator suite provided by Crunchy Data for this purpose.  If you want, you can choose instead to provide another PostgreSQL instance, which is called an external instance.   The choice to use an internal or external instance should be made before deploying the software. Once you have decided to use the external PostgreSQL database, there is no turning back: after the deployment has been completed with the external PostgreSQL, it is not possible to change to use an internal PostgreSQL (it is also true in the other way around). If you change your mind, you will have to unconfigure and re-deploy from zero.     External PostgreSQL Requirements   SAS Viya supports managed PostgreSQL instances hosted by a cloud provider, or directly configured and maintained by you: PostgreSQL (Open Source) Microsoft Azure Database for PostgreSQL - Single Server (PostgreSQL 11 only) Microsoft Azure Database for PostgreSQL - Flexible Server Amazon RDS for PostgreSQL GCP: Cloud SQL for PostgreSQL   PostgreSQL versions 11 - 14 are the currently supported versions. SAS Viya platform 2022.10 or later is required if you want to use PostgreSQL 13, while SAS Viya platform 2023.03 or later is required for PostgreSQL 14.    The PostgreSQL server should be set up and running before starting the SAS deployment. The SAS Viya Operations document, in the System Requirements section, lists all the requirements such as role permissions and configuration settings. The most important one to consider for proper instance sizing is supporting a maximum number of connections and max_prepared_transactions of at least 1024. Please refer to the links in the appendix to find the proper PostgreSQL instance types that can satisfy that requirement for each supported cloud vendor.     How to Configure SAS Viya to Use an External Instance of PostgreSQL   Once the external PostgreSQL server has been identified, configured, and confirmed to be running, it’s time to deploy SAS Viya with the proper settings to use it. Step-by-step instructions are provided in the README file at $deploy/sas-bases/examples/postgres/configure/README.md (for Markdown format) or at $deploy/sas-bases/docs/configure_postgresql.htm (for HTML). Starting with the release Stable 2022.10 there have been breaking changes from previous instructions, and older configuration files cannot be reused. The good news is that the actual process has not changed much: find the connection details, put them in the proper configuration files, and reference those files in SAS Viya base kustomization.yaml.   Find the Connection Details   To be able to connect to an external PostgreSQL you will need the following parameters from the database administrator:   Database role (i.e. username) and password Database name (optional) Server host name and port   The database name parameter should point to the database inside the PostgreSQL server SAS Viya will use; if omitted, it defaults to "SharedServices".   SAS recommends securing the connection between SAS Viya and the external database using TLS encryption. If the certificate used is provided by a well-known Certificate Authority, then SAS Viya will automatically recognize and validate it. Instead, if the certificate is self-signed, your database administrator or cloud provider will have to provide the server CA certificate, which you will then include in the SAS Viya trust store following the procedure outlined in the SAS Viya: Security guide.   Create the Configuration Files   Template configuration files are provided under $deploy/sas-bases/examples/postgres/. Simply copy the postgres-user.env and dataserver-transformer.yaml files to $deploy/site-config/postgres/ and change the placeholders with the actual values collected in the previous step, as guided by the comments in the files.   # Replace {{ DB-ROLE }} with the name of the database role SAS Viya will use. username={{ DB-ROLE }} # Replace {{ DB-ROLE-PASSWORD }} with the password of the database role SAS Viya will use. password={{ DB-ROLE-PASSWORD }}   postgres-user.env   # Example dataserver-transformer to modify DataServer CustomResources. # - In the following code, user-defined values are indicated by a capitalized # and hyphenated name set off by curly braces and a space at each end. To # replace the variable, replace the curly braces, interior spaces, and the # variable name. For instance, {{ DB-NAME }} could be replaced by # SharedServices. # - Replace all instance of {{ DATASERVER-NAME }} with the name of the # DataServer CustomResource you are targeting. Refer to the below list to # find the correct name based on the PostgreSQL server you are modifying. # - For Platform PostgreSQL, use "sas-platform-postgres" # - For CDS PostgreSQL, use "sas-cds-postgres" apiVersion: builtin kind: PatchTransformer metadata: name: {{ DATASERVER-NAME }}-dataserver-transformer patch: |- # # (Optional) Uncomment this section to change which database inside the # # PostgreSQL server Viya will use by default. Defaults to "SharedServices" # # if not specified. Replace {{ DB-NAME }} with the name of the database. # - op: replace # path: /spec/databases/0/name # value: {{ DB-NAME }} # Replace {{ POSTGRES-USER-SECRET-NAME }} with the name of the # secretGenerator containing PostgreSQL user credentials. - op: replace path: /spec/users/0/credentials/input value: secretRef: name: {{ POSTGRES-USER-SECRET-NAME }} usernameKey: username # For internal use, do not modify passwordKey: password # For internal use, do not modify # Replace {{ DB-HOST }} with the host name for the PostgreSQL server. - op: replace path: /spec/registrations/0/host value: {{ DB-HOST }} # Replace {{ DB-PORT }} with the port number for the PostgreSQL server. - op: replace path: /spec/registrations/0/port value: {{ DB-PORT }} - op: replace path: /spec/ssl value: true target: group: webinfdsvr.sas.com kind: DataServer version: v1beta1 name: {{ DATASERVER-NAME }}   dataserver-transformer.yaml   Edit kustomization.yaml   You add PostgreSQL servers to SAS Viya via the DataServer webinfdsvr.sas.com CustomResource. This CustomResource can be configured to reference an external PostgreSQL cluster. In that case, the CustomResource also references a Kubernetes Secret that contains username and password used for the connection. This is accomplished by adding the two files above to kustomization.yaml:   include the dataserver-transformer.yaml file in the transformers block reference the postgres-user.env in the secretGenerator block   Note: be sure to use the same name, for the secretGenerator in kustomization.yaml, as the value you used for the {{ POSTGRES-USER-SECRET-NAME }} placeholder in dataserver-transformer.yaml   transformers: - site-config/postgres/dataserver-transformer.yaml ... secretGenerator: - name: platform-postgres-user envs: - site-config/postgres/postgres-user.env Deploying with SAS Viya 4 IaC tools   Instead of the manual configuration process outlined so far, it is possible to use the SAS Viya 4 Infrastructure as Code GitHub tools, and have them create and use an external database for PostgreSQL.​   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     You have 2 different choices:   Use the viya4-iac-azure, viya4-iac-aws, viya4-iac-gcp, viya4-iac-k8s projects to create a new instance of PostgreSQL. The tools abstract the platform-specific commands and provide many options that can be tweaked to tune the database server. > Note: SAS does not provide any tools to automatically create infrastructure artifacts for Red Hat OpenShift. If OpenShift is your target platform, you can use third-party Operators to create a PostgreSQL instance, or reference a database server running in a different environment. Reuse an existing database server instance.   The next step is then to use the SAS Viya 4 Deployment GitHub tool to deploy SAS Viya with the external PostgreSQL; depending on which of the two previous choices you made, you now have two different paths:   Integrate with the SAS Viya IaC tool. One of the final steps of IaC tools is to output a “state file” that contains all details of the artifacts they created, including PostgreSQL. The SAS Viya 4 Deployment tool can read this state file in input and automatically configure SAS Viya. Integrate with the custom server instance. In this case, you have to provide, in the form of Ansible variables, the same connections details that we highlighted previously: V4_CFG_POSTGRES_SERVERS: default: internal: false admin: password: "" fqdn: server_port: 5432 ssl_enforcement_enabled: true database: SharedServices ​ Conclusion   SAS Viya provides many integration points to leverage external components and services. In this post, we have seen how to create an external PostgreSQL database server and configure SAS Viya Stable 2022.10 and later to use it.   Appendix: Links Microsoft Azure Maximum Database Connections for Single Server: https://learn.microsoft.com/en-us/azure/postgresql/single-server/concepts-limits   Maximum Database Connections for Flexible Server: https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-limits   Viya4-iac-azure configuration defaults:  https://github.com/sassoftware/viya4-iac-azure/blob/main/docs/CONFIG-VARS.md#postgres-servers Amazon AWS AWS PostgreSQL Instance Types: https://aws.amazon.com/rds/instance-types/   Maximum Database Connections:  https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html#RDS_Limits.MaxConnections   Viya4-iac-aws configuration defaults: https://github.com/sassoftware/viya4-iac-aws/blob/main/docs/CONFIG-VARS.md#postgresql-server Google Cloud How to create PostgreSQL instances on Google Cloud: https://cloud.google.com/sql/docs/postgres/create-instance   Maximum Database Connections: https://cloud.google.com/sql/docs/postgres/flags#postgres-m   Viya4-iac-gcp configuration defaults: https://github.com/sassoftware/viya4-iac-gcp/blob/main/docs/CONFIG-VARS.md#postgres-servers Open Source Kubernetes Viya4-iac-k8s configuration defaults: https://github.com/sassoftware/viya4-iac-k8s/blob/main/docs/CONFIG-VARS.md#postgresql-servers Viya4-Deployment   Viya4-deployment PostgreSQL configuration: https://github.com/sassoftware/viya4-deployment/blob/main/docs/CONFIG-VARS.md#postgres   Find more articles from SAS Global Enablement and Learning here. ... View more

Previous articles presented the evolved architecture and capabilities that helped SAS/CONNECT move to dynamic cloud environments with SAS Viya. This time the focus is on the programmers: let's see some sample code that can leverage these new capabilities!   The first step in every program that leverages SAS/CONNECT is to submit a SIGNON statement from a client session to start a remote server session: that remote SAS/CONNECT session will be available to execute SAS code and to upload or download data from/to the client.   From a programmer’s point of view, there are two key information pieces that a client needs to provide – either explicitly written or implied with a default value – to be able to successfully start a SAS/CONNECT session:   where will the remote session start – i.e., what is the backend server name? how does the client session authenticate?   In the following sections, we present some sample code snippets that show how to perform the initial SIGNON to SAS Viya in different cases - from the same Kubernetes clusters, or from an external client; across different releases (SAS Viya and SAS 9.4) – and leveraging different authentication methods: with interactive or hard-coded username and password, using authinfo files, or sharing OAuth tokens.   Spawner Sign-on from Internal Clients   Let’s start with the simplest use case: you are working within SAS Studio in a SAS Viya environment, and you want to start a SAS/CONNECT session in the same cluster, authenticating as yourself. The code is really simple:   /* launch a SAS/CONNECT session on the same cluster */ %let session1=sas-connect-spawner 17551; /* authenticate re-using the already logged-in credentials */ SIGNON session1 user="_OAUTH_";   You can see that we used sas-connect-spawner as the hostname; this maps to the internal Kubernetes services with that name, listening on the SAS/CONNECT default port 17551. Kubernetes takes care of finding the correct pod on the correct host and directs the connection there. The authentication uses the OAuth protocol to sign in as the same user already logged into the client – more on that later.   Spawner Sign-on from External Clients   A more common use case is when you are writing your code in a client executing in a different environment than the cluster where the SAS/CONNECT server will start. For example, you want to connect a legacy SAS 9 environment to a modern SAS Viya cluster running in the cloud.   First, your SAS Administrator should have already configured all the required parts:   Enable the SAS/CONNECT spawner to support external clients: https://go.documentation.sas.com/doc/en/sasadmincdc/v_032/dplyml0phy0dkr/n08u2yg8tdkb4jn18u8zsi6yfv3d.htm#p1n0mb2fhaarjrn0zt9oqo2vprrw Ensure the TLS certificate presented by SAS/CONNECT spawner includes the external hostname used to connect to the server and any desired DNS alias: https://go.documentation.sas.com/doc/en/sasadmincdc/v_032/calencryptmotion/n1xdqv1sezyrahn17erzcunxwix9.htm#p19femzwwlxh39n1vz8g1qnc759y Ensure the client accepts as valid the TSL certificate presented by SAS/CONNECT spawner, for example by adding the ROOT CA certificate into the client truststore: https://go.documentation.sas.com/doc/en/sasadmincdc/v_032/calencryptmotion/n1xdqv1sezyrahn17erzcunxwix9.htm#n0309hwuyq56x2n17mkapfltqau6 and https://go.documentation.sas.com/doc/en/sasadmincdc/v_032/calencryptmotion/n1xdqv1sezyrahn17erzcunxwix9.htm#p09ebu28fz7zpjn17r6jf6iwpyyw   The details of these steps are out of scope for this article, you can reference the linked documentation pages for more info.   Once that’s done, as a programmer you can connect like this:   /* launch a SAS/CONNECT session from outside the cluster */ /* set TLS connection options */ OPTIONS NETENCRYPTALGORITHM=SSL; /* connect using a fully-qualifies hostname or DNS alias */ %let session1=connect.gelenable.sas.com 17551; /* authenticate with username and password */ SIGNON session1 user="Alex" password="My1Password!"; Notice the explicit set of the NETENCRYPTALGORITHM option. While this corresponds to the default for SAS Viya, SAS 9 clients may have a different value and the connection may fail if not set like this.   While this example shows a connection to the default 17551 port, your administrator could have configured the service to listen on a different one.   The hostname specified to connect is either the fully qualified hostname of the service exposing the SAS/CONNECT spawner (usually a cloud load balancer) or a DNS alias that points to it.   Note: to be able to connect from SAS 9 or SAS Viya 3 clients, they need to be patched with the hotfixes documented within the SAS Usage Note https://support.sas.com/kb/68/611.html .   Grid Sign-on   Starting with SAS Viya 2022.1.4 and later, it is possible to use the grdsvc_enable function to start a SAS/CONNECT server session in a dynamically launched pod. This is the evolution of the similarly named function used in SAS 9 SAS Grid Manager clients, but with SAS Viya it comes with a great advantage: it just works with any SAS Viya license - no SAS Workload Management required!   For the initial release, it is only possible to use this capability from a client co-located in the same Kubernetes cluster as the server.   The grdsvc_enable function is used to specify the 'context' to be used by the SAS/CONNECT service to have the Launcher service start the SAS/CONNECT server pod.   %put gs_rc=%sysfunc(grdsvc_enable(session1,"context=default-launcher")); SIGNON session1; Notice that we did not specify a username or a password; the client automatically sent in the OAuth token to sign in as the same user already logged into the client. Similarly, we did not specify any server/service hostname: grid-enabled sign-ons automatically connect to the SAS/CONNECT service that is in the same environment as the client. It is possible to automatically use this capability for all sessions and not only for the current one:   %put gs_rc=%sysfunc(grdsvc_enable(_ALL_,"context=default-launcher")); SIGNON session1 signonwait=NO; SIGNON session2 signonwait=NO;   The "signonwait=NO" helps run your code in parallel because it lets SAS/CONNECT sessions start asynchronously.   Architecture note: this capability bypasses the SAS/CONNECT spawner and uses the SAS/CONNECT service instead. This should be completely transparent to users.   MP CONNECT Sign-on   This is also referred to as SASCMD sign-on in the documentation.   MP CONNECT sign-ons always start a new process in the same pod that the client process is running in: there is no connection to any spawner or service. While this permits leveraging parallelization in your code, it is limited in scalability because all sessions share the same resources (CPUs, memory, I/O), since they are all hosted in the same pod, on the same node.   By default, if you omit both USER= and PASSWORD= options in the SIGNON statement, SAS Viya performs an MP CONNECT sign-on – ignoring any server name you may have specified to connect to.   SIGNON session1; The SIGNON statement supports the SASCMD option to specify further options to influence how the children sessions are started. With SAS Viya, though, all options are ignored, and the default command is used because the LOCKDOWN option is enforced by default. To be able to use the SASCMD option, an administrator would have to disable LOCKDOWN (not recommended for the security implications).   Authentication   After discussing multiple ways to connect to SAS Viya using a SAS/CONNECT session, let’s talk about authentication, and how to specify user credentials on the SIGNON command. SAS/CONNECT on SAS Viya can currently authenticate clients by supporting four different methods:   username/password OAuth authinfo files Kerberos Username and password   While this is the simplest authentication method, it is also the less secure, since it requires to hard-code user credentials in your code:   SIGNON session1 user="Alex" password="My1Password!"; Passwords can be encoded with PROC PWENCODE, but that only offers basic security: for example, it does not protect from replay attacks.   OAuth   OAuth authentication relies on the exchange of access tokens between services, which is the default SAS Viya internal authentication method. No passwords are exchanged, and this is considered the most “cloud-native” model.   To enable OAuth authentication, specify “_OAUTH_” as the username. To sign on from an internal client, do not specify any password: the SAS/CONNECT client will automatically pass the OAuth token that it acquired at initial logon to the SAS/CONNECT spawner:   SIGNON session1 user="_OAUTH_"; To sign on from an external client, the OAuth token should be acquired in some other way,  then specified as the password:   %let token=eyJhbGcz1N ... ... ... OlXlI1UN6XQ; SIGNON session1 user="_OAUTH_" password="&token"; As an example, if you have the SAS Viya command-line interface installed on the client, you can use that to sign into SAS Viya, and that will save a usable token on your client, in the credentials.json file located in your home directory.   Notice that the token is usually a long sting, and SAS may output a warning note that can be safely ignored:   87  %let 87! token=eyJhbGciOiJSUzI1NiIsImprdSI6Imh0dHBzOi8vbG9jYWxob3N0L1NBU0xvZ29uL3Rva2VuX2tleX ... ... 87! 1UN6XQ; 88  SIGNON session1 user="_OAUTH_" password="&token"; NOTE: The quoted string currently being processed has become more than 262 bytes long.  You might have unbalanced quotation marks. Authinfo files   Hard-coding your username and password credential in the SIGNON statement not only is a bad security practice, but does not permit sharing code between different people. A better way to store and specify credentials, while still using username and password to SIGNON, is provided by AUTHINFO files.   An authinfo file contains user IDs and passwords that are used for authentication. To use an authinfo file, specify _AUTHINFO_ as the value in the PASSWORD= option in the SIGNON statement. If there is more than one user ID in the authinfo file, specify a value for the USER= option to select which one to use.   By default, SAS/CONNECT looks for an authinfo file in the client user home directory (on Windows, that’s usually C:\Users\<username> ); you can specify another location, for example with the AUTHINFO= system option.   SIGNON session1 password="_AUTHINFO_"; Kerberos   Within SAS Viya, Kerberos is enabled by default, but it has to be properly configured by your administrator. Once that is done, signing on with Kerberos is as simple as writing:   SIGNON session1 user="_SSPI_"; Debugging   While testing the code for this article, configuration errors and wrong statements were inevitable. Why not capture the error messages that SAS outputs?   Here they are, all collected for your reference, together with possible solutions if you’ll make the same mistakes I did!     ERROR: Windows SSL error -2146893019 (0x80090325) occurred at line 2696, the error message is "The certificate chain was issued by an authority that is not trusted. " ERROR: A communication subsystem partner link setup request failure has occurred. ERROR: Network request failed (rc 0x033C2F90) - Windows SSL error -2146893019 (0x80090325) occurred at line 2696, the error message is "The certificate chain was issued by an authority that is not trusted. "   Or   ERROR: Windows SSL error -2146869244 (0x80096004) occurred at line 2696, the error message is "The signature of the certificate cannot be verified. " ERROR: A communication subsystem partner link setup request failure has occurred. ERROR: Network request failed (rc 0x3B2C40F0) - Windows SSL error -2146869244 (0x80096004) occurred at line 2696, the error message is "The signature of the certificate cannot be verified. "   This means the SAS Viya Root CA was not imported into the client truststore. See   https://go.documentation.sas.com/doc/en/sasadmincdc/v_032/calencryptmotion/n1xdqv1sezyrahn17erzcunxwix9.htm#n0309hwuyq56x2n17mkapfltqau6   and   https://go.documentation.sas.com/doc/en/sasadmincdc/v_032/calencryptmotion/n1xdqv1sezyrahn17erzcunxwix9.htm#p1rk4fxh9clzvin1sfb5p32cffee     ERROR: A communication subsystem partner link setup request failure has occurred. ERROR: Network request failed (rc 0x033C2EF0) - SSL Error: Invalid subject name in partner's certificate. Subject name must match machine name.   This means the TLS certificate presented by SAS/CONNECT spawner does not include the external hostname or DNS alias used to connect to the server. See    https://go.documentation.sas.com/doc/en/sasadmincdc/v_032/calencryptmotion/n1xdqv1sezyrahn17erzcunxwix9.htm#p19femzwwlxh39n1vz8g1qnc759y     ERROR: A communication subsystem partner link setup request failure has occurred. ERROR: The connection has timed out.   This is a generic error when the client cannot reach the SAS/CONNECT spawner. It may mean that the spawner is not listening on the hostname/port specified. Also, make sure that the SAS/CONNECT spawner has been enabled to support external clients. See https://go.documentation.sas.com/doc/en/sasadmincdc/v_032/dplyml0phy0dkr/n08u2yg8tdkb4jn18u8zsi6yfv3d.htm#p1n0mb2fhaarjrn0zt9oqo2vprrw   Another reason may be that there is a firewall blocking access, in which case refer to your IT department to have it open.     ERROR: A communication subsystem partner link setup request failure has occurred. ERROR: Cannot negotiate encryption algorithm.   This could be due to a mismatch between client and server TLS configuration. See   https://go.documentation.sas.com/doc/en/pgmsascdc/v_031/viyaconnref/n0pj55r718uoj3n1jjp89k1jfrgx.htm     ERROR: A communication subsystem partner link setup request failure has occurred. ERROR: Communication request rejected by partner: security verification failure.   Either username/password credentials are wrong, or, if using a AUTHINFO file, you may have specified USER=”_AUTHINFO_” instead of PASSWORD=”_AUTHINFO_”.     ERROR: A communication subsystem partner link setup request failure has occurred. ERROR: Retrieval of credentials from authinfo file failed.   When using an AUTHINFO file, the SAS client was unable to either find or read it.   Conclusion   In this article, we have seen the different ways a programmer can use the SIGNON statement to specify the connection and authentication parameters to start a SAS/CONNECT session in a SAS Viya cluster. Are you leveraging SAS/CONNECT in your SAS Viya environments? Let us know in the comments!   Find more articles from SAS Global Enablement and Learning here. ... View more

Starting with SAS Viya 2021.2.4, you may encounter errors while executing programs that were previously fine. If you have long-running sessions, such as complex analytical models in SAS Model Studio or batch jobs submitted to SAS Workload Management, this may impact you. In this post, we’ll show how it may depend on changes implemented to enforce stricter security by default, and how to solve these issues.   SAS Logon manager is one of the key services for authentication in SAS Viya: it provides both an end-user interface for authentication and internal authentication to other services​. This enables single sign-on within the SAS Viya environment between services​.   SAS Viya authentication architecture is built around the OAuth 2.0 protocol, and its extension for authentication, OpenID Connect 1.0 (OIDC). According to these protocols, SAS Viya services exchange OAuth tokens to access protected resources. In layman's terms, SAS Viya services use these tokens to prove to each other that user authentication has occurred and is valid.   OAuth tokens include an expiration field; this is a security feature because tokens are usually non-revocable: once they are issued, they must be unconditionally accepted by services until expired.   With security being at the front and center of most SAS decisions, SAS is gradually reducing the default expiration for Access Tokens from the original value of 10 hours. Shorter-lived tokens reduce the window that a compromised token can be used by an attacker. To avoid possible disruption, this change is being implemented in steps. As of the stable release 2021.2.4 (February 2022), the Access Token default lifetime is decreased to 4 hours; in a later release, it will be further reduced to 1 hour.   The default behavior   Most SAS Viya applications are fine with short-lived Access Tokens. In general, it is the responsibility of client applications to interact with any called service using a valid token, i.e., a token that is non-expired for the lifetime of the request. This includes direct requests to called services, and subsequent requests where the service needs to call other services to carry out the request, and those services in turn call other services, and so on.   Most API calls complete quickly, allowing a short-lived access token to be used across many requests.   Client applications that call backend APIs on behalf of end-users (think SAS Studio, Model Studio, etc) usually create user sessions that may last longer than the original Access Token; these clients can use Refresh Tokens to re-authenticate on behalf of the user and get a new token. This ensures that there is always a valid, non-expired token when servicing a request.   Some applications inherently need longer-lived tokens, for example, applications that are involved in batch processing of long-running jobs, where the app may need to make calls to other services throughout the lifecycle of the jobs.  These client applications may be registered with an expiration value specific to their needs, and this will have the effect of overriding the system-wide default value when any tokens are issued to the registered client ID. Such an example is SWAT, used to submit code – possibly long-running – from Python clients to CAS.   Why you may need to tune the default settings   There are situations where clients have long-running sessions that try to use expired Access Tokens – and obviously fail. In these cases, the default configuration may not be adequate. Here are a few examples. When a batch job is submitted to SAS Workload Management via the sas-viya CLI, it sends the OAuth token of the authenticated user as input along with the job​. The sas-viya CLI framework does not refresh tokens for jobs. When jobs sit in a queue for too long, the OAuth token may expire before the job completes and this can result in job failures. When executing SAS jobs, ​either in batch or through interactive interfaces such as SAS Studio, the OAuth token of the authenticated user is provided at the startup of the backend compute session. Long-running jobs may try to use the token after it has already expired, resulting in failures. PROC HTTP or SAS/CONNECT SIGNON are examples of code that can use OAuth tokens to authenticate to SAS Viya services. When submitting very large models within SAS Model Studio, some nodes in the pipeline may require more time to execute than the default timeout and this can result in model execution failures.   In these cases, the cause of the problem is not always clear because the message for the authentication failure can be buried in the log of a service called by the front-end client. The client may simply present a generic error to the user, such as “A call to service X has failed” or “the job failed”.   A possible solution may be to increase the timeout for OAuth tokens to more than the longest job​ executing in that environment. Each customer should evaluate their business and security requirements and tune the environment to accomplish the right balance between them.   How to tune token expiration   The system-wide default token expiration can be changed on a per-deployment basis by SAS administrators in SAS Environment Manager by creating or modifying an existing configuration for the sas.logon.jwt configuration definition.  There are two properties involved: policy.accessTokenValiditySeconds, and policy.global.accessTokenValiditySeconds. The latter is meant for multi-tenant deployments and is meant to apply to all tenants. The system-wide default will apply to all applications that are not registered with their own value.   Here is a simple step-by-step guide. Log on to SAS Environment Manager as an administrator and opt into Assumable Groups. Navigate to the Configuration area and filter for "sas logon". Select SAS Logon Manager, then click New Configuration Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Select sas.logon.jwt​ Modify the following two parameter values, setting them to the desired value in seconds (for example, 108000 corresponds to 30 hours), and save: policy.accessTokenValiditySeconds policy.global.accessTokenValiditySeconds   Note: assess the max runtime required by the longest running job and set the timeout value accordingly. Restart the SAS Logon service # delete the pod(s), kubernetes will restart it kubectl delete pod -l app=sas-logon-app # now wait for the restarted service to be ready kubectl wait --for=condition=Ready pod -l app=sas-logon-app --timeout=180s   Conclusion In this article we have seen how changes implemented to enhance SAS Viya default security may create failures in jobs that previously run fine. We have described how this may be due to the new default lifetime of OAuth tokens used by SAS Viya services to authenticate between themselves. Finally, we have seen how to tune the timeout to satisfy the requirements of long-running jobs.   Find more articles from SAS Global Enablement and Learning here. ... View more