SAS Innovate is an amazing venue where customers can deep dive into SAS’ latest breakthroughs. This April I had the privilege to attend and present at SAS Innovate 2024 in Las Vegas. Amongst many energizing sessions, I particularly enjoyed the Super Demo “Need for Speed: Fast-Tracking Batch Workloads with SAS Viya Workload Management”, presented by Wes Gelpi, Director R&D, SAS Viya Compute, and Prasad Poozhikalayil, Sr Product Manager, SAS Viya Platform. The topic caught my attention, and I decided to look deeper into some details and test the new capabilities.   In short, starting with SAS Viya 2024.04 stable release, SAS introduced new batch capabilities that benefit our customers who engage in extensive batch processing, involving time sensitive, high volumes of SAS jobs of very short duration. The new features address some inefficiencies noted by customers in their previous experience, such as longer turn-around time, excessive resource utilization, potential timeouts and job failures due to sudden surge in number of pod creation.   This article is the first in a series where I share the result of my reading and researching. It describes reusable batch servers.   The second article will focus on optimized input and output file storage.   Finally, the third and last will present some performance tests that compare the throughput and total run time of a bust of short-running jobs when run on default batch servers versus high-throughput batch servers.   SAS Viya Batch Servers   The SAS batch processing infrastructure in the SAS Viya platform is designed to provide an ad-hoc set of servers and services that, by default, are tailored to processing long-running, scheduled batch jobs. It also supports interactive execution to simplify development and testing, but at its core the focus are long-running, fire-and-forget batch jobs.   In this default configuration, each job submission is processed by a batch server (a dedicated SAS process) running in a Kubernetes pod that is started on-demand, executes that one job and then shuts down.   What if your use case centers on managing a significant volume of small, time-sensitive data jobs? With the default batch server, you might face a critical performance bottleneck in the latency from when a pod starts to when it actually begins running each job. Starting a new pod can add an overhead of up to 10/15 seconds to the total execution time. Even worse, if your Kubernetes cluster is configured for dynamic auto-scaling and there are no compute nodes available, starting and initializing a new node might add an additional overhead of a few minutes. All these overheads do not impact too much a set of scheduled jobs when each job runs for a few hours – or even a few minutes. Executing a job in 20 minutes or 20 minutes and 15 seconds is a small difference of about 1%.  But when your use case is submitting quick burst of hundreds of 15-seconds batch jobs, adding 15 seconds to start up a pod each time means doubling the total execution time, or, if want to look at it from a different angle, the total throughput in terms of jobs per minute halves. That’s a significant performance hit! The high number of pod creation and destruction also stresses the Kubernetes API services, with the risk of excessive resource utilization leading to potential timeouts and failures.    Animation showing default batch servers processing 3 short jobs.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Reusable Batch Servers   Starting with SAS Viya 2024.04 stable release, the SAS Viya platform provides a new feature: “Reusable Batch Servers”. A reusable batch server is a batch server pod that can process multiple jobs sequentially, without stopping/restarting the pod. These servers decouple jobs from pod creation and initialization.   SAS developed this feature to benefit most of our customers who engage in extensive batch processing, involving time sensitive, high volumes of SAS jobs of very short duration. It addresses the inefficiencies noted above, such as longer total execution time, excessive resource utilization due to sudden surge in number of pod creation, potential timeouts and job failures.  Animation showing reusable batch servers processing 3 short jobs.   How to Configure and Enable Reusable Batch Servers   To use reusable batch servers, you must enable the capability in the batch context, configure a few settings – such as the minimum number of servers that run all the time – and specify a shared account under which they should run.   Let’s describe these steps in more detail, including considerations about creating an efficient workload management configuration.   Configure the shared account credential   To configure the batch server to run under a shared account, you must use the SAS Viya CLI to create a new credential for this shared account. With the CLI, login with an administrative account and use the batch plugin to store the shared account userid and password with the following command:   sas-viya batch credentials create   The CLI will prompt for a username and password, then save them in the credentials microservice.     If you do not want to store a password, you also have the option to use an OAuth token. This is useful, for example, to support SCIM authentication providers. See the documentation for detailed steps.   (Optional) Created a dedicated workload queue.   While not strictly required, creating a dedicated workload queue to be used for high-throughput batch servers can simplify further tuning of the SAS Viya platform. For example, you can segregate high throughput batch jobs from other kinds of compute workloads. An administrator can use the Workload Orchestrator page of SAS Environment Manager to create and configure a dedicated queue.     Create and configure a new batch context   You may want to create a new batch context for high-throughput batch, and leave the default context configured as is, to support long running jobs. An administrator can use the SAS Viya CLI:   sas-viya batch contexts create \     --name "high_throughput" \     --launcher-context-name "SAS Batch service launcher context" \     --queue "batch_high_throughput" \     --run-as "user1" \     --reusable \     --min-servers "7" \     --max-servers "7"   The same result can be accomplished with the Contexts page of SAS Environment Manager.     As soon as the new context is created, the batch service starts the preset number of minimum servers.   An administrator can use to Workload Orchestrator page of SAS Environment Manager to verify the newly started servers. Notice that they are running under the shared account set in the context configuration.     About Server Sizing   You may have noticed in the screenshots above that both the new workload queue and the new batch context use the same value for the number of servers, in the examples above 7. How did we come to that number?   In this case, we only had one compute node with 8 cores, and we decided to fully dedicate it to this test. A common practice with batch jobs is to dedicate 1 core to each job. We wanted to leave 1 core to the OS and Kubernetes services, which means that our batch service can use 7 cores, leading to the above setting of 7 reusable servers. The workload queue parameter was set accordingly.   Also, for this environment we use a sustained flow of batch jobs, so we set the minimum and maximum number of servers to the same value. This optimizes server allocation, because the maximum number of servers is immediately available at startup, but it comes with the associated cost of full infrastructure utilization. If you know that your batch jobs come in burst of variable throughput, you can set a lower minimum - even down to zero. As usage falls off and the system is increasingly idle, the unused batch servers will terminate automatically, leaving the minimum number up and running and saving on infrastructure cost. The default server idle timeout is 300 seconds, and a custom value can be set when defining the new batch context.   Each customer environment is unique, and these values should be set after proper system sizing and architecture design. But do not feel compelled to find the perfect tuning at first try. SAS Workload management gives you the tools to put into practice the mantra “Plan. Monitor. Refine”.   For further reading, a very detailed discussion about the factors that influence the tuning of these setting is presented by Rob Collum in his article Determine how many SAS Viya analytics pods can run on a Kubernetes node – part 3.   Coming up next   When discussing this new feature with some colleagues, I noticed we were using many different names. We call it high-performance batch, high-throughput batch, reusable batch… In the end, call it as you want: what matters is that your demands for high-volume batch processing can be satisfied in a shorter time. Who does not like that?   In the next article of this series, we’ll explore another change that improves batch performance by eliminating the need to download and upload files exchanged between batch clients and servers. Stay tuned!   Edoardo Riva     Find more articles from SAS Global Enablement and Learning here. ... View more

The SAS extension for Visual Studio Code supports a special file type called “SAS Notebooks.” These are just like Python notebooks that you might have used in Jupyter or other environments. Notebooks allow you to mix code with documentation and results, encapsulating a work product that is easy to understand and demonstrate. What if you want to use SAS Studio to work with SAS notebooks as well? Read on to find out how!   I was privileged to teach a Hands-on Workshop at SAS Innovate in Las Vegas, Nevada this year. Titled “Interacting with SAS® Viya® Using SAS® Enterprise Guide® and Other Clients”, the session included an exercise dedicated to exploring the capabilities of  the SAS Extension for Visual Studio Code. One of these functionalities is the support for SAS Notebooks, which, just like Python notebooks that you might have used in Jupyter or other environments, allow you to mix code with documentation and results in an interactive online page. SAS Notebooks can integrate SAS, SQL, Python code and inline documentation in a seamless execution environment.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   While I was demonstrating this capability, one of the attendees asked “I often work on the same SAS programs both in VSCode and SAS Studio. Can I open a SAS notebook with SAS Studio as well?”. I’m glad they asked: I was just going to show how to close the circle from VSCode to SAS Studio, so my answer was a cryptical “Well, no, but almost yes”. What does that mean?   While at the moment you cannot use SAS Studio to directly open a “.sasnb” file (the format of SAS notebooks), it’s really easy to export a SAS notebook from VSCode as a “.flw” file (the format of SAS flows). Every cell in the SAS notebook becomes a node in the SAS Studio flow. SAS Studio can then open the file and let you work with the flow. Voila'.   Since this is a file conversion, changes made in the flow would need to be synched back to the notebook, but that is beyond the scope of this post. Always use good version management processes to synch such changes.   Step by step   Here is a guided step-by-step guide directly taken from the SAS Innovate session that shows you how to do it.   After creating a SAS notebook in Visual Studio Code, save it by pressing CTRL+S. In this example, it is saved under “My Folder” with the name "Innovate24.sasnb". In the explorer pane, locate the file by expanding "My Folder", then right click on the "Innovate24.sasnb" file and select "Convert to Flow..." (this is a custom action that the SAS Extension to VS Code adds to the right-click context menu).     Press "Enter" to accept the default filename. VS Code will save the notebook as a SAS Studio flow, located in the same folder as the original file. Open the browser and sign into SAS Studio. In SAS Studio explorer pane, navigate to "Folder Shortcuts" > "Shortcut to My Folder" and double-click on the "Innovate24.flw" file to open it.     The SAS Studio flow that is automatically generated by VS Code includes each cell in a dedicated swimlane to preserve the sequential execution nature of notebooks, both visually and in the generated code:     You can verify the code in each node. As an example, select the Python Program node in SAS Studio to look at the code:     you should recognize the same code entered in VS Code in the original notebook:     Now that the flow is open in SAS Studio, you can submit it using the “Run” button to get the same results as the original notebook.   Conclusion   One of the objectives I had with this SAS Innovate session was to show how you can leverage your knowledge of a SAS client to jump into a different one, without having to re-learn how to use it from scratch. This can open the doors to benefiting from additional capabilities. As an example, SAS notebooks are inherently interactive and cannot be scheduled or submitted in batch from VS Code. Once you transform a notebook into a SAS Studio flow, that becomes possible:     If you want to see another example of how to use VS Code, including this integration with SAS Studio, see Ryan King’s post Using The SAS VS Code Extension to Check Prime Distribution.     Find more articles from SAS Global Enablement and Learning here. ... View more

The SAS Infrastructure Data Server is a fundamental component of the SAS Viya architecture, that can be deployed both in the same Kubernetes cluster of the other components or outside the cluster.   Are you looking to choose an external PostgreSQL instance for use with the SAS Viya platform? In this post, we’ll highlight useful considerations that can help choose the right instance when opting for an external, managed cloud instance. ​ A Quick Summary   Before diving into the topic at hand, let’s recap what are we talking about. If you’ve been following this space, you may recognize there have been a few posts on the subject of external (or internal) PostgreSQL:   Choosing Between Internal or External PostgreSQL for SAS Viya Deploying SAS Viya with an External PostgreSQL Configure an external PostgreSQL for SAS Viya in your Open-Source Kubernetes deployments – part 1 Configure an external PostgreSQL for SAS Viya in your Open-Source Kubernetes deployments – part 2   You can refer to these articles for additional details, but, in short, we can say that the SAS Infrastructure Data Server stores SAS Viya user and system content, such as reports, authorization rules, attachments, user preferences, and so on. The SAS Infrastructure Data Server requires an instance of PostgreSQL. The SAS Viya platform deployment can install and configure its own open-source PostgreSQL database inside the SAS Viya Kubernetes cluster; this is referred to as an internal instance. If you want, you can choose instead to provide another PostgreSQL instance, which is called an external instance.   External PostgreSQL choices   For an external PostgreSQL server, the SAS Viya platform support has changed over time and with the different releases. For this reason, you are always encouraged to refer to the official documentation corresponding to your SAS Viya platform release. You can find the latest one here: https://go.documentation.sas.com/doc/en/itopscdc/default/itopssr/p05lfgkwib3zxbn1t6nyihexp12n.htm#n0yp4jk7n9mcl1n1vu0oakq63bfc   As a general rule, with the SAS Viya platform, you can use managed PostgreSQL instances hosted by any supported cloud providers, or the Open Source version of PostgreSQL – for self-managed deployments:   Customer-managed PostgreSQL instance​   PostgreSQL (Open Source)​   Managed PostgreSQL instances hosted by cloud providers:​   Microsoft Azure Database for PostgreSQL - Single Server Microsoft Azure Database for PostgreSQL - Flexible Server​ Amazon RDS for PostgreSQL​ Google Cloud SQL for PostgreSQL   Two of the communities.sas.com articles previously referenced focus on the provisioning and configuration of self-managed Open Source PostgreSQL. The rest of this post will focus on the cloud-hosted managed options.   What drives the choice of a specific managed instance?   The official documentation highlights all the requirements that should be met when using an external PostgreSQL instance. For example, starting with SAS Viya platform 2023.11, PostgreSQL 11 is not supported anymore: this automatically excludes any instance of “Microsoft Azure Database for PostgreSQL - Single Server”, which only supports PostgreSQL up to version 11 (and, by the way, has been put on the retirement path by Azure).   Cloud architects, when designing the database to use for each project, must balance business drivers that sometimes drive in opposite directions. For example, it’s quite difficult to satisfy scalable performance, high availability, and low cost, all at the same time. However, it’s possible to get good performance and minimal failure rates for a given cost when considering the specific software requirements.   While other solutions may require high I/O throughput or fast storage, the most important factor that drives proper instance sizing with the SAS Viya platform is that PostgreSQL should support at least 1024 “maximum number of connections”.  How can you satisfy this on cloud-managed instances? Let’s see what it means for each of the supported cloud vendors.   Azure Database for PostgreSQL   Azure Database for PostgreSQL powered by the PostgreSQL community edition is available in two deployment modes: Single Server Flexible Server   “Azure Database for PostgreSQL - Single Server” is on the retirement path and is scheduled for retirement by March 2025. We do not recommend that you choose this edition for any new deployment.   “Azure Database for PostgreSQL – Flexible Server” provides different instances that meet the 1024 concurrent connections requirements. For production use, Burstable instances are usually not recommended because of their lack of predictable performance.   Highlights of the minimum “Azure Database for PostgreSQL – Flexible Server”  instance sizes that support at least 1024 max user connections Screenshot as of January 2024 of https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-limits#maximum-connections.   The viya4-iac-azure utility can be used to provision a managed PostgreSQL server on Azure. By default, it creates an instance type GP_Standard_D16s_v3 (16vCPU, 64GiB RAM) that exceeds the SAS Viya requirement for 1,024 concurrent connections and is suited for baseline production use.   You can choose to further customize the maximum number of connections independently of the chosen instance size. Microsoft recommends against this custom setting, for the risk of exhausting the memory of the PostgreSQL instance, risking crashes or high latency.   Amazon RDS for PostgreSQL​   Amazon RDS for PostgreSQL lets you choose from a variety of instance types with varying combinations of CPU, memory, storage, and networking. The maximum number of concurrent connections is calculated based on the DB Instance memory, with a maximum of 5000.   Highlights of the “Amazon RDS for PostgreSQL”  rules to calculate the max user connections parameter Screenshot as of January 2024 of https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html#RDS_Limits.MaxConnections   The viya4-iac-aws utility can be used to provision a managed PostgreSQL server on Amazon. It defaults to an instance type db.m5.xlarge (4vCPU, 16GiB RAM) that meets SAS requirement for 1,024 connections: 16 GiB = 17,179,869,184 bytes, which, divided by 9,531,392 is equal to 1,802 connections allowed.   Google Cloud SQL for PostgreSQL   Google provides Cloud SQL for PostgreSQL as a managed service. The most important factor impacting image sizing is the server memory. The underlying instance needs more than 120GB of RAM to meet the 1024 connections requirements.      Highlights of the “Google Cloud DQL for PostgreSQL”  rules to calculate the max user connections parameter Screenshot as of January 2024 of https://cloud.google.com/sql/docs/postgres/flags#postgres-m   The viya4-iac-gcp utility can be used to provision a managed PostgreSQL server on Google Cloud. By default, it creates an instance type db-custom-8-30720 (8 vCPU and 30 GB of RAM): you can see in the above table that this supports 600 connections by default. You can choose to further customize the maximum number of connections independently of the selected instance size.   Additional Considerations   If you are deploying multiple SAS Viya instances, you can use multiple PostgreSQL servers with a single database in each (one server per deployment), or you can share a single PostgreSQL server with multiple databases inside (one database per deployment).   Some SAS offerings require an additional PostgreSQL cluster called SAS Common Data Store, or CDS PostgreSQL. It is configured separately from the required platform PostgreSQL cluster that supports SAS Infrastructure Data Server. You can find a list of offerings that require this separate PostgreSQL cluster at https://go.documentation.sas.com/doc/en/itopscdc/default/itopssr/p05lfgkwib3zxbn1t6nyihexp12n.htm#n03wzanutmc6gon1val5fykas9aa. All of the considerations written in the previous sections apply to this instance, as well.   Conclusion   The SAS Viya platform requires an instance of PostgreSQL, and you can choose an external, cloud-based PostgreSQL instance for use with the platform. We have seen in this article different options and minimum instance sizes that support at least 1024 concurrent connections, which is the requirement for the SAS Viya platform. ... View more

The SAS Viya platform includes by default (and for free!) SAS/CONNECT with all its capabilities. Previous articles, including Moving to the cloud with SAS/CONNECT, have shown SAS/CONNECT evolution to adapt to cloud environments and support our customers in their journey to modern architectures. Innovations keep coming! Starting with SAS Viya version 2023.10, SASCMD sign-ons (a.k.a. MP Connect sign-ons) always start new SAS sessions in new pods by default.   Let's start by stepping back a moment to see what MP Connect is, and why this is a welcome change.   Distributed computing, with a single machine?   Before cloud computing, before any kind of multi-machine distributed computing, SAS customers had no choice but to run their code on a single machine (yes, you can tell I’ve been at SAS for quite some time).   SAS capabilities have always been disruptive, even in those limited environments. Multi-process CONNECT (or MP Connect) was conceived to divide time-consuming tasks into multiple units of work and to execute these units of work in parallel. It did that – and still does – by providing a framework that lets you start and coordinate multiple child SAS processes from a controlling parent SAS session. This way you can parallelize code that otherwise runs sequentially, for the purpose of reducing the total elapsed time necessary to execute a particular application.    An example of a process split into multiple parallel subprocess.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   From machine to pod   MP Connect capabilities are automatically available in SAS Viya platforms, because they are provided by the SAS/CONNECT product, which is included by default in every SAS Viya environment. What happens when you move your existing MP Connect code to SAS Viya? Apparently, it's business as usual: child processes are spawned from the parent session to execute part of your code in parallel. But, under the hood, the execution environment is completely different. What used to be running on a physical machine, now runs in Kubernetes, inside a pod.   MP Connect spawning child sessions in a compute pod.   Maybe your original machine had 8 cores and 64GB of RAM (not too much, in modern terms) and your code is written to take full advantage of that computing power by spawning 7 additional sessions that use, in total, almost 100% of all CPUs. Kubernetes is built to control the execution environment and prevent a single pod from exhausting the resources of the node where it's running. As soon as your code starts requesting all that CPU power, Kubernetes will throttle down your pod - in a default configuration, down to 2 CPUs maximum. As a result, your existing code will run 4 times slower! It's obvious that this setup does not scale as expected. You could argue that the issue can be easily fixed by configuring the Kubernetes cluster to give more resources to SAS compute and connect pods. Yes, this solution could work, but it suffers of two problems:   It simply moves the bar a bit further. Today you want back your 8 CPUs, what will happen when you'll have more data and you will need 16, 32, or more CPUs? A single pod can only scale up to the size of the node where it's running. Scaling up an individual gigantic pod is an anti-pattern in the cloud. Cloud environments are designed to scale out by launching multiple, smaller pods. The Kubernetes cluster hosting your SAS Viya platform is probably designed to support multiple users with multiple applications. You cannot hog all resources for yourself!   What could be a better solution? Obviously, embracing a cloud-native design and scaling out to multiple pods!   Scaling out to multiple pods   If you've been following so far, you can now understand how the new default behavior is welcome in cloud environments. Starting with SAS Viya 2023.10, every new MP Connect sign-on always starts a child SAS process in its own dedicated pod, embracing cloud-native scalability and elasticity. Kubernetes can spread out the pods on multiple nodes, and, if the cluster is configured for auto-scaling, your limit is the sky... or better, your budget!    MP Connect launching child sessions in dedicated connect pods.   It's a matter of choices   Every time a new default is introduced in existing environments, it's important to give SAS Administrators to option to embrace this new capability, or to reset the SAS Viya platform to behave just like before. In this case, you can use a new environment variable, SAS_LOCAL_MPCONNECT. When set to true, it re-enables local MP Connect Sign-Ons, i.e. the original functionality of spawning a child session in the same pod where the parent process is running.   A SAS Administrator can use SAS Environment Manager to set the SAS_LOCAL_MPCONNECT environment variable to true in sas.compute.server: startup_commands and sas.connect.server: startup_commands configuration instances. In this case, the setting is configured for every SAS compute and connect server running in the environment.    Setting the SAS_LOCAL_MPCONNECT option for all compute server sessions.   A more limited scope could be achieved by setting the option case-by-case, as needed. As an end-user, you can add the following line in your code just before submitting the SIGNON statement:   options set=SAS_LOCAL_MPCONNECT=true;   In this case, only the current execution reverts back to the previous functionality.   Why would I revert back?   This new capability seems the obvious choice when you are writing code that uses MP Connect in SAS Viya. So why would you revert back to the previous functionality? The most obvious answer is when you are migrating existing code that could be broken by the new behavior. It's easy to understand that if the existing code uses local resources to share data between MP Connect sessions, it cannot work as-is when these sessions are launched in different pods. Here are a couple of examples:   The code saves datasets in the SASWORK library of one session to retrieve them from another session. SASWORK locations can be shared between SAS processes with the INHERITLIB= option of the SIGNON statement. You are using SASESOCK libraries to stream data between sessions through local pipes.   Those issues can be solved by re-architecting your application - for example, by sharing data through Kubernetes volumes mounted on all pod sessions, instead of using local directories. Yet, reverting back to the previous behavior can be an interim step to keep the code running during your migration.   Conclusion   I am always excited when I see how SAS Viya keeps evolving by embracing existing functionality and integrating it into modern cloud-native architectures. We have seen in this post how traditional MP connect code can be used without sacrificing Kubernetes capabilities to manage resource utilization, provide scalability when required, and ensure user-level process isolation.     Find more articles from SAS Global Enablement and Learning here. ... View more

The SAS/CONNECT Programming with SAS Viya: SIGNON article presents multiple ways to use the SIGNON statement in a SAS client to start and connect to a remote server session executing in the SAS Viya platform. One of the options covers the use case of external clients, with a brief list of preliminary configuration steps required to enable the capability. This article dives into the detailed steps behind that summary list, including a step-by-step recorded demo.   What are we trying to accomplish?   In a default deployment, the SAS/CONNECT spawner service is only reachable from internal clients. An example of an internal client is a SAS Studio session that submits the SIGNON command.    An internal client connecting to the SAS/CONNECT spawner to launch a server. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   External clients need additional configuration to reach the SAS/CONNECT spawner from outside the Kubernetes cluster. You have to create either a LoadBalancer or NodePort service. Examples of external clients are a SAS 9.4 Display Manager session executing on a laptop, or a SAS Studio session running in a different SAS Viya cluster or namespace.    External clients connecting to the SAS/CONNECT spawner via a Kubernetes LoadBalancer or Nodeport to launch server sessions.   The configuration steps Preliminary: verify that internal connections work as expected   Before even starting any configuration change, it’s a good practice to test that the default functionality is working as expected. You can verify that internal SAS/CONNECT connections are OK, in your SAS Viya environment, by logging into SAS Studio and submitting this simple test code – no changes required:   /* launch a SAS/CONNECT session on the same cluster */ %let session1=sas-connect-spawner 17551; /* authenticate re-using the already logged-in credentials */ SIGNON session1 user="_OAUTH_"; %put The client session is running on host: &SYSHOSTNAME; rsubmit; %put The server session is running on host: &SYSHOSTNAME; endrsubmit; SIGNOFF;   This code will do the following: Sign on to the SAS Connect Spawner using the Kubernetes service name sas-connect-spawner , which is only available inside the namespace, re-using the same credentials you entered when logging into SAS Studio Locally output the hostname of the pod where the client SAS session is running Use rsubmit to return the hostname of the pod where the SAS/CONNECT session is running Sign off from the remote session If this all works, you are ready to perform the steps to enable external access. 1.      Enable external access to the SAS/CONNECT spawner.   You can follow the instructions from the README file at <deploy>/sas-bases/docs/configure_sasconnect_spawner_in_the_sas_viya_platform.htm , in the section “Provide External Access to sas-connect-spawner via a Load Balancer”.   As you can read there, SAS provides a sample file ( sas-bases/examples/sas-connect-spawner/enable-external-access/sas-connect-spawner-enable-loadbalancer.yaml ) that you can copy in your sas-config directory, customize as needed, and include in the kustomization.yaml used for the original deployment. Then it’s just a matter of re-applying it using the same method as the original deployment (manual kubectl commands, sas-orchestration CLI, etc.)   An effective customization I always like to do is to reuse the load balancer frontend, which was already provisioned for the SAS Viya Ingress. This can bring a couple of benefits: Lower infrastructure costs, by avoiding the creation or provisioning of another load balancer. May simplify the TLS configuration, as detailed in the next step, by re-using the same external IP address and DNS alias as the Ingress. If you want to do it, you have to find the IP of the ingress load balancer with a command such as:   NS=ingress-nginx kubectl -n ${NS} get service ingress-nginx-controller -o json | jq -r ".spec.loadBalancerIP"   Then add it to the sas-connect-spawner-enable-loadbalancer.yaml as the last line, as in this example:    Example configuration of sas-connect-spawner-enable-loadbalancer.yaml   After the new configuration has been applied and submitted to Kubernetes, it's always a good practice to verify that the load balancer service gets created as expected, listening on an IP address external to the cluster:   NS=sasviya kubectl -n ${NS} get service sas-connect-spawner-loadbalancer Sample result: NAME                               TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)           AGE sas-connect-spawner-loadbalancer   LoadBalancer   10.42.230.223   10.XXX.XX.X    17551:31883/TCP   15m   2.      Verify the DNS name presented by the SAS/CONNECT spawner TLS certificate.   In the previous step, we added the new Kubernetes LoadBalancer service to an existing external load balancer that had already been configured in the corporate DNS and whose DNS alias was already specified in SAS Viya kustomization.yaml. For that reason, the certificates generated by SAS Viya already include that DNS name. The simplest way to verify that is to find the DNS name used for the ingress, and confirm that it is included in the SAN field of the SAS/CONNECT spawner TLS certificate.   To find the DNS name of the ingress, you can look into the kustomization.yaml file used to deploy SAS Viya:   grep INGRESS_HOST kustomization.yaml   Sample result:   - INGRESS_HOST=sasviya.gelenable.sas.com   Now list all the aliases in the SAN field of the SAS/CONNECT spawner TLS certificate:   NS=sasviya CONNECT_SPAWNER_SECRET=$(kubectl -n ${NS} get secrets | grep sas-connect-spawner | awk '{print $1}') kubectl -n ${NS} get secret ${CONNECT_SPAWNER_SECRET} -o jsonpath="{.data['tls\.crt']}" | base64 -d | openssl x509 -text -noout | grep -A 1 "Subject Alternative Name"   With these commands, we first get the name of the Kubernetes secret used by the SAS/CONNECT spawner. Then, we extract the certificate from tls.crt field of that secret, base-64 decode it, and pass the result to an OpenSSL command that prints the details of the certificate in a human-readable format. Finally, we filter the results to only show the "Subject Alternative Name" details.   The result should be similar to the following. Note that it includes the full DNS name found in the previous step.   X509v3 Subject Alternative Name:     DNS:localhost, DNS:sas-connect-spawner, DNS:sas-connect-spawner-8b9cd797b-5wswp, DNS:sasnode07, DNS: sasviya.gelenable.sas.com, IP Address:10.42.126.166, IP Address:10.43.17.98, IP Address:10.XXX.XX.X, IP Address:127.0.0.1   If you did not reuse the existing load balancer, and you created a new load balancer with either a new frontend IP address or a different DNS alias for the SAS/CONNECT spawner, then you’ll have to follow the additional steps described in the Encryption in SAS® Viya®: Data in Motion guide in the section Add the External Host Name or IP Address to the SAN in the Signed Certificate.   3.      Configure the client to trust the SAS Viya root CA certificate.   It is quite common to deploy the SAS Viya platform using a self-signed SAS Viya root CA certificate. This CA certificate is unknown to clients external to the cluster, so they cannot trust certificates presented by any SAS Viya server - including SAS/CONNECT. You can tell you are in this situation when, the first time you log into a web application, such as SAS Studio, your browser presents you with a security warning that you have to accept before you can get in. In this case, to be able to use SAS/CONNECT from an external Windows client, you have to obtain the SAS Viya root CA certificate, and then import it into the Windows client truststore. After that, the client will trust the certificates presented by SAS Viya servers. The official documentation lists a simple command to save the root certificate into a temporary file:   NS=sasviya kubectl -n ${NS} get secret sas-viya-ca-certificate-secret -o go-template='{{(index .data "ca.crt")}}' | base64 -d > /tmp/SASViya_ca.crt   Then you have to copy the resulting SASViya_ca.crt file to your Windows client (you can email, ftp, network copy… ), and import it in the Windows truststore, for example following the instructions in the Encryption in SAS® Viya®: Data in Motion guide in the section Import CA Certificates into the Windows Trusted Root Certificate Authorities Store.   Finally: Connect!   After you get this far, it’s time to enjoy the fruit of your work. From the client machine, you can start a SAS session and submit some code to verify if all is fine and SAS/CONNECT is configured for external connections. You can use code like the following, after substituting the DNS alias or IP address of the load balancer in the session1 macro variable:   /* launch a SAS/CONNECT session from outside the cluster */ /* set TLS connection options */ OPTIONS NETENCRYPTALGORITHM=SSL; /* connect using a fully-qualifies hostname or DNS alias */ %let session1=sasviya.gelenable.sas.com 17551; /* authenticate with an authinfo file */ SIGNON session1 password="_AUTHINFO_"; rsubmit; %put &SYSHOSTNAME; endrsubmit; SIGNOFF;   The resulting submission should print, in the SAS log, the name of the SAS/CONNECT server pod that gets started in the SAS Viya cluster, as in the following screenshot:       Recorded Demo   A recording of the configuration of a demo environment, showing all the steps listed above, is available on the SAS YouTube channel: SAS Demo | Configuring SAS/CONNECT for external access to SAS Viya Summary   When you have an external SAS client that you want to connect to a SAS Viya environment, you have to perform some configuration steps before you can connect them. Enable the SAS/CONNECT spawner to support external clients. Add the external host name or IP address to the SAN in the SAS/CONNECT spawner certificate. Obtain the SAS Viya root CA certificate and add it to the truststore of the external client. Links Official documentation to configure SAS/CONNECT for external access: The README file available in your deployment directory at <deploy>/sas-bases/docs/configure_sasconnect_spawner_in_the_sas_viya_platform.htm , in the section "Provide External Access to sas-connect-spawner via a Load Balancer" The "External Client Sign-On to TLS-Enabled SAS Viya SAS/CONNECT Spawner" section of the SAS® Viya® Platform Administration documentation  Sample SAS/CONNECT code to sign-on from internal or external Clients Ensure the TLS certificate presented by SAS/CONNECT spawner includes the external hostname used to connect to the server and any desired DNS alias Ensure the client accepts as valid the TSL certificate presented by the SAS/CONNECT spawner: Obtain the SAS Viya Platform Generated Root CA Certificate and Import CA Certificates into the Windows Trusted Root Certificate Authorities Store SAS 9.4 and SAS Viya 3.x client requirements   Find more articles from SAS Global Enablement and Learning here. ... View more

Starting with the 2023.03 SAS Viya release, the SAS Viya platform supports using an external instance of OpenSearch to provide searching and indexing capabilities. What are the requirements, and how to enable this new capability? If you are interested, read on!   OpenSearch OpenSearch provides search and indexing features for software that runs on the SAS Viya platform. Starting with the 2020.1.3 Stable release, the SAS Viya platform includes an Apache 2.0-licensed distribution of OpenSearch with enhanced security and a SAS-provided Kubernetes operator to handle its operation and management.   Starting with the 2023.03 release, new configuration options are added to the platform to support either: an internal instance of OpenSearch automatically included in the deployment (just as before) an external OpenSearch instance that you administer and maintain OpenSearch is required by every SAS Viya offering except SAS Viya Programming.   Why? The OpenSearch instance deployed by default inside the SAS Viya platform includes opinionated deployment settings to configure it according to the platform requirements. For example, the pods must run under a particular user ID, and require specific kernel settings on the nodes underlying the Kubernetes cluster. Some customers may have security policies that prevent these settings in their Kubernetes clusters. And, let’s be honest, a production instance of OpenSearch does not fit typical cloud-native environments: for example, every update to the SAS Viya platform requires a full stop and restart of the OpenSearch cluster.   An external OpenSearch instance is managed outside of SAS Viya and can alleviate these considerations. Supporting an external OpenSearch removes some requirements from the Kubernetes cluster dedicated to the SAS Viya platform. As an added benefit, an external OpenSearch reduces the footprint of the SAS Viya platform deployment.   In the end, adding support for an external option provides you with more flexibility in your architectural choices!   Internal VS External A simple table can easily show the main comparison points between using an internal OpenSearch instance or an external one. Internal External Dedicated instance deployed by the same process as the rest of the SAS Viya platform OpenSearch instance deployed by you before starting the SAS Viya platform deployment Deployed in the same namespace as the SAS Viya platform Deployed outside the SAS Viya platform namespace SAS Viya platform cluster sizing and design must account for OpenSearch (including kernel requirements, and storage requirements) OpenSearch requirements and sizing do not impact the SAS Viya platform cluster Contains all required plugins (including enhanced security) Must include required plugins Automatically configured as required by the SAS Viya platform Must be configured according to SAS specifications Managed by a SAS-provided Kubernetes operator Managed as you choose to Supported by SAS Supported by you   Requirements Whether you decide to use an internal or an external OpenSearch, once your deployment is done, there is no turning back. It is not possible to reconfigure an existing SAS Viya platform to use a different OpenSearch. If you change your mind, you will have to un-deploy and re-deploy.   The official requirements for an external deployment are described in the readme file at <$deploy>/sas-bases/examples/configure-Elasticsearch/external/README.md . Here is a summarized listing, valid at the time of writing (always refer to the official documentation to have up-to-date requirements): You can only use the opensource OpenSearch edition, not Elasticsearch. The supported OpenSearch version is 2.5. It must include a specific list of plugins used by the SAS Viya platform. The OpenSearch server should be set up and running before you start the SAS Viya platform deployment. Managed cloud subscriptions to Elasticsearch and OpenSearch are not supported; only instances that you deploy and configure. TLS between the SAS Viya platform and OpenSearch is strongly recommended, but not required. SAS Visual Investigator requires additional configuration settings. You can find the listing of SAS Visual Investigator specific requirements in the README file at <$deploy>/sas-bases/examples/configure-Elasticsearch/external/config/README.md , and a sample config.yaml is provided in the same directory.   How to deploy the SAS Viya platform with an external OpenSearch The deployment instructions are provided in the same README file as the requirements.   To use an external OpenSearch instance, you need to gather the following parameters from the OpenSearch administrator: Username and Password of an account with administrative privileges for your external OpenSearch instance URL (and port) of your external OpenSearch instance NOTE: If the connection uses TLS and OpenSearch is using custom-signed certificates, you also need the server CA certificate so that you can import it into SAS Viya. This step is the same as with any other external CA certificate, whether it is used to protect a connection to OpenSearch, a database, an LDAP server, etc. You can follow the official instruction in the document Incorporate CA certificates into SAS Viya   After you have those connection details, follow these steps: Copy the provided external-opensearch-transformer.yaml , secret.yaml ,and  client-config-tls.yaml  files from: $deploy/sas-bases/examples/configure-elasticsearch/external/ to $deploy/site-config/external-opensearch/ . No need to change anything in its content. Adjust the username, password, and URL values in the latter two files as instructed by the in-line comments. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     Edit the base kustomization.yaml to reference these files: add external-opensearch-transformer.yaml to the transformers block transformers: ... - site-config/external-opensearch/external-opensearch-transformer.yaml add the secret.yaml and client-config-tls.yaml to the resources block resources: ... - site-config/external-opensearch/client-config-tls.yaml - site-config/external-opensearch/secret.yaml Start the SAS Viya platform deployment with your preferred method. And finally, a reminder: if you copied your base kustomization.yaml from an environment that uses an internal OpenSearch instance, be sure to remove any existing lines referencing the internal instance, otherwise you will get an unsuccessful deployment!   Conclusion   In this article we have described the benefits and considerations of the new option that you have when deploying the SAS Viya platform starting with the 2023.03 release, the support of an external instance of OpenSearch to provide searching and indexing capabilities. We have also listed the steps required to perform a deployment connecting to an external OpenSearch instance.   Find more articles from SAS Global Enablement and Learning here. ... View more

Starting with the 2023.02 SAS Viya release, there are small but significant changes to the process required to configure SAS temporary storage (SASWORK). In this post, we will delve into the implementation details and the steps you must follow to keep your configuration current.   What has not changed   Let’s start by listing what has not changed. The performance requirements for SASWORK have not changed. The architectural considerations are still the same.   In short, the two main reasons to use a separate, high-I/O throughput volume for SASWORK are still:   To improve the performance of SAS compute processing. To protect other volumes in your deployment from being filled up if users fill the disk used by SASWORK.   There are plenty of articles on these topics, see for example:   Some SASWORK storage options for SAS Viya on Kubernetes Using generic ephemeral volumes for SASWORK storage on Azure managed Kubernetes (AKS) SAS Viya temporary files (SASWORK and CAS Disk Cache) in Azure   How to apply the customization   The high-level steps needed to perform this customization have not changed as well. You can find official instructions in the README file $deploy/sas-bases/docs/ sas_programming_environment_storage_tasks.htm .   In summary:   Copy the change-viya-volume-storage-class.yaml  example from sas-bases to site-config Replace the {{ VOLUME-STORAGE-CLASS }} placeholder with the desired storage class (i.e., hostPath) Add a reference to this file into the base kustomization.yaml Apply the new configuration to your cluster using the same method originally used to deploy SAS Viya   No need to restart any pod. The next time a compute server will start, it will use the new definition.   What has changed, then? Digging a bit, you’ll find that the differences from previous releases lie in the content of the sample file change-viya-volume-storage-class.yaml and how to reference it in the base kustomization.yaml (i.e., points 1) and 3)).   Let’s see both in detail.   The patch file   Here is a comparison of the content of the sample file change-viya-volume-storage-class.yaml, before and after the change:   Up to SAS Viya 2023.01: Kustomize Patch Starting with SAS Viya 2023.02: Kustomize Transformer Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   As you may notice, although the syntax is quite different, both the old and the new methods perform the same 2 actions:   Delete the current definition of the viya volume Define a new viya volume using the storage class you will place in the placeholder {{ VOLUME-STORAGE-CLASS }}   Kustomization.yaml   The second change is how to reference that custom file in the base kustomization.yaml. Here is a partial printout to show a comparison of the old and new syntax:   Up to SAS Viya 2023.01: add a new entry in the patch section Starting with SAS Viya 2023.02: add a reference to the custom file in the transformers section     Notice that the old method placed the "target" lines with a PodTemplate selector in the patch section of kustomization.yaml, while, with the new syntax, the same goes in the change-viya-volume-storage-class.yaml file (twice, once per Transformer)   Also, note that the new instructions require placing the line referencing the custom change-viya-volume-storage-class.yaml before the existing line ' - sas-bases/overlays/required/transformers.yaml ', as shown above.   Why this change?   Given that the end result is still the same as with previous releases, you may wonder what's the reason for this change. Well, it all has to do with a change in the supported version of the tool used to parse the configuration directory, Kustomize.   Since the release of SAS Viya 2020, we’ve always been using the same tried and true Kustomize 3.7.0. The time for an upgrade was so due, that we did it twice in a row. SAS Viya stable 2023.02 requires Kustomize 4.5.7, while SAS Viya 2023.03 and later releases (both stable and LTS) require Kustomize 5.0.0. (Note: this is accurate as of the time of writing. Always refer to the official requirements for the correct version for your release)   Kustomize dropped support for the old syntax a few versions ago, so we had to change to avoid parsing errors.   Am I affected by this change?   Let’s start with the easy answer. If you have an existing environment where you accepted the default configuration, nothing has changed there, so you are not affected.   If you are going to deploy or change an environment for the first time, then you have to use the syntax following the instructions included in your release. If this is SAS Viya stable 2023.02 or later, that means the new instructions: no need to care about what earlier SAS Viya releases required.   The only case where you should care is if you have an existing environment, deployed with a release older than 2023.02 (either stable or LTS) where you applied the old-style patch to configure a custom location for SASWORK. Even in this case, the runtime environment will not be affected if you leave it "as is": users can still use their applications with no errors. But, as soon as you try to upgrade the installation to the 2023.02 release or later, the upgrade process will fail if you leave the previous file and configuration in place. You have to change the custom change-viya-volume-storage-class.yaml that was placed under your site-config directory, then change the base kustomization.yaml, as shown above. If you do not, you may initially not get any error, but the resulting manifest (site.yaml) will be invalid. The old custom files, used with kustomize 4 or later, will output PodTemplates totally missing any SASWORK definition:       Comparing a “default” manifest (right) with one created with the old-style patch and kustomize 5.0.0 (left)   As soon as you try to apply that incorrect manifest, the operation will fail with errors such as:   Error from server (Invalid): error when applying patch: ... Resource: "/v1, Resource=podtemplates", GroupVersionKind: "/v1, Kind=PodTemplate" Name: "sas-batch-cmd-pod-template", Namespace: "viyaorch" for: "site.yaml": PodTemplate "sas-batch-cmd-pod-template" is invalid: template.spec.containers[0].volumeMounts[1].name: Not found: "viya" Conclusion   Starting with the 2023.02 SAS Viya release, there have been small but significant changes to the steps required to configure SAS temporary storage (SASWORK). While the high-level steps have not changed, there are differences from previous releases in the content of the sample file change-viya-volume-storage-class.yaml and how to reference it in the base kustomization.yaml. The changes are required by the newer version of Kustomize used with recent SAS Viya releases, and the end result is still the same as with previous releases. Be sure to check and update your existing customizations to avoid any errors!   Find more articles from SAS Global Enablement and Learning here. ... View more

In the first article of the series, we have seen how to leverage the OpenShift Local Volume Operator to make Azure VM ephemeral disks available to the pods running in the OpenShift Container Platform cluster. Now that this is done, the next step is to use them for SAS Viya temporary storage.   In this article we will focus on CAS: how can you configure CAS to use the local storage backed by the Azure ephemeral disk as your CAS DISK CACHE?   A quick summary   Here is the TL;DR recap of where we left in the previous article:   We are working in a test environment based on OpenShift deployed on Azure virtual machines The virtual machines we selected provide a fast local SSD disk that we want to use for temporary storage We configured those disks and the cluster so that the OpenShift Local Storage Operator creates a PersistentVolume (PV) on each node to reference them, through a custom Storage Class called `sastmp` .  A PersistentVolume referencing an Azure ephemeral disk as configured in the last article.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The process   Just as with every endeavor, let’s plan our steps before starting the journey. Configuring CAS DISK CACHE is described in the official documentation, so it will be quite simple:   Verify the current (default) configuration Configure CAS DISK CACHE Resolve issues Verify the new configuration   Since we are in a test environment, we use a “crash test method”: perform the changes, apply, and, if something does not work, fix the issues. In a production environment, steps 2 and 3 should be inverted: you should find and fix issues in your dev/test environments so that you can apply the right sequence in production.   1. Verify the current (default) configuration   From the official documentation we know that, by default, CAS servers are configured to use a directory that is named `/cas/cache` on each controller and worker pod. This directory is provisioned as a Kubernetes emptyDir and uses disk space from the root volume of the Kubernetes node. With the default CAS DISK CACHE configuration, a CAS pod uses an empyDir that points to the node root disk.   We can run some code in SAS Studio to verify this setup so that we will be able to confirm the changes after applying the new configuration. No need to reinvent the wheel, we can take an example from @Rob Collum’s article A New Tool to Monitor CAS_DISK_CACHE Usage. Here is the code: cas sess1; proc cas; session sess1; accessControl.assumeRole / adminRole="superuser"; builtins.getCacheInfo result=results; print results.diskCacheInfo; run; quit; cas sess1 clear; Sample code in SAS Studio to show the default CAS configuration.    The Results page confirms our knowledge: The CAS disk cache is located at the path `/cas/cache` (This is the path as seen by the CAS process inside the pods). On all nodes of our test environment (controller, workers), that path resides on a disk with ~127GB of capacity, on a device called `sda4` or `sdb4` . As we have verified in the previous article, these correspond to the OS root disks. This indirectly confirms that we are using a default `emptyDir` . 2. Configure CAS DISK CACHE   We will follow and adapt the instructions from the official documentation to configure CAS so that the CAS Disk Cache will use the local temporary storage we configured in the previous article.   The example in the "Tune CAS_DISK_CACHE" section shows how to mount a volume of type `hostPath` , but you can choose the type that best suits your needs.   If your environment is based on Kubernetes 1.23 or later, you can leverage the generic ephemeral volumes that became GA in that release. This combines the advantages of using a PersistentVolumeClaim with the advantages of ephemeral storage: You can specify, in the pod definition, a template volume claim that mimics a statically provisioned PVC (such as the one we used in the previous article to test the local storage) OpenShift will take care of managing the PVC lifecycle: it creates a PVC bound to the pod when the pod starts and deletes the PVC when the pod is terminated. This is perfect for temporary storage, such as CAS DISK CACHE!   Here is a sample yaml transformer, based on the one provided in the official documentation, that defines a generic ephemeral volume that references our `sastmp` Storage Class. # # this defines the volume and volumemount for CAS DISK CACHE location --- apiVersion: builtin kind: PatchTransformer metadata: name: cas-cache-custom patch: |- - op: add path: /spec/controllerTemplate/spec/volumes/- value: name: cas-cache-sastmp ephemeral: volumeClaimTemplate: spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "sastmp" resources: requests: storage: 100Gi - op: add path: /spec/controllerTemplate/spec/containers/0/volumeMounts/- value: name: cas-cache-sastmp mountPath: /cas/cache-sastmp # # mountPath is the path inside the pod that CAS will reference - op: add path: /spec/controllerTemplate/spec/containers/0/env/- value: name: CASENV_CAS_DISK_CACHE value: "/cas/cache-sastmp" # # This has to match the value that is inside the pod target: version: v1alpha1 group: viya.sas.com kind: CASDeployment # # Target filtering: chose/uncomment one of these option: # # To target only the default CAS server (cas-shared-default) : labelSelector: "sas.com/cas-server-default" # # To target only a single CAS server (e.g. MyCAS) other than default: # name: {{MyCAS}} # # To target all CAS Servers # name: .*   After creating the cas-cache-custom transformer, the process to apply it is the same as any other customization. Since it depends on the method originally used to deploy SAS Viya, please see the SAS Viya official documentation referenced above for the exact steps.   The next step now is to stop and restart the CAS server to pick up the new configuration. It is enough to terminate the CAS server pods: the CAS Operator will take care of restarting them. You can use a quick command such as:   oc -n delete pods -l app.kubernetes.io/managed-by=sas-cas-operator 3. Resolve issues   If you followed this far, you are now expecting to have a working environment with CAS using the locally attached disk as its temporary storage. And that could be true if we were using any other Kubernetes platform.   Yet, as of February 2023, we may still need a crucial step to get there on OpenShift.   After applying the changes above and terminating CAS pods, you may find that the CAS operator is unable to restart your CAS server. The sas-cas-operator pod log may contain errors such as: "sas-cas-server-default-controller" is forbidden: unable to validate against any security context constraint .   What’s going on?   Simply put, OpenShift Is doing one of its jobs: enforcing a strict security policy.   Before installing SAS Viya, as a preliminary step, the cluster administrator created some security context constraints as instructed by the Preparing for OpenShift documentation page. One of these describes the permissions granted to CAS, including a list of permitted volume types: anything else gets automatically denied.   Assuming you used the sas-cas-server security context constraint, you can check what volumes it permits with the following code:   oc get scc sas-cas-server -o jsonpath='{.volumes}'   By default, you should get a listing like the following: notice that the "ephemeral" volume we tried to use is not there.   ["configMap","downwardAPI","emptyDir","nfs","persistentVolumeClaim","projected","secret"] In summary, the missing step is to add ephemeral volumes to the list of permitted entries in the CAS security context constraint. Here is a simple command to do it:   oc -n get scc sas-cas-server -o json | jq '.volumes += ["ephemeral"]' | oc -n apply -f -   This one-liner uses the OpenShift CLI to read the current definition of the sas-cas-server security context constraint and feed it into the `jq` tool which appends the entry "ephemeral" to the array of permitted volumes. Then, `jq` feeds the result back into another OpenShift CLI that re-loads the updated definition into the cluster.   This should be all. At this point, OpenShift should successfully validate the CAS controller request to allocate ephemeral volumes for the CAS pods, and the CAS server should be automatically started.     The ephemeral disk is now used by the CAS pod 4. Verify the new configuration   As a last step, we just have to verify that the new setting is actually working (trust but verify!).   We can use the same code in SAS Studio as we did initially. The new result should give us the information we are looking for: The CAS disk cache is now located at the new path `/cas/cache-sastmp` , as we specified. This is the path as seen by the CAS process inside the pods. On all nodes (controller, workers), that path resides on a disk with ~299GB of capacity, on a different device than the previous run. We can recognize that these correspond to the Azure temporary SDD disks. Bigger, faster. SUCCESS!   Sample code in SAS Studio to show the updated CAS configuration. Conclusion   In this series of articles, we are discussing how to leverage locally attached ephemeral disks for pods running in an OpenShift Container Platform cluster. They are a convenient choice for SAS Viya temporary storage and, in this article, we have seen how to configure CAS_DISK_CACHE to use them as Kubernetes generic ephemeral volumes.     ... View more

SAS compute sessions and CAS servers require temporary storage for multiple purposes. Choosing a suitable location for SAS Viya temporary storage can be a key tuning step for optimal production deployments. In today’s cloud environments, this means addressing multiple layers of abstraction, from the underlying infrastructure, to the Kubernetes cluster, up to the applications running inside it.   In this series of articles, we explore how to provision local temporary storage to a Red Hat OpenShift Container Platform cluster deployed on Azure, and how to leverage it in SAS Viya.   Introduction   Storage design can differ considerably between infrastructure providers and from cluster to cluster. To successfully deliver a working environment, by default, SAS Viya compute servers are configured to use a minimum common denominator for temporary storage that is guaranteed to be always available. Both SAS compute sessions and CAS servers leverage a directory provisioned as a Kubernetes emptyDir, which uses disk space from the root volume of the Kubernetes node where they are running.   This default configuration is acceptable for testing and evaluation, but not for production workloads. If disk space in the root volume of the node becomes low, then Kubernetes begins evicting pods, and, in extreme circumstances, the node itself can crash.   SAS recommends using fast storage for temporary locations. Often, the best performance is provided by using disks that are locally attached to the node, such as NVMe or SSD disks. Since this is temporary storage, the disks that are used do not need to persist beyond the duration of each pod, nor follow the pod if it is moved to a different node. For this reason, ephemeral storage is ideal.   Most cloud providers offer virtual machines that include a temporary disk for ephemeral storage. In this article, we will walk through the steps we used to provision Azure ephemeral disks as a temporary location for the pods in our OpenShift Container Platform cluster. A note of caution   These articles are the result of testing within the environments we use for the “SAS Viya 4 - Deployment on Red Hat OpenShift Container Platform” workshop. For our convenience, we deploy on-demand OpenShift on Azure clusters; although that works fine for our objectives, this infrastructure is not included in the official SAS Viya system requirements. So, why discussing this in a public post? Because it can still provide value for multiple situations:   although SAS Viya on OpenShift is currently fully supported only on VMware vSphere, it is possible to deploy it on different infrastructure providers, such as Azure, under the “SAS Support for Alternative Kubernetes Distributions” policy. the high-level process and steps are similar across different infrastructures. If your server has an available fast disk, you can use it as described here, independently from how the disk itself was provisioned. Some details may need to be adjusted from the example commands shown here, but it should all be manageable by a Kubernetes admin. finally, maybe you end up reading this post and decide it can be useful to you for a different use case on a similar infrastructure.   In summary, we are not stating that your SAS Viya cluster should be designed as described here, nor endorsing this architecture as fully supported.   Provisioning temporary storage   Let’s restart from the beginning. What is the objective? To provide via OpenShift some dedicated, fast storage that can be used for SAS and CAS working directories (SASWORK and CAS DISK CACHE).   Verify the current disk status   In our environment, we are using the Azure cloud as the underlying infrastructure. @Hans-Joachim Edert discusses a similar environment in his article Some SASWORK storage options for SAS Viya on Kubernetes, listing in the “hostPath” section some Microsoft virtual machine types that can provide the storage that fits our needs. All the virtual machines used by our cluster fall in this category and, according to Microsoft’s documentation, they have an additional internal SSD drive that could be used to host temporary storage. But there is a catch: Azure VMs automatically mount this drive to their OS at the /mnt path and, if we were using AKS, this would be a mount point which could be directly used as Kubernetes storage. With OpenShift things are slightly different.   OpenShift uses a dedicated operating system – Red Hat Enterprise Linux CoreOS (RHCOS) – which, by default, does not automatically mount these disks. We know from the documentation that the disks are there, but they are almost invisible. OpenShift provides a great tool for cluster administrators to quickly interact with the underlying RHCOS: the oc debug command. Using this tool, we can operate in a privileged session, just as if we connected to the node via ssh as the root user.  Here is an example of how to use it to verify the disks available on the first worker node using the lsblk command:   # Find the 1st worker node's name WORKER=$(oc get nodes -l "node-role.kubernetes.io/worker" -o name | head -1) echo ${WORKER} # Query disk status using the lsblk command in a debug session oc debug ${WORKER} # now we are in the debug container chroot /host lsblk -o NAME,SIZE,FSTYPE,LABEL,MOUNTPOINT # exit the debug container exit exit   As an example, here is the output in our test environment:   $ # Find the 1st worker node's name $ WORKER=$(oc get nodes -l "node-role.kubernetes.io/worker" -o name | head -1) $ echo ${WORKER} node/itaedr-r-0097-worker-1 $ # Query disk status using the lsblk command in a debug session on the node $ oc debug ${WORKER} Starting pod/itaedr-r-0097-worker-1-debug ... To use host binaries, run `chroot /host` Pod IP: 10.255.1.8 If you don't see a command prompt, try pressing enter. sh-4.4#  # now we are in the debug container sh-4.4#  chroot /host sh-4.4#  lsblk -o NAME,SIZE,FSTYPE,LABEL,MOUNTPOINT NAME     SIZE FSTYPE LABEL             MOUNTPOINT sda      128G |-sda1     1M |-sda2   127M vfat   EFI-SYSTEM |-sda3   384M ext4   boot              /boot `-sda4 127.5G xfs    root              /sysroot sdb      300G `-sdb1   300G ntfs   Temporary Storage sdc        1G ext4                     /var/lib/kubelet/pods/08cef6f4-8032-449d-bcea-309355cb383c/volumes/kubernetes.io~azure-disk/pvc-fa675cfb-cd0d-49e6-843e-44adec80241fsr0 sh-4.4#  # exit the debug container ...   Let’s try to interpret this output. We should be able to understand what disks are available to this node, and to the pods running there.   The first disk is sda . This has some partitions mounted on the node OS. In the above output, sda4 is the OS root, that is seen by the debug pod as /sysroot. The disk size is 128GB, which corresponds to what we requested when we created this Azure virtual machine. The second disk is sdb and it does not show any mountpoint. It contains a single partition sdb1 ; from its label Temporary Storage we can understand that this is the temporary disk that Azure provides. Its size depends on the virtual machine type used to instantiate this node (Not all Azure machine types include a temporary disk!). The FSTYPE column show that this disk, although currently not used, contains a partition of type ntfs, which is a Windows filesystem. There can be additional disks; these are dynamically created by OpenShift as requested by running pods PVCs. In our environment the default storage class is managed-premium (kubernetes.io/azure-disk), so, for each PVC, OpenShift creates a dedicated Azure Disk. In the example above, sdc is a 1GB disk mounted on the pod with id 08cef6f4-8032-449d-bcea-309355cb383c as a volume named pvc-fa675cfb-cd0d-49e6-843e-44adec80241f   👉 on each worker node, the names sda and sdb may be switched, because linux does not guarantee the order in which disks are configured at boot time.     A diagram of the disks available on a node in our test environment Prepare the temporary disk   As we have seen above, the temporary disk, although currently not used, contains a partition of type ntfs, which is a Windows file system. In the next steps, we are going to use tools that expect/support Linux file systems. Since this is a temporary empty partition, we can overwrite it and create a new empty partition with a Linux xfs file system.   Again, we can use an OpenShift debug container, this time on all worker nodes, to re-format the temporary partition with the desired xfs file system:   # fix the partition on the temporary disk on all worker nodes AZURE_TMP_DEV=/host/dev/disk/azure/resource for _NODE in $(oc get nodes -l node-role.kubernetes.io/worker -o name); do   # run inside the debug privileged container   oc debug -q ${_NODE} -- bash -c "\     parted ${AZURE_TMP_DEV} --script -- \       mklabel gpt \       mkpart xfs 1MiB -2048s ;\     sleep 15;\     lsblk ${AZURE_TMP_DEV};\     mkfs -t xfs -f ${AZURE_TMP_DEV}-part1 \   " done   Within the previous commands we are using a trick to overcome the issue that, on some nodes, the temporary disk is sdb , while on other nodes it is sda : the node OS automatically creates a link called /dev/disk/azure/resource that points to the correct disk, and another link called /dev/disk/azure/resource-part1 that points to the first partition inside that disk. To use these links from inside the OpenShift debug container, we have to prefix /host to their paths.   Deploy the Local Storage Operator   To leverage the Azure temporary disks, it is possible to manually create local Volumes using standard Kubernetes practices, but that would require the cluster administrator to fully manage lower-level storage lifecycle operations.   OpenShift simplifies local storage management thanks to the Local Storage Operator.   This operator is not installed by default. We can install the Local Storage Operator following RedHat instructions using the OperatorHub from the web console.     Provision local storage using the Local Storage Operator   We can now use the Local Storage Operator to provision local storage for SAS Viya pods. The operator can create Persistent Volumes by looking for available file systems at the paths specified in a Local Volume custom resource. Here are a couple of yaml examples, that leverage the xfs file system previously created at /dev/disk/azure/resource-part1 :   Create a Local Volume, available on all worker nodes (OpenShift controller nodes are excluded by default). apiVersion: "local.storage.openshift.io/v1" kind: "LocalVolume" metadata:   name: "lv-sastmp"   namespace: "openshift-local-storage" spec:   storageClassDevices:     - storageClassName: "sastmp"       volumeMode: Filesystem       fsType: xfs       devicePaths:         - /dev/disk/azure/resource-part1 Create a Local Volume, that only uses a subset of nodes, i.e. only the SAS CAS nodes. In this case, the yaml definition should include a nodeSelector. As an example, you could filter on the workload.sas.com/class=cas label: apiVersion: "local.storage.openshift.io/v1" kind: "LocalVolume" metadata: name: "lv-sastmp-cas" namespace: "openshift-local-storage" spec: nodeSelector: nodeSelectorTerms: - matchExpressions: - key: workload.sas.com/class operator: In values: - cas storageClassDevices: - storageClassName: "sastmp" volumeMode: Filesystem fsType: xfs devicePaths: - /dev/disk/azure/resource-part1   You can create the Local Volume resource in the OpenShift cluster by pasting the yaml code in the web console, in the Local Storage operator page, or by saving the content in a file and then applying it: oc apply -f localVolume.yaml   The operator should automatically create the storage class referenced in the Local Volume, if it did not already exist. Then, it should start a pod to manage the storage on each node. Finally, it should create a Persistent Volume for each matching disk discovered on each node.     The disk view after defining a Local Volume that references the ephemeral disk   Test the newly referenced local storage   It's always a good practice to test your infrastructure before using it in SAS Viya – or in any production software!   Local volumes are accessed by pods through Persistent Volume Claims (PVCs).   Let's create a PVC that references our local volumes, and a pod to test the storage, with the following two yaml definitions:   kind: PersistentVolumeClaim apiVersion: v1 metadata:   name: localpvc-sastmp spec:   accessModes:   - ReadWriteOnce   volumeMode: Filesystem   resources:     requests:       storage: 100Gi   storageClassName: sastmp kind: Pod apiVersion: v1 metadata:   name: gel-test-pod spec:   containers:   - name: gel-test-pod     image: gcr.io/google_containers/busybox:1.27     command:       - "/bin/sh"     args:       - "-c"       - "df -h | grep sd && touch /sastmp/SUCCESS && ls -l /sastmp/SUCCESS && exit 0 || exit 1"     volumeMounts:       - name: sastmp         mountPath: "/sastmp"   restartPolicy: "Never"   volumes:     - name: sastmp       persistentVolumeClaim:         claimName: localpvc-sastmp   You can notice in the pod definition that we reference the PVC just created (localpvc-sastmp) and mount it at the /sastmp path inside the container, so that we can use some commands to test reading and writing from that location.   The final step is to create these resources in the OpenShift cluster. The PVCs should be in the same namespace as the pods that will use them; we can specify the SAS Viya namespace:   oc apply -f localPVC.yaml -n gel-viya oc apply -f localTestPod.yaml -n gel-viya   To check the successful allocation of the disk, we can read the log of the test pod:   oc logs gel-test-pod -n gel-viya   The test commands that we used in the pod definition should output something similar to the following:   /dev/sdb1               299.9G      2.1G    297.7G   1% /sastmp /dev/sda4               127.5G      9.2G    118.3G   7% /etc/hosts /dev/sda4               127.5G      9.2G    118.3G   7% /dev/termination-log -rw-r--r--    1 root     root             0 Jan 13 23:31 /sastmp/SUCCESS   This shows that the ~300GB temp disk has been mounted at the /sastmp path as requested, and the pod was able to successfully write there a test file called "SUCCESS".     A final view showing the ephemeral disk used in the test pod   Conclusion   In this first article, we have seen how to leverage OpenShift Local Volume Operator to make Azure VM ephemeral disks available to the pods running in the OpenShift Container Platform cluster. Now that this is done, the next step will be to use them for SAS Viya temporary storage. You can read it in the next article: SAS Viya Temporary Storage on Red Hat OpenShift – Part 2: CAS DISK CACHE    Find more articles from SAS Global Enablement and Learning here. ... View more

Starting with the 2022.10 release, SAS Viya uses Redis to provide a distributed cache technology. Redis Server and the Redis Operator replace Apache Geode and SAS Cache Server. What are these components? And how do they impact SAS Viya architecture?   A brief history   As the SAS 9 platform evolved into a clustered, multi-tier architecture, some services deployed on the middle-tier and server-tier required a distributed cache to share run-time information between the multiple web application server instances. This was implemented by embedding some java components, provided by the Apache Geode project, into SAS Web Applications, then running an instance of the Cache Locator on each machine. The Cache Locator was used by the applications to locate other members and form a distributed data cache/data store.   With the evolution from monolithic SAS 9 applications to SAS Viya 3 stateless micro-services, it became clear that the distributed cache could not be embedded anymore in these services and had to be externalized. The SAS Cache Server was born, still based on the Apache Geode java objects, and the Cache Locator is still there to support it. Together, SAS Cache Locator and SAS Cache Server provide a distributed cache technology to microservices in SAS Viya 3.   Finally, with the advent of the cloud-native SAS Viya platform deployed in Kubernetes, these components have been adapted to run in the new environments, but they were not designed to be cloud-native. The new platform exposed some deficiencies, including susceptibility to network instability, the lack of support for new services written in the Go language, and the lack of a Kubernetes operator to manage their lifecycle.     Enter Redis   Until the 2022.09 release, SAS Viya includes two Kubernetes statefulsets, one for the SAS Cache Server and one for the SAS Cache Locator, running two pods each to ensure High Availability.   Starting with the 2022.10 release, these pods are deployed but not started. Finally, with the 2022.11 release, SAS Cache Server and SAS Cache Locator are completely removed from SAS Viya.   Starting with the 2022.10 release, SAS Viya uses Redis to provide a distributed cache technology: Redis Server and the Redis Operator replace Apache Geode and SAS Cache Server.   SAS Redis server, based on the open-source Redis project, provides a distributed cache to SAS Viya microservices. SAS Redis Operator is a Kubernetes operator, developed by SAS, that manages the life cycle of the SAS Redis Server.   The default deployment creates a Redis Cluster with 3 Kubernetes Statefulsets, each one including a controller and a replica node. The cluster is managed by the operator, which runs in a stateless pod.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Redis pods as seen in Lens   The cached data is sharded (distributed according to a hash algorithm) between the 3 controllers, then replicated. In case of a controller failure, after a timeout, the corresponding replica is automatically promoted to controller and the cluster can still service its clients.     The current implementation does not support scalability: if you change the number of nodes, it does not reallocate the cached data, and the clients cannot find it anymore. For this reason, the number of controller nodes is currently hard-coded to 3 in the definition of the distributedredisclusters custom resource:   $ kubectl get crd distributedredisclusters.redis.kun -o json | jq .spec.versions[].schema.openAPIV3Schema.properties.spec.properties.masterSize { "default": 3, "format": "int32", "maximum": 3, "minimum": 3, "type": "integer" }   Although it is possible to scale up the number of replicas, no official performance testing has been conducted so far; the default implementation should be sufficient for most deployments.     Conclusion   In this article, we have introduced the new SAS Redis Server and SAS Redis Operator available with SAS Viya 2022.10 and later. For more information about the management of SAS Redis Server and SAS Redis Operator, you can consult the updated SAS Viya Administration: Infrastructure Servers page.   Find more articles from SAS Global Enablement and Learning here. ... View more