Organizations benefitting from SAS Grid Manager in their SAS 9 environments, who are eager to modernize their projects, can now move to SAS Workload Management on SAS Viya.   As organizations move their analytic and data management workloads to the cloud, they are faced with challenges in balancing cost and agility. Is it possible to balance good performance with acceptable cloud infrastructure costs? A simple lift and shift of legacy software, not specifically designed for the cloud, may seem simpler to implement than a more comprehensive redesign.  In the long term, however, that decision may result in an environment that is not optimized.   What does a grid modernization plan look like? What are the key factors to consider?   Grid Modernization, Layer by Layer   Within the Global Enablement and Learning team, we’ve been teaching SAS Grid Manager architecture, deployment, and administration to hundreds of professionals for more than a decade.   In the latest version of our workshops, we present this slide in the introductory modules:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   This slide shows a simple, high-level view. We briefly explain the “filling inside the sandwich” as follows:   SAS Infrastructure: This is the SAS 9 Platform, including the SAS Metadata Server, SAS Middle Tier components, etc. Grid Infrastructure: Also called grid middleware, includes specialized software that sits between computing processes and the underlying Operating System.   Is this relevant while discussing a modernization plan? It is, because modernizing a SAS 9 grid environment to SAS Workload Management on SAS Viya requires that you consider all the elements in the picture above and know how each one maps in the new world.   Most of the activities that you will perform use migration procedures from SAS 9 to SAS Viya described in the Full-System Migration and Content Migration documentation – the same that you would perform in environments that do not include SAS Grid Manager or SAS Workload Management.   Let’s discuss each layer of the sandwich in more detail, from the bottom up.     Provision a cloud-hosted Kubernetes environment to replace the current hardware and operating system components of the current SAS 9 environment This may range from using the SAS Viya Infrastructure as Code (IaC) projects to provision a cloud environment similar to your existing SAS 9 one, to a complete architecture redesign. See the SAS Global Forum 2020 paper “Important Performance Considerations When Moving SAS® to a Public Cloud” by Margaret Crevar. No grid-specific considerations - except, when required, the design of high-performance cloud storage. This may be a surprise for architects used to design SAS 9 environments. When SAS Grid Manager is included with SAS 9, you have specific architecture considerations, i.e. in terms of distributed storage. But remember, every SAS Viya environment in Kubernetes is, by default, distributed on multiple nodes. SAS Workload Management does not change that. Modernize the SAS platform from SAS 9 to SAS Viya This step includes deploying and configuring SAS Viya in the infrastructure prepared in the previous step. See the SAS Viya: Deployment Guide Grid specific considerations: be familiar with SAS Workload Management deployment; for example, the requirement to assign the workload.sas.com=compute label to compute nodes. You can read more in the article SAS Workload Management on SAS Viya: Deployment Architecture. Move the grid environment configuration This point covers migrating the grid itself. Re-design the grid configuration: queues, host types, grid administrators, daemon tuning, etc. See SAS Viya: Workload Management. If you are already leveraging SAS Grid Manager, with SAS Workload Orchestrator as your grid provider, you will recognize that most grid configuration settings are the same between the two environments. For SAS Grid Manager for Platform, after moving from SAS 9 to SAS Viya, you will use SAS Workload Orchestrator instead of IBM Spectrum LSF, and SAS Job Flow Scheduler instead of Platform Process Manager. Currently, there are no tools to automatically export the existing grid configuration from SAS 9 into SAS Workload Orchestrator on SAS Viya. You will have to manually re-create the grid configuration. Work is being evaluated to provide such tools to analyze the source environment and, eventually, migrate the configuration to the new target. In any case, perform an assessment of the existing environment. Business objectives may have changed, grid usage patterns evolved, and some configuration items may not make sense anymore in the new Kubernetes cloud environment. You may reach the conclusion that it may be better to re-design the grid configuration. Migrate the software clients that use the grid, including content and programs that run on the grid Be familiar with "standard" SAS 9 to SAS Viya modernization considerations. Examples include: how to migrate from SAS Enterprise Guide or SAS Data Integration Studio to SAS Studio for grid batch submission, switch from the sasgsub cli to the sas-viya batch cli. To simplify code, report, and content analysis, SAS provides the SAS 9 Content Assessment, a collection of applications designed to examine the characteristics of your existing SAS 9 system. Work is underway to add features to allow scheduled flows to be exported from SAS 9 and imported into SAS Job Flow Scheduler on SAS Viya. No other grid-specific considerations - unless the code contains explicit SAS grid syntax which may need to be evaluated, removed, or re-written. Examples include explicit references to grid nodes or the grdsvc_enable statement.   For an in-depth review on how to approach a grid modernization project, see the SAS Global Forum 2021 paper “Stretch Your Mind and Your SAS® Grid” by Diane Hatcher. It was written before the SAS Workload Management release and only addressed SAS Grid Manager on SAS 9 “uplifting” to cloud environments, but it still contains a lot of useful suggestions on how to structure and plan your digital transformation to the cloud.     Modernize Grid Capabilities   A complementary approach in designing your modernization plan is to look at SAS Grid Manager functionalities that bring value to your business, and how they map into the new world.   SAS Grid Manager provides a rich set of capabilities including workload management (through queueing and job prioritization), central administration, high availability, parallel processing, scalability, and simplified, non-disruptive, maintenance.   While it’s clear that SAS has brought SAS Grid Manager functionality to SAS Viya, it may not be so obvious what are the modernization paths. SAS Workload Management brings the core capabilities of SAS Grid Manager - workload management, parallel processing, and central administration - to the SAS Viya platform by extending the core scheduling capabilities of the underlying Kubernetes infrastructure. Other capabilities, such as scalability, elasticity, high availability, and non-disruptive maintenance, leverage Kubernetes' native features. SAS Workload Management is integrated into the Viya platform including integration with SAS Studio, SAS Compute, SAS Batch, SAS/CONNECT, and Job Flow Scheduler.   In summary, some of the grid functionality is inherent in SAS Viya’s Kubernetes architecture. Other features are provided by SAS Workload Management which is layered on top of SAS Viya. To get a more comprehensive review, capability by capability, see the article SAS Grid and SAS Viya: together to provide advanced workload management. Also be on the lookout, in the coming months, for additional grid and migration related articles provided by GEL.   Conclusion   Modernizing a SAS 9 grid environment to SAS Workload Management on SAS Viya does not require complex procedures, but does require proper planning. At the end of the journey, you will be able to leverage new capabilities not available before, such as dynamic scaling up to accommodate increased workloads, and down to control cloud costs.   Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Workload Management is generally available on SAS Viya starting with the October/November 2021 releases, including the 2021.2 Long Term Support cadence and the 2021.1.6 Stable cadence.   It’s an add-on offering in the SAS Viya portfolio: this means that it can be added to any SAS Viya product to provide capabilities currently present in SAS Grid Manager on SAS 9, as explained in the post SAS Grid and SAS Viya: together to provide advanced workload management.   How is the architecture impacted when SAS Workload Management is added to a SAS Viya environment? What additional pods are deployed in Kubernetes?   SAS Workload Management Deployment Architecture at a Glance   SAS Workload Management provides multiple additions to a SAS Viya deployment:   SAS Workload Orchestrator Manager – implemented as a Kubernetes Statetefulset SAS Workload Orchestrator Daemon – implemented as a Kubernetes Daemonset SAS Workload Orchestrator Page in SAS Environment Manager SAS Workload Orchestrator Dashboards in Grafana Plus additional Kubernetes artifacts such as supporting configmaps, secrets, service account, etc.   It’s worth noting that, just like with SAS Grid Manager on SAS 9, a deployment of SAS Workload Management happens “inside” the corresponding SAS Viya environment, i.e. it is deployed inside the same SAS Viya namespace with the pods it will manage. For this reason, there is a 1-1 relationship between each SAS Workload Management deployment and SAS Viya deployment.   The following diagram shows a sample Viya environment deployed on 8 nodes. In this example, nodes 1-2-3 are dedicated to stateful and stateless pods; nodes 4-5 are for compute servers, and nodes 6-7-8 host an MPP CAS instance. The blue horizontal band highlights that, when SAS Workload Management is added to the license, additional pods are deployed on the nodes.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     Let’s review the 4 components listed above, and the pods that host them.   SAS Workload Orchestrator Manager SAS Workload Orchestrator (SWO) is the main component of SAS Workload Management. This managing component hosts the “business logic”, i.e. the engine of SAS Workload Orchestrator in charge of receiving job submissions, taking scheduling decisions, maintaining the state of the system, and so on.   It is implemented as a Kubernetes Statetefulset, represented in the diagram above with the red pods on the left. For High Availability considerations, the statefulset is defined with 2 replicas by default: the environment will have two pods, called sas-workload-orchestrator-0 and sas-workload-orchestrator-1. By default, sas-workload-orchestrator-0 is active, while sas-workload-orchestrator-1 is on standby, ready to step-in in case the other fails. They have the same labels and tolerations as other SAS stateful pods, so they are started by Kubernetes on the same nodes.   SAS Workload Orchestrator Daemon SAS Workload Orchestrator Server instances run on every compute server node with the objective of monitoring the node resources and managing the locally running jobs. It then shares this information with the SAS Workload Orchestrator Manager and receives from the Manager the commands to start/stop jobs on its node.   The SAS Workload Orchestrator Server Daemon is implemented as a Kubernetes daemonset with a request to run pods on nodes with the workload.sas.com/class=compute label.   It is important to note that when SAS Workload Management is licensed, it becomes the only entity in charge of starting new SAS compute sessions, by leveraging the SAS Workload Orchestrator Daemon to perform the start/stop commands on the nodes where it is running. For this reason, it becomes mandatory to assign the workload.sas.com/class=compute label to at least one node, and possibly more. Otherwise, the daemon pods will remain forever pending, and no compute jobs will ever be started.   SAS Workload Orchestrator currently does not manage any workload running inside CAS. For this reason, there are no SAS Workload Orchestrator Server Daemon pods running on nodes dedicated to CAS.   SAS Workload Orchestrator page in SAS Environment Manager   SAS administrators interact with SAS Workload Orchestrator using a new dedicated page in SAS Environment Manager:     This page becomes available when SAS Workload Management is licensed and permits to manage, monitor, and configure SAS Workload Orchestrator. In detail SAS administrators can:   monitor and manage jobs that are sent to SAS Workload Orchestrator queues monitor, manage, and define queues monitor hosts running jobs from queues define host types view SAS Workload Orchestrator logs and manage log levels Administrators used to work with SAS Workload Orchestrator on SAS 9 will be immediately familiar with this tool: most of the functionality has been ported to this new release, maintaining a similar interface to streamline the migration experience between the two versions. SAS Workload Orchestrator Dashboards in Grafana A SAS Workload Management license gives access to dedicated dashboards in Grafana to monitor the status of cluster nodes and jobs.     If you choose to install the SAS Viya monitoring components, the SAS Launched Jobs - Node Activity and SAS Launched Jobs - User Activity Grafana dashboards are provided to display information about SAS jobs. You can filter the information in the dashboards by queue or by job name, as well as by other criteria.   For more information, see Use the SAS Launched Jobs Dashboards in the official SAS documentation.   Deployment Considerations Deploying SAS Workload Management is as simple as adding its license to the SAS Viya environment that you are going to install. After this, follow normal SAS Viya instructions to install the software, and all the components that we have described will be there.   To enable full functionality, though, a Kubernetes administrator may have to perform a few additional steps. Define ClusterRoles and ClusterRoleBindings The SAS Workload Orchestrator daemons require information about resources on the nodes that can be used to run jobs. In order to obtain accurate resource information, add a ClusterRole and a ClusterRoleBinding to the SAS Workload Orchestrator service account. This can be done during the deployment by adding the sas-workload-orchestrator overlay to the resources block of the base kustomization.yaml file:   resources: ... - sas-bases/overlays/sas-workload-orchestrator This can also by done by a Kubernetes administrator by using the ClusterRole and ClusterRoleBindings yaml files found in the same directory. Tune memory and cpu limits and resources for SAS Workload Orchestrator pods. The memory limit request values for the SAS Workload Orchestrator stateful set containers or daemon set containers might need to be increased so that more jobs can be processed. The default settings provide reasonable values for an initial deployment but may need tuning as more jobs get submitted to the environment. Use node labels to control where SAS jobs run. We have already seen that SAS Workload Orchestrator can only run jobs on nodes on which the SAS Workload Orchestrator daemonset is present and that the daemonset requires the workload.sas.com/class=compute label. Kubernettes administrators can define additional labels and assign them to nodes; SAS administrators can then reference those labels in queue definitions via host types definitions. In this way, it is possible to segregate certain types of processing to specific nodes – for example, batch or interactive jobs, or jobs that require specific hardware capabilities. All these tasks are described in the official SAS documentation, in the page SAS Workload Orchestrator Tasks for Kubernetes Administrators.   Conclusion   SAS Workload Management extends the workload management capabilities of Kubernetes bringing SAS Grid Manager capabilities to SAS Viya. Including SAS Workload Management in your SAS Viya license impacts the deployment architecture with additional pods deployed in the environment and provides dedicated management interfaces for SAS Administrators. You can read @FrederikV's hands-on experience with SAS Workload Management in his recently published article "SAS Workload Management"    Find more articles from SAS Global Enablement and Learning here. ... View more

In the SAS Global Forum 2020 paper "SAS® Grid Manager and SAS® Viya®: A Strong Relationship" you can read how SAS Viya 3.x and SAS Grid Manager on SAS 9 complement each other in providing a highly available and scalable environment to process large volumes of data and produce rapid results.   The paper addresses questions raised by customers who are trying to understand how to leverage the two separate technologies to get the most value from using them together.     Since then, two major shifts happened that partly change some of the perspectives offered in the paper:   a general industry move from on-prem data centers to cloud environments the availability, at the end of 2020, and subsequent increasing customer adoption, of a new SAS Viya release, capable of leveraging cloud environments thanks to its integration with Kubernetes.   A third change is coming in November 2021, with the first official release of SAS Workload Management on SAS Viya.   So it’s natural now to ask the question that stems from these changes: why would I want to leverage this new offering?   The immediate answer is easy: our customers see tremendous value in managing their SAS 9 workloads with SAS Grid Manager, so they’re excited to realize many of those same capabilities with SAS Workload Management in SAS Viya.     With its first release, SAS Workload Management on SAS Viya will be able to manage jobs submitted using the SAS launcher service, such as SAS Compute servers, SAS batch, and SAS/CONNECT workloads. Initially, there will be no advanced workload management for other engines, such as SAS Cloud Analytic Services (CAS) or SAS Micro Analytic Service (MAS).   At a first glance, Kubernetes already provides workload management, so why would a customer with SAS Grid Manager want to keep the license when moving to the new SAS Viya platform?   To provide a useful explanation, let's focus on actual customer needs. What are the benefits that drive customer adoption of SAS Grid Manager?   SAS Grid Manager Benefits If you are familiar with SAS Grid Manager on SAS 9, you may recognize the features that it provides:   Multi-user workload balancing Parallelized workload balancing Centralized workload management and monitoring Distributed enterprise scheduling High availability Scalability   These capabilities enable benefits for business users, such as better response times, thanks to better usage of available resources and ability to parallelize processing on different machines, as well as control over priorities, uninterrupted business, and self-managed control of personal jobs (monitoring, suspending, terminating).   IT departments may benefit from more efficient use of available resources, thanks to better sharing of computing resources across users and business units, as well as increased governance and incremental scalability of the hardware environments.   If all you need is basic distribution of jobs, auto-restart of failed services, and batch scheduling, then SAS Viya on Kubernetes can already do all of that. But if you want advanced features you’re already familiar with, which are offered by SAS Grid Manager, such as queues, priorities, pre-emption, then SAS Workload Orchestrator – the main component of SAS Workload Management on SAS Viya – is for you.   Multi-User and Parallelized Workload Balancing Kubernetes provides capabilities to match workloads to compute nodes so that jobs can be scheduled and executed. It determines which nodes can be used to start pods for work, based on available resources. SAS Workload Orchestrator extends the workload management capabilities of Kubernetes by adding priority-based queues, which provide the ability to control where and when jobs are processed based on prioritization and resource allocation.   Workload capabilities satisfied by SAS Viya on Kubernetes:   SAS Compute jobs submitted by multiple users are spread out among the available cluster nodes Kubernetes ensures no machine becomes overloaded (CPU, memory, disk monitoring) If a SAS processing task is made up of many independent sub-tasks, it is possible to manually rework it in such a way that each of the sub-tasks is run in parallel on the cluster, on different pods.   Additional workload management capabilities provided by SAS Workload Orchestrator on SAS Viya:   SAS Compute jobs can be placed in a waiting queue if necessary SAS Workload Orchestrator can assign priority to some users or applications over others SAS Workload Orchestrator can take actions on jobs – such as suspending, resuming, or terminating – based on business rules defined in advance SAS Workload Orchestrator can manage hosts (open and close) so that SAS Compute jobs are allowed or prevented from running there.   Centralized workload management and monitoring Kubernetes administrators can easily manage and monitor any workload running inside a cluster, but this capability is usually not available to end-users, and maybe not even to SAS administrators.   If you have a SAS Workload Management license, the Workload Management page in SAS Environment Manager makes the following tasks available:   monitor and manage jobs (pods that are launched on Kubernetes by SAS Workload Orchestrator) monitor and manage queues, which control where and when jobs are processed based on prioritization and resource allocation monitor and manage hosts, which are compute nodes (machines) that process the jobs from the queues view the SAS Workload Orchestrator logs and specify logging levels for the SAS Workload Orchestrator loggers configure SAS Workload Orchestrator, including defining queues and host types, specifying how workloads are managed, and specifying user access to queues   End-users can perform these tasks only on their own jobs, or on queues on which they have administrative rights. SAS administrators have full access to all these capabilities.   Distributed Enterprise Scheduling SAS Viya includes by default scheduling capabilities so that SAS jobs can be scheduled to run at a specific time. Triggered SAS jobs are started by Kubernetes on the best available node according to its own default policies.   SAS Workload Orchestrator applies the additional workload management capabilities previously highlighted to scheduled jobs, too. Remember that, on Kubernetes, scheduling a job to run at 3:00 AM does not mean that the job will run at 3:00 AM. It means the job will be submitted to Kubernetes at 3:00 AM, but, if at that time the cluster is too busy and has already reached its scalability limits, scheduling may timeout and fail. With SAS Workload Orchestrator jobs are held in priority queues: if the cluster is full, scheduled jobs may have to wait until their priority allows them to run, but they should eventually be launched on a node and run.   SAS Workload Orchestrator can also be configured to enforce preemption of running low priority jobs to allow time-sensitive scheduled jobs to run first – the low-priority jobs will be terminated and then restarted when possible.   High Availability – Scalability As you can read in the post SAS Viya High Availability: Fault Tolerance with the 2020.1 and Later Releases, fault tolerance can be provided by availability capabilities that exist at multiple levels   Infrastructure (Cloud Provider):​ cloud environments provide availability to the infrastructure (i.e. storage, load balancers, Kubernetes control plane, all have a guaranteed uptime). Kubernetes: it ​provides a basic level of availability by automatically distributing services on multiple nodes, monitoring and restarting failed pods, and routing communication only to healthy instances. SAS Viya: ​application servers and services can be clustered to increase their availability.   Clustering at the infrastructure, Kubernetes, or SAS Viya levels also provides scalability capabilities. Although SAS Viya servers and services – including CAS – are usually configured to run with a fixed number of pod replicas, SAS Compute sessions are dynamically launched with multiple pods, scaling up as long as there are enough hardware resources – or a limit set by an administrator is reached.   For these reasons, availability and scalability are areas where the capabilities provided by SAS Grid Manager to SAS 9 environments are already included in any default SAS Viya deployment on Kubernetes. A big win for SAS customers!   Bonus SAS Viya on Kubernetes includes by default a batch server, which can run batch jobs submitted using the batch plug-in for the SAS Viya command-line interface.   This batch client CLI can submit SAS programs or commands from a command line to a SAS Viya environment running in a Kubernetes cluster for batch processing. It can even start an interactive SAS session in a Kubernetes pod. If you are an experienced SAS Grid Manager on SAS 9 user, you may already see where I am going.... does this sound familiar? Compare it with the capabilities provided by the SAS Grid Manager Client Utility (SASGSUB). They are really similar: in fact, the new batch CLI is SASGSUB, ported to SAS Viya.   Another capability included by default in every SAS Viya deployment on Kubernetes!     Conclusion SAS Workload Orchestrator is a welcomed addition to the SAS offerings available on the newest SAS Viya platform. Customers will be able to benefit from advanced workload management capabilities, priority-based queues for SAS jobs, and more granular control on workload placement.   Find more articles from SAS Global Enablement and Learning here. ... View more

In previous articles, we've discussed SAS Viya and CAS availability capabilities and how the integration with Kubernetes streamlines and simplifies the management and recovery of failed components.   How does that translate to end-user experience?   Designing and managing the availability and failover capabilities of an enterprise system such as SAS Viya can involve multiple architectural levels, including infrastructure services, platform components, and deployed applications. Software architects, SAS administrators, and installation engineers have to consider and discuss how to best configure all these layers, but, in the end, what is really important is to address how users are (or, better, are not) impacted.   Given this objective, I decided to record a demo of a SAS Viya environment where you can see a real case of a client application, SAS Visual Analytics, surviving a CAS worker failure, thanks to proper system configuration, in order to create an optimal user experience.     Conclusion With this demonstration, I hope you've seen how, thanks to a properly configured SAS Viya environment, users can keep working while a backend disruption happens to a critical component such as a CAS worker. In the end, user satisfaction is the objective of every SAS administrator!   Find more articles from SAS Global Enablement and Learning here. ... View more

In my previous article, I presented some considerations to ponder when architecting High Availability with SAS Viya. This article covers more details about the actual capabilities and configuration settings of SAS Viya deployments for the 2020.1 and later releases. Fault Tolerance at multiple levels As you can remember from the previous article, fault tolerance can be provided by availability capabilities that exist at multiple levels: ​   Infrastructure (Cloud Provider) ​ Cloud environments provide availability to the infrastructure (i.e. storage, load balancers, Kubernetes control plane, all have a guaranteed uptime). Kubernetes cluster​ Kubernetes provides a basic level of availability by automatically distributing services on multiple nodes, monitoring and restarting failed pods, and routing communication only to healthy instances. SAS Viya deployment​ Viya servers and services can be clustered to increase their availability.   Let’s enter into more details about the default capabilities for each one.   Infrastructure Cloud providers guarantee, by default, the availability of many IAAS resources to a level that would have required careful design, high cost, and complex maintenance to any on-prem data center just a few years ago.   Using a computing node (virtual machine) as an example, an individual VM underpinning an AKS cluster on Azure can guarantee an SLA of 99.9% without any special configuration (source: https://docs.microsoft.com/en-us/azure/architecture/high-availability/building-solutions-for-high-availability#sla-offered-by-availability-zones). This means less than 45 minutes of downtime per month!   Using cloud-provided storage for Persistent Volumes can also guarantee a default SLA level for your data and configuration files, depending on the chosen storage provider.   To achieve even better resilience, it is possible to deploy your Kubernetes cluster across multiple zones (in the same region) to protect it from data center outages. You can choose to spread across multiple zones only the managed Kubernetes control plane, only the nodes, or both. Control plane availability impacts the capability to control, monitor, and configure your cluster (including starting/stopping/scaling resources); nodes availability impacts SAS Viya availability.   While multi-zone clusters can improve availability for end-users, they may come with additional considerations:   Performance – Crossing zone boundaries may introduce network latencies that can slow down your applications – CAS workers are particularly impacted. Cost – Network traffic between different zones may not be free. Storage architecture – Mounting Persistent Volumes (PV) across different zones may not be possible, or limit pod placement to a specific zone. For example, in AKS, PV implemented using Azure Disks cannot be mounted to pods running in different zones than the one where they were initially created. In GKE, each regional PV can only be replicated in 2 of the 3 zones in which a cluster can be deployed. Specialized storage such as Azure NetApp Files does not suffer these limitations. Resiliency – If one AZ is lost, do you have enough nodes in the surviving AZ to run all the pods that should be moved? Including CAS? Kubernetes Kubernetes provides multiple artifacts and automations that impact the availability of your applications, and SAS Viya 2020.1 has been designed to take native advantage of many.   Kubernetes Controllers In previous SAS releases, in case of server/service failure, there were few options to restart them: Manual intervention by an administrator 3rd party/OS High Availability tools SAS Grid Manager High Availability management In Kubernetes, the pods are managed by controllers. A controller acts on the current state of the managed pod, to come closer to the desired state: this includes automatically restarting failed or unresponsive pods. Examples of controllers are Statefulsets and Deployments with Replicasets. SAS Viya includes custom controllers in the form of Operators to manage some components such as CAS and Elasticsearch. Probes Kubernetes can monitor the life lifecycle of applications via probes. Probes can take different forms, such as checking a TCP endpoint, checking an HTTP endpoint, or executing a command and validating its return code. SAS Viya pods include probes to allow Kubernetes to monitor their startup, liveness, readiness. Kubernetes can react to probe failures, for example by automatically restarting stuck pods, or allowing communication only to responsive pods. Horizontal Pod Autoscaling Horizontal Pod Autoscalers (HPA) are specialized controllers that can automatically scale pods based on dynamic conditions, such as CPU utilization. For the current release, SAS Viya can optionally use HPAs to keep 2 replicas of Stateless Services up and running. Pod disruption budgets Pod disruption budgets (PDB) limit the number of Pods of a replicated application that can be down simultaneously from voluntary disruptions. For example, when draining a node, Kubernetes will make sure that killing pods will not result in services going below their minimum PDB. SAS Viya specifies PDBs for quorum-based applications (Consul, RabbitMQ) to ensure that the number of running replicas is never below the minimum number needed to maintain a quorum. For every other service, PDBs ensure that at least one replica is always running when using HPAs. Soft anti-affinity for pods Pod definitions can include affinity and anti-affinity to tell Kubernetes on which nodes they should be scheduled. If all pods of the same service are on the same node, failure of the node completely disrupts the availability of that service, and draining the node cannot be completed since there would be no surviving instances of that service. SAS Viya specifies soft anti-affinity for stateless and stateful services to ensure that identical pods have a preference to be scheduled on separate nodes (but they are not required to be, in case there are no other options) SAS Viya SAS Viya servers and services can be clustered to increase their availability. With clustering, if a member of the cluster goes down, the surviving ones keep servicing client requests. Stateful services are configured for High Availability by default at initial deployment. By default, your SAS Viya environment starts with: 3 replicas for Consul 3 Replicas for Postgres 3 Replicas for RabbitMQ 2 Replicas for CacheLocator 2 Replicas for CacheServer 1 Replica for everything else   CAS High Availability can be enabled by choosing to deploy across several nodes (MPP) and with a backup controller – as was discussed in a previous article.   An optional Kustomize transformer can be used to enable two replicas for the stateless microservices.   In a similar way, another Kustomize transformer can configure a highly-available cluster for Open Distro for Elasticsearch, as described in this article.   Do not forget that your environment has dependencies on additional software that can be running in other namespaces in the same cluster as SAS Viya. For example, software like cert-manager, NGINX Ingress Controller, and the SAS Viya Monitoring solution for Kubernetes might be critical to the availability of SAS Viya and may have been deployed by default with single replicas, making them less highly available than SAS Viya itself. To increase the availability of these additional software solutions, consult their documentation for more information.   Conclusion SAS Viya integration with cloud platforms is not a simple lift and shift of previous technologies. You can get real benefits from the capabilities provided at multiple levels by infrastructure services, platform components, and deployed applications, to provide end-users a better experience while managing costs and complexity.   Find more articles from SAS Global Enablement and Learning here. ... View more

One of the enterprise capabilities that our customers often request during the architecture planning of a new implementation is High Availability. Many times, this comes in form of a technical request: “Can I cluster SAS Viya?” or “How can I create multiple instances of service X?”. Occasionally, the focus of the question is placed on the consequences of a possible failure: “What happens to the data loaded in memory in case of failure?”.   Sometimes, asking the right question is more difficult than articulating a satisfying answer. If the question already hints at a pre-conceived solution, any explanation that deviates from the anticipated line may seem incomplete, at least partially.   When discussing a broad topic such as High Availability, it’s easy to stick to concepts and solutions that proved valuable in past implementations. Maybe they are not the best when moving to a newer, cloud-native platform such as SAS Viya, where multiple infrastructure layers can provide different capabilities, all working together to increase the overall system resiliency to failures. Technical Capabilities and Business Requirements What capabilities does the system have? How are they implemented? When confronted with these questions, I tend to ask back: What business requirement are you trying to address?   Yes, if you are a SAS Administrator tasked with the role of maintaining happy users, it’s important to know how SAS Viya works, how services are configured and managed. But, if you are an Architect designing the environment, it’s more important to focus on business requirements and how to address them.   Let’s see how this translates when discussing SAS Viya availability. Capabilities These are technical capabilities of a system such a SAS Viya deployment   Availability. You can deploy multiple instances of a server or service so that, in case one fails, the others are still available to provide service​. This is a basic capability often provided by clustering   Failover. This is a step further than availability: in case of failure, the services that were provided by the failed instance are moved over to the surviving ones​. This capability requires additional techniques, such as session replication, using locks to synchronize transactions, etc.     Non-functional Requirements Implementing capabilities such as the ones just highlighted can satisfy multiple business objectives: Fault tolerance (FT). The overall system can tolerate the failure of one or multiple components without disruption of services provided to end-users. Availability and failover can be tools to provide fault tolerance. Building on SAS 9 experience, fault tolerance has been a key goal for SAS Viya since the beginning: SAS is used for enterprise-class systems and customers expect a reliable system. Non-Disruptive Updates (NDU). Software updates can be applied online and "during working hours"; active users of the system do not perceive any outage. An administrator should be able to perform the following operations without impacting live users of the system: Change configuration Apply patches and fixes Upgrade the software to a supported release Install add-ons and changes in products This is a new and more complex objective of SAS Viya 2020.1 and later; although many services satisfy this requirement, a few still require user disruption (such as stop and restart). Disaster Recovery (DR). It’s the ability to re-establish service, within an acceptable timeframe, if a major disruption occurs to the facility hosting the environment. Service (and data) can be restored using a separate environment. Although often confused with High Availability, DR is a different business requirement, addressed by implementing proven business practices on top of software capabilities. Finding the right question How are users impacted by a system failure? This is probably the most relevant question. In fact, by moving the focus from capabilities to business requirements, it becomes natural to assess how users are affected. ​We can evaluate the business impact with a classification like the following:   Level Description None No customer impacts. Small A failure could cause some user disruption, but the system will recover on its own in a few minutes. Medium A failure will cause user disruption and the system will not recover on its own. User intervention is required (e.g. users need to log back on). Users may need to recreate work in progress. Large A failure will cause user disruption and the system will not recover on its own. An administrator has to act to re-establish a functional environment for end-users.   SAS Viya Architecture and Deployment Choices​ After assessing what is the acceptable level of user disruption in case of failures, it’s possible to focus on the actual High Availability configuration required to satisfy it. SAS Viya balances costs and agility with a cloud-native platform and built-in automation. In fact, availability capabilities exist at multiple levels: ​ Infrastructure (Cloud Provider) ​ Cloud environments provide availability to the infrastructure (i.e. storage, load balancers, Kubernetes control plane, all have a guaranteed uptime). Kubernetes cluster​ Kubernetes provides availability by automatically distributing services on multiple nodes, monitoring and restarting failed pods, and routing communication only to healthy instances. SAS Viya deployment​ Viya servers and services can be clustered to increase their availability. You should consider all levels, as they build on each other. They all influence the assessment of how failures can impact users: ​ What are the SAS Viya availability requirements (RTO, RPO)? ​In other terms, how much downtime is acceptable (minutes, hours)? Maybe the SLA provided by the infrastructure is enough to satisfy these business requirements. What is the desired balance between cost, performance, availability? (pick 2) ​ Depending on the availability needs of your specific environment, the cost and design complexity will vary. Is the Kubernetes cluster itself highly available? Is it deployed across multiple zones? ​ How many pod replicas should be used? ​Is the default SAS Viya configuration enough? Conclusion In this article, we have seen the importance of having the right perspective when designing a highly available SAS Viya environment. Before diving into technical details about how services can be configured, consider the user and business requirements, the capabilities available at all levels (Cloud Infrastructure, Kubernetes Platform, SAS Viya), and how they all relate. Maybe you will find that a default deployment already satisfies all your needs!   In the next article, you will find the levers and knobs at your disposition, across these multiple levels, to influence SAS Viya resilience to failures.   Find more articles from SAS Global Enablement and Learning here.   ... View more

Starting with SAS Viya 2020.1 and beyond, the SAS Cloud Analytic Services server can leverage Kubernetes capabilities to simplify management and operations. Service availability is one key area where this integration provides enhanced capabilities when compared to previous releases. In this article, we'll look into some of the integration details to help SAS Administrators build and maintain a reliable SAS Viya environment.   CAS Availability Architecture: the Basics. SAS Cloud Analytic Services (CAS) can be deployed in a distributed analytic cluster, to use multiple machines for AI, data management, and analytics (this is also called MPP mode - Massively Parallel Processing mode). In this configuration, CAS is more resilient to failures: even if a CAS worker node fails, the service as a whole is still available.   Starting with SAS Viya 3.3, CAS can also have a backup (secondary) controller. The two controllers continuously keep themselves in sync; this enables the backup controller to provide service rapidly in case of failure of the primary. To detect failures, the controllers exchange heartbeat messages every few seconds.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.    A CAS backup controller can only be used in a distributed server architecture: there are no failover capabilities for single-node CAS SMP. Deploying a backup controller is optional, and each CAS deployment supports at most one primary and one backup controller.   The primary and backup controllers share some specific storage directories, using RWX PVCs (see Rob Collum's Kubernetes Primer for SAS Viya: Storage for CAS for additional info on CAS storage). This is a significant improvement compared to SAS Viya 3: in that release, each controller has a dedicated, local permstore area, that can get out of sync in case of failover. To recover, SAS Administrators have to synchronize the permstore manually as documented here.   Managing the pods Let's now discuss some possible failure scenarios, and see how the system reacts. Before covering the details, it's important to understand how, starting with SAS Viya 2020.1 and beyond, CAS can leverage Kubernetes capabilities. SAS Viya provides a Kubernetes CustomResourceDefinition which defines, amongst a CASDeployment custom resource, the operator that manages it: the CAS operator. The CAS operator manages the lifecycle of CAS pods; focusing on CAS availability, the CAS operator monitors the pods and manages their automatic restart policies.   1. CAS SMP CAS SMP runs inside a single pod; if it fails, the CAS operator restarts it. All active connections and sessions are terminated; any in-memory data is lost. The new pod may end up on a different node and get a new internal IP address. This is not a problem for CAS clients that want to re-connect: the Kubernetes service sas-cas-server-default-client, used to reach the CAS controller pod, gets automatically re-routed to the new pod.     2. CAS MPP without a Backup Controller What if the controller of an MPP CAS server without a backup controller fails?   Again, the CAS operator manages the restart policies for the CAS server pods; as soon as it registers the controller failure or termination, all pods (including the workers) are automatically deleted and restarted. The reason for this behavior is simple: after the only controller is gone, all CAS sessions are lost and workers become useless. Forcing a complete restart enables the CAS operator to reset CAS to a clean state. Obviously, this implies that all active connections and sessions are terminated, and any in-memory data is lost.   Just as in the previous case, each pod may end up on a different node and get a new internal IP address; the Kubernetes service sas-cas-server-default-client gets re-routed to the new controller pod and clients can reconnect.   3. CAS MPP with a Backup Controller If the controller of an MPP CAS server with a backup controller fails, the result is different. The CAS operator declines to restart a failed controller – either primary or backup – when the other one can step in. In this case, if the primary controller dies or is terminated, it remains in a stopped state and its pod gets evicted.   The backup controller takes control of the cluster, and the Kubernetes service sas-cas-server-default-client gets re-routed to the backup controller pod.     After this transfer, all active connections and sessions are maintained, any in-memory data is preserved, and processing continues uninterrupted.   Starting with SAS Viya 2020.1 and beyond, there is an important difference in restart behavior when compared to previous versions. With SAS Viya 3.x, when an administrator gives a stops command to the primary controller, CAS interprets it as a command to stop the whole cluster, and all nodes come down. With SAS Viya 2020.1 and beyond, when an administrator stops the primary controller, it is considered a deviation from the desired state, and the CAS operator starts a failover to the backup controller, just as if it were a failure. The reason for this change is that now we use a declarative approach, versus the traditional imperative one. And, if you want to know, the proper way to stop the CAS server is to describe the desired "shutdown state" as "true":   kubectl -n name-of-namespace patch casdeployment default --type=json -p='[{"op": "add", "path": "/spec/shutdown", "value":true}]'   Recovering CAS server from a failure As we have seen in the SMP or MPP cases without a backup controller, the CAS operator reacts to a failure of the primary by restarting the whole CAS server: nothing else is left for the administrator to recover. Yes, they may need to reload data and manage client disconnections, but nothing is left to be done with the backend.   Instead, when a backup controller steps in after the primary controller has failed, we have seen that the CAS operator declines to restart it. How can an administrator get back to the initial situation, with the primary in charge of the CAS cluster?   For the current release, recovering from a controller failure requires a planned outage. An administrator has to stop and restart the CAS server: after CAS restarts, the primary controller resumes its role and the backup controller falls back to providing fault tolerance.   4. What about CAS Workers? So far, we have seen multiple scenarios covering the CAS controllers. What about CAS workers? The rule is always the same: the CAS operator manages the restart policies for the CAS server pods. As soon as it registers a worker failure or termination, it starts a new pod to take its place. The new CAS worker starts empty: no loaded data, no user sessions. This is not an issue: data availability is still addressed as it is with previous SAS Viya releases; if tables are loaded in-memory with redundant copies, they are still there after a worker failure and restart. One of the surviving workers activates or loads the data blocks for tables that were previously managed by the failed worker, and CAS keeps working.   Closing Discussing the availability of an enterprise system such as SAS Viya can be a complex topic that can be seen from multiple viewpoints. In this article we have covered an overview of CAS availability with SAS Viya 2020.1 and beyond for SAS Administrators, highlighting how the integration with Kubernetes streamlines and simplifies the management and recovery of failed components. A different perspective could be to consider how client applications react to CAS failures and how users are (or are not) impacted. That will be the topic of another article.   Find more articles from SAS Global Enablement and Learning here.   ... View more

SAS Viya has fully embraced a Continuous Delivery strategy, and new functionality gets released monthly. Sometimes this includes new components and changes or additions to the architecture; it’s important for SAS Administrators to keep up to date with the software.   The SAS Viya 2020.1.3 stable release includes an open-source distribution of Elasticsearch providing search capabilities to SAS Viya applications. This post describes its architecture and deployment in 7 points.   Those of you who are have worked with SAS Visual Investigator, which is included in offerings such as SAS Intelligence and Investigation Management, may be already familiar with Elasticsearch. The same holds if you are already using the SAS Viya Monitoring for Kubernetes framework. Starting with the 2020.1.3 stable release, it will also empower search capabilities with SAS Information Catalog.   1.      What software is this? Starting with the 2020.1.3 stable release, SAS Viya includes a framework built around Open Distro for Elasticsearch, in short ODFE.   ODFE is an open-source distribution, covered by an Apache 2.0 license, of Elasticsearch, enhanced with open-source, enterprise-level plugins.   Elasticsearch itself is a distributed search engine.   Why this nesting? It all boils down to licensing and legal terms. Although the core Elasticsearch is open-source, most of its enterprise-level plug-ins (i.e the ones providing secure access) are covered by a commercial license. On top of it, Elasticsearch (the company owning the trademark and the copyrights) has recently changed the license of Elasticsearch (the product) to SSPL, in practice prohibiting any commercial offering of managed hosting services, to counter Amazon Elastisticsearch Service.   ODFE, sponsored by Amazon, avoids these legal issues by providing both the core and select enterprise-level plug-ins under the Apache license.   SAS Viya provides Open Distro for Elasticsearch 1.7.0 as a base, upgraded to work with Elasticsearch 7.6.2, along with various ODFE plug-ins and a SAS-built Kubernetes operator to handle their installation and management.   Included ODFE plugins: opendistro_security, opendistro_index_management, opendistro-job-scheduler.   We also include additional Elasticsearch plugins, without modification: analysis-icu, analysis-kuromoji, analysis-nori, analysis-phonetic, analysis-smartcn, analysis-stempel, mapper-murmur3.   A final note about the release. SAS provides a version compatible with Elasticsearch 7.6.2 because this is the latest version currently supported by Janusgraph (see below what this is). As soon as possible (depending on Janusgraph) SAS will probably upgrade to Elasticsearch 7.10 (the latest version fully covered by the Apache license).   2.      In which Software Offerings is it included? For the current release, Open Distro for Elasticsearch is included in SAS Information Catalog, which is included in all SAS Viya offerings except SAS Data Science Programming.    SAS Information Catalog stores information inventory for SAS Viya data assets in the SAS Infrastructure Data Server (PostgreSQL), then uses ODFE to index pointers to that content. Indexed information is then used to search content. SAS Information Catalog indexes various information about the objects that are getting pointed to, like name, descriptive text, keywords, etc. In future releases of SAS Viya, by upgrading the license to SAS Information Governance, you will have the option to use a specialized graph database (such as JanusGraph) instead of PostgreSQL.   In the future, additional SAS Viya applications may be able to leverage the same ODFE provided with SAS Information Catalog.   3.      How is it packaged? SAS Information Catalog includes the following container images: sas-catalog-services sas-information-catalog-app sas-opendistro sas-opendistro-operator   (SAS Information Governance will add sas-janusgraph in a subsequent release)   Of these, Open Distro for Elasticsearch is implemented by pods containing the sas-opendistro container, while the operator pod uses the sas-opendistro-operator container. The operator also uses a Kubernetes custom resource, called OpenDistroCluster, to describe the desired state of the cluster to instantiate.   4.      Topology Each SAS Viya deployment supports a single Open Distro for Elasticsearch cluster. Each ODFE cluster can be composed by one or more ODFE nodes, controlled by a Kubernetes Statefulset: each ODFE node runs in a pod named sas-opendistro-default-0, sas-opendistro-default-1, etc. These pods have a SAS workload class = stateful.   The sas-opendistro OpenDistroCluster is the resource used to define the cluster, the operator then creates the StatefulSet objects and manages them. The operator runs in a single pod with a SAS workload class = stateless, and is implemented using the Operator SDK.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Supported topologies and constraints are documented in the included Readme file available at <sas-bases/overlays/internal-elasticsearch/README.md>: default: a single ODFE node (1 node serving all Elasticsearch roles) production: two ODFE node sets (X nodes with dedicated master role, Y nodes with dedicated data role)   Elasticsearch documentation recommends dedicated master nodes and data nodes for production systems. Availability considerations (see below) suggest 3 master nodes and 3 data nodes. The number of data nodes can be increased to support searching more data.   For the current release of SAS Viya (2020.1.3), the topology (number and role of ODFE nodes) must be chosen at deployment time and cannot be changed after.   5.      Storage Open Distro for Elasticsearch requires persistent storage. By default, each ODFE node (whether with master or data role) creates one PVC with a default size of 128Gi and accessMode RWO, using the default Kubernetes StorageClass.   For production environments it’s recommended to use local, fast block storage (aka disks), not a remote filesystem such as NFS. An example kustomize transformer to use Azure Premium disks is described in the included Readme file available at <sas-bases/examples/configure-elasticsearch/internal/storage/README.md>   Internal testing in SAS labs have shown some approximate sizing: 1M objects: disk space used 380MB 5M objects: disk space used 1.89GB (1M objects is what you will get if you scan approximately 10K tables, 100 columns per table. Each is 1 item in the search index)   6.      Deployment Open Distro for Elasticsearch is seamlessly deployed with SAS Viya 2020.1.3 and later. A sample kustomization.yaml file can be found in the deployment instructions.   When upgrading from a previous release, a few lines specific to the new software should be added to the existing kustomization.yaml as explained in the deployment notes for Version Stable 2020.1.3   In practice, the instructions are the same in both cases: add few new lines to kustomization.yaml. They include one note worth calling out:   Note: The sysctl-transformers.yaml transformer uses a privileged container to set vm.max_map_count. If privileged containers are not allowed in your deployment, do not add this line. Instead, the Kubernetes administrator must set the vm.max_map_count property for stateful objects manually.   That property should be set on all the Kubernetes nodes that can host a SAS workload class = stateful, by logging into the node and setting the property at the OS level as described in these instructions.   7.      High Availability The default deployment with one single Open Distro for Elasticsearch node does not provide high availability.   For the current release, which is only used by SAS Information Catalog, this should not be an issue. In fact, SAS Information Catalog rebuilds the search index from the information catalog stored in PostgreSQL if it goes down. Even in case of disaster recovery, the search index is not backed up nor restored; SAS Information Catalog uses the information catalog service to rebuild the index if it needs to.   It is possible to configure ODFE for High Availability by following the instructions in the README file available at <sas-bases/examples/configure-elasticsearch/internal/topology/README.md> When designing a High Availability environment, consider the following points: Master nodes require 3 ODFE nodes to support a quorum capable of electing the master instance. Data is replicated and can survive failures if you have at least 2 OFDE data nodes, but this minimal topology would run in a diminished capacity in case one node fails. To create a cluster capable of seamless execution even in case of a single failure, deploy at least 3 data nodes. The sample kustomize transformer described above shows the minimal case with 3 masters and 2 data; the recommendation is to configure 3 masters and 3 or more data nodes. If performance is not a concern, master and data roles can be shared on all nodes, and the minimal High Availability topology becomes 3 ODFE nodes (each with both roles) Conclusion Open Distro for Elasticsearch offers new exciting capabilities to SAS Viya. Correct understanding of its architecture will empower SAS administrators to build the best environment to suit end users’ requirements.   Many thanks to Terry Quigley, Iain Jackson, and Nancy Rausch for the great information provided while writing this post.     Find more articles from SAS Global Enablement and Learning here.   ... View more

SAS Viya 3.5 introduced support for clustering the last key component of the architecture that was still a single-point-of-failure, PGPool. The initial design only covered traditional on-premise environments. SAS has recently released detailed sample instructions to implement a highly available environment for each of three major cloud environments: Azure, AWS, and Google Cloud.   Before spending any additional word, let's place front and foremost the actual content: here is the link to the Github project covering the 3 major clouds:   https://github.com/sassoftware/sas-viya-3.5-ha-deployment   As the readme clearly states, and each subproject reinforces, these are sample instructions based on internal testing by SAS QA. They should not be interpreted as "official instructions" or "mandatory architecture".   You may wonder, why only provide examples, and not "the officially supported way"?   Simply stated, cloud environments provide so much freedom and flexibility to your architectural design and implementation, that we neither can nor we want to mandate a unique way to do it. Rather, we want to show that it can be done, and how we did it.   Architecture diagram of a sample HA deployment of SAS Viya 3.5 on Azure. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   What's the big deal? With SAS Viya, many components can be clustered to provide High Availability. SAS provides Pgpool-II open-source software to manage the PostgreSQL clusters that implement the SAS Infrastructure Data Server: Pgpool resides and operates between SAS Infrastructure Data Servers and clients, and also monitors the PostgreSQL cluster to react to failures. Until SAS Viya 3.4, Pgpool itself is a single point of failure. This was a conscious decision because the software release that was initially deployed with SAS Viya did not support any reliable way to establish quorum in case of network issues (split-brain problem). SAS Viya 3.5 removes this limitation, and Pgpool can be clustered.   The approach for BareOS uses a virtual IP that watchdog (a process included with PGPool) dynamically assigns to the current cluster master. Watchdog assigns and removes this IP using normal OS commands ( sudo ip addr add / sudo ip addr delete ), then advertises the change in IP ownership using the ARP protocol ( sudo arping ).   We soon found out that this approach does not work in cloud environments. Google explains the challenges with trying to use floating IPs on the Google Cloud Platform: "the VPC network handles ARP requests based on the configured routing topology, and ignores gratuitous ARP frames. In addition, it's impossible to directly modify the VPC network routing table with standard routing protocols".   Azure and AWS behave similarly, requiring a different solution to support clustering.   That's only part of the story. While PGPool clustering is required, there are other critical infrastructure requirements to consider when designing a “Highly Available” SAS Viya environment – such as a shared file system and an HTTPS Load-balancer (with associated TLS certificates). The choice of an appropriate shared file system in a public cloud environment requires careful considerations, depending on the licensed products and customer needs. Also, the architecture should be designed considering typical cloud tenets such as nodes placement, availability zones, network and security design, and more.   As you can imagine, these considerations span such a wide array of topics and implementation details that required an ad-hoc project to be tackled.   Enter the members of the village: representatives from GEL, DevOps Engineering, Customer Lifecycle Testing, Technical Writing, with direction from Product Management, all worked together to bring clarity to our customers. As one of our managers stated: "with commitment, collaboration and perseverance we were able to get this very valuable documentation into the hands of our customers. "   Links to additional resources To get additional information about PGPool clustering with SAS Viya 3.5, please consult the official documentation available at: https://go.documentation.sas.com/?cdcId=calcdc&cdcVersion=3.5&docsetId=dplyml0phy0lax&docsetTarget=p0844sho9jm6e8n1mwlu0iwocf2k.htm   Kudos I’d especially like to thank Raphael Poumarede, Joseph Cho, Susan Pearsall, Prity Shah for being my co-villagers in this adventure! And the final kick to place the ball in the goal came from Joe Furbee, who helped publish the projects on Github.   Conclusion Providing enterprise-class SAS Viya deployments is a must for our customer's cloud environments. This project demonstrates that it is possible on the major cloud platforms; it provides enhanced availability and scalability, with integration with native cloud technologies.   Find more articles from SAS Global Enablement and Learning here. ... View more

NOTE - Sept 2022: Since the time of publishing, SAS Viya has already evolved. Some of the content of this article does not apply anymore to recent SAS Viya versions. See the newer article Moving to the cloud with SAS/CONNECT to learn about the changes.   "SAS/CONNECT software is a SAS client/server toolset that provides the ability to manage, access, and process data in a distributed and parallel SAS environment. As a client/server application, SAS/CONNECT links a SAS client session to a SAS server session." This is the first sentence in the official SAS/CONNECT for SAS Viya User’s Guide.   What you do not find in this initial introduction is one of the capabilities that makes SAS/CONNECT so versatile: it can work across different releases and across different operating systems, including the new SAS Viya. For this reason, it is often used as a "bridge" when other connection or integration strategies do not work: If you can't connect, SAS/CONNECT! One Says Goodbye, One Gets a New Life With previous SAS releases, starting with SAS 8, SAS Integration Technologies included Integrated Object Model (IOM) servers.  IOM provides distributed object interfaces for conventional SAS features, which, in layman's terms, means that a client can access SAS capabilities provided by a remote SAS server thanks to a shared, proprietary protocol.   Thanks to IOM and to the removal of licensing limitation, since late 2019 it is even possible to directly connect SAS Enterprise Guide 8 to a SAS Viya 3 environment, with no additional server-in-the-middle.   While still available, thanks to the inclusion of a SAS Workspace Server in SAS Viya 3 deployments, IOM is a legacy, proprietary protocol that has been replaced with more modern and open REST interfaces that better fit today's API-first, cloud-native, service-oriented architectures.   For this reason, SAS Workspace Servers are not available anymore with the new SAS Viya, and IOM is gone with them.   This leaves a question: how to connect the new SAS Viya with older SAS environments, that were not designed for REST interfaces? Enter SAS/CONNECT, another SAS protocol that has been available to make different SAS sessions talk, even before the introduction of IOM. Now that IOM is no more, SAS/CONNECT may be the easier way to bridge existing environments to the new SAS Viya.   Despite being around for years, SAS/CONNECT is not just "legacy technology": innovations keep coming, including in the latest version of SAS Viya. Starting with the latest release, SAS/CONNECT is now integrated both with Kubernetes and with many other SAS services available with the new SAS Viya. Thanks to this new integration, SAS/CONNECT servers can now be started in two different ways: a modern, cloud-native one, available to newer clients, and a more traditional one for older clients. Two Ways to Connect When started from clients within the same deployment and same Kubernetes namespace (which also implies the same software release), SAS/CONNECT servers are launched by the SAS Launcher service, just like a Compute server or a Batch server. This is a new capability with SAS Viya, briefly highlighted in a previous post, SAS Programming Run-Time Architecture with the new SAS Viya.   Clients connect to a SAS/CONNECT spawner, and then the spawner offloads the actual launch to the Launcher service. The launcher service starts a new pod using a specific sas-connect Kubernetes PodTemplate, and the SAS/CONNECT server runs inside this new pod. After initialization, the SAS/CONNECT server opens a new TCP port listening for an incoming connection, and sends the IP address and port information to the original client via the launcher. With this information, the client can connect directly to the server and send it some code to execute.   The following diagram shows this first use case - along with some additional services involved in steps we have not detailed above.   A Compute server used by SAS Studio acts as a SAS/CONNECT client. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   A SAS/CONNECT server can still be started in the same way that was available with previous releases, i.e. directly spawned by a SAS/CONNECT spawner, with no usage of the Launcher service. The main difference from previous releases is that here everything is using Kubernetes pods instead of physical or virtual machines.   So, in this second way, when a client submits a SIGNON statement to start a new server, the SAS/CONNECT spawner spawns a SAS/CONNECT server directly. This means that the SAS/CONNECT server is started as a new process running inside the spawner pod, in the same container. The SAS/CONNECT server can inherit the TCP connection from the spawner, and so the client is immediately connected with no additional listening ports.   A client, external from the cluster, submits code to a SAS/CONNECT Server.   In this second use case, the external client can be at any release, including SAS 9, SAS Viya 3.x, the new SAS Viya 2020.1 and later, or even older release, and on any platform. The only requirement is to have SAS/CONNECT licensed and installed on the client, because the new SAS Viya always includes it by default. Also, the SAS/CONNECT endpoint has to be exposed "outside the cluster" using a Kubernetes Nodeport or LoadBalancer, as explained by Rob Collum in K8S Primer for SAS Viya: Networking. Requirements to Launch A few requirements have to be satisfied, for a SAS/CONNECT spawner to use a SAS Viya launcher service to start a SAS/CONNECT server: The client should use a new release of the SAS/CONNECT protocol. This includes the new SAS Viya 2020.1 and later clients, but excludes SAS 9 or SAS Viya 3.x clients. We have seen above that a SAS/CONNECT spawner, in SAS Viya, sends the IP address and port of the newly started SAS/CONNECT server back to the client. Earlier versions of the protocol “do not understand” this information, so the client would not be able to connect to the server. The client should be in the same Kubernetes cluster as the spawner (and the launched server, obviously). In fact, the final connection from the client to the server happens using a TCP/IP address that is only routable inside the cluster. The client submits a SIGNON to sas-connect-spawner, which is a hostname that Kubernetes can resolve, for internal clients, to a pod running the SAS/CONNECT spawner (in Kubernetes lingo, it is a ClusterIP service whose endpoints are the sas-connect-spawner pods) Controlling the Launch There are some new options and variables that can be used to force SAS/CONNECT to only use the new SAS Viya way, or to only use the traditional one. TCPNOIPADDR server environment variable (default: either not set or set to zero) When set to any value other than zero, this forces a SAS/CONNECT server to be always spawned in the same pod as the SAS/CONNECT Spawner (not dynamically launched). It could be useful, for example, when a SAS Viya 2020.1 and later client is running in a different cluster than the SAS/CONNECT spawner, but the spawner thinks it is local. SERVICELAUNCH spawner option (default: set) When specified, enables the new behavior of SAS/CONNECT spawners to launch SAS/CONNECT servers by using the Launcher service, if possible. Ignored for SAS/CONNECT clients that are not able to handle and connect to the IP addresses obtained by the server’s initialization information (i.e. older versions) NOLOCALLAUNCH spawner option (default: not set) When specified, prevents any SAS/CONNECT server from being spawned in the spawner’s pod (i.e. forces the launch by using the Launcher service). As a consequence, it prevents any SAS 9, SAS Viya 3, or remote SAS Viya 2020.1 (and later) clients from connecting. While the first server variable can be set in SAS code by an end-user, the two spawner options can only be configured by an administrator.   For additional instructions about these options and variables, consult the official documentation. Sizing and Scaling Considerations The two use cases – launch or spawn – can require different considerations while architecting and provisioning the environment.   Each launched SAS/CONNECT server instance runs in a new, dedicated pod: from a resource consumption perspective, it is similar to Compute servers. So similar, that launched SAS/CONNECT servers run inside pods assigned to nodes with a label equal to workload.sas.com/class=compute. For this reason, they can have horizontal scalability: the more client connections arrive, the more pods are automatically started and distributed across all compute nodes. In the cloud, for example on Azure, if cluster autoscaling is enabled, additional nodes are automatically added, as required, to the compute pool to support the increased workload.   Spawned SAS/CONNECT servers, on the opposite, always run inside the SAS/CONNECT spawner pod. Multiple sessions all consume resources (CPU, memory, disk) from that same pod. SAS/CONNECT spawners are not included in auto-scaling even when enabling High Availability. It's important to have vertical scalability, or plan in advance: dedicate a node with adequate resources to the SAS/CONNECT spawner pod. To simplify handling of this different use case, SAS/CONNECT spawners are assigned to nodes with a label equal to workload.sas.com/class=connect. Storage Considerations SAS/CONNECT can be used to parallelize code execution by starting multiple server sessions. In Kubernetes, sharing intermediate data between those parallel remote sessions by saving it on disk merits special considerations. Traditional programming paradigms may not work anymore; an example is sharing SASWORK directories between sessions, as was suggested in this old trick: https://support.sas.com/rnd/scalability/tricks/connect.html#workpath.   In both SAS Viya 2020.1 (and later) use cases – whether the server is launched or spawned – there may be issues. For launched SAS/CONNECT servers, each one runs in a dedicated pod, so it does not see any other pod’s local file systems. This is similar to grid processing with SAS 9. For spawned SAS/CONNECT servers, although they all run in the same pod, they execute with LOCKDOWN enabled by default, so each session is only allowed access to a limited list of subdirectories - which does not include other sessions ones. Possible solutions include: follow SAS Grid Manager practices: provision a clustered filesystem, mount it in all SAS/CONNECT containers follow cloud practices: use object storage instead of a traditional file system as a staging/sharing area, follow SAS Viya practices: load all common data into CAS and offload code execution to it In the end, the key point is that this should be carefully evaluated with the customer IT department, including security considerations associated with file sharing. Conclusion SAS/CONNECT can be a valuable mechanism to connect or integrate existing environments with SAS Viya. Both traditional and new capabilities enable SAS clients from different versions and operating systems to manage, access, and process data in SAS Viya.   Special kudos to the colleagues within the R&D division who have answered our numerous requests for help and clarification. ... View more

Throughout the evolution of analytics, something has remained constant, preserving the capability to execute the same programs that you could run some time ago. Over the years and the different releases, SAS has always been able to execute classic SAS program code. How does the new SAS Viya (2020.1 and beyond) run SAS code?   SAS Cloud Analytic Services (CAS) is SAS Viya principal execution engine, providing an optimal run-time environment for data management and analytics thanks to its in-memory, massively parallel processing capabilities.   Yet, when you want to execute some code that requires traditional Foundation SAS, you can still use a traditional SAS engine. It can be just very basic code, such as   proc print data=sashelp.cars; run; This program will run in any environment, from SAS 6 on a mainframe, to SAS Viya 2020.1 in the Azure cloud. Obviously, the underlying mechanics of how the different host environments execute it will change.   What are the components involved with running SAS code in the new SAS Viya (2020.1 and beyond)? All SAS Programming Run-Time Components   This slide by itself may be overwhelming, but it is a guide to the intricacies involved in running SAS code. For this post, it will simply be a visual reference for a light introduction, without the expectation of explaining every single box and arrow.   The key point to get from this diagram is perfectly stated, with a concise sentence, in the official documentation: A programming run-time environment includes several SAS Viya servers.   If you look at the diagram following horizontal lanes, you may recognize 3 main servers: SAS Compute, SAS/CONNECT, and SAS Batch. All 3 can execute SAS code submitted by end-users.   For all 3 components, the flow is represented moving from the left to the right. Clients interact with front-end services (SAS Compute service, SAS/CONNECT spawner, SAS Batch service), which all use an intermediate SAS Launcher service to start the back-end servers, thanks to SAS Launcher integration with Kubernetes API.   Other SAS Viya services provide additional capabilities, ranging from authentication (SAS Logon service) to context definitions storage and retrieval (SAS Infrastructure Data Server), but describing them all would require a level of detail that would push this post to be unbearable by many. Three different implementations The Compute service provides REST API endpoints to clients like SAS Studio, SAS Model Studio, and others to submit and execute SAS code, obtain log and list output, view results, and more. The SAS Compute server executes the code. SAS/CONNECT software provides the essential tools for sharing data and processing power across multiple computing environments, even across different operating systems and SAS releases. The sas-batch command-line interface uses the Batch service to submit SAS programs to the cluster for batch processing. The Batch service launches the Batch server to process the SAS programs. All three use the Launcher service, a microservice that provides API endpoints for running a process. A new Kubernetes pod is created each time a service makes a request to the Launcher service to start a new process. With the current release, the pods are launched through a Kubernetes job. Evolution from earlier releases A brief note to help you transition, if you are familiar with previous SAS releases.   The SAS Compute server has a similar architecture in SAS Viya 3; a notable difference is the removal of the SAS Launcher server, which is a process used in SAS Viya 3 by the Launcher service to start a back-end Compute server process.   As you can see in the above diagram, the Launcher service, in the new SAS Viya (2020.1 and beyond), talks to Kubernetes, which is responsible for launching the back-end servers in a new Kubernetes pod.   On the other side, SAS Viya 3 did not have a batch client or server, yet these components are not totally new: they are an evolution of the SAS grid client utility (a.k.a sasgsub) and SAS batch sessions available with SAS Grid Manager in SAS 9. Although the current SAS Viya release (2020.1) still does not include all the workload capabilities available with SAS Grid Manager, Kubernetes provides the required infrastructure to support submitting batch jobs and distributing them on multiple back-end servers (executing in Kubernetes pods). The Common Base All three SAS Viya servers - SAS Compute Server, SAS Connect Server, SAS Batch Server - are implemented using the same common base image, called sas-programming-environment.   To execute SAS code, the common image contains all SAS Foundation binaries, but no license - this prevents any fraudulent distribution of SAS. The license is applied at run-time from Kubernetes and controls what PROCs can be executed, based on the solution that has been licensed.   The following picture represents the Kubernetes artifacts involved with the execution of the back-end servers.     Again, horizontal lanes represent the three different servers - Compute, SAS/CONNECT, Batch. On the left, you can see different Kubernetes pod-templates: sas-compute-job-config, sas-connect-pod-template, sas-batch-pod-template. They contain information for Kubernetes about how to instantiate a new pod; they all reference a common base image, called sas-programming-environment, plus specific information to describe how to instantiate various kinds of pods, which correspond to specific servers.   Kubernetes uses that information to create specific containers, called either sas-programming-environment, just as the image, or sas-batch.   The containers, finally, run inside the pods that represent an instance of a Compute server, SAS/CONNECT server, or Batch server. Conclusion This overview clearly only scratches the surface of the SAS Programming Run-Time architecture in the new SAS Viya 2020.1. For additional information, please refer to the official documentation. Even better, you could deploy Viya, and have a look at it yourself!   Note: This article addresses features of SAS Viya 2020.1 (and later) ... View more

In my previous post about Azure storage, you can find details on Azure disk storage options, i.e. storage dedicated to a single host. In this new installment, the focus is storage shared between multiple virtual machines. What options does Azure provide to SAS Architects? Evolving technologies In the Cloud world, the industry is moving toward object storage mechanisms, such as Azure Blob storage, which is the foundation for Azure Data Lake Storage Gen2 (ADLS2). Object storage is optimized for storing massive amounts of unstructured data, such as logs, images, video, audio.   SAS is continuously implementing new features to provide robust integration with data stored in these new providers. SAS Cloud Analytic Services (CAS) can access CSV and ORC files stored in ADLS2.   Even more, when direct access is not available, you can leverage third party projects, for example Azure Blobfuse.   This does not mean that traditional file storage does not exist in the cloud; file storage is here and will stay as a long-term requirement.   SAS, as well, has not abandoned file storage; on the contrary, it is still one of the most widely used methods to read, write and share data, probably only rivaled by access to relational databases.   To read more about object storage see Cloud Object Storage and SAS by Stephen Foerster. The rest of this post, instead, focuses on the options available on Azure to provide shared file systems so that multiple hosts can access common data, including SAS 9, SAS Grid Manager, and SAS Viya. Choices, choices, choices When moving from traditional on-prem environments to the cloud, you can be overwhelmed by the amount of options for the simple task of sharing a file system between multiple hosts. As a SAS Architect, you will very often need such shared file system in the infrastructure and you should be able to articulate the different requirements for common usages in many SAS 9.x or SAS Viya scenarios: to host deployment artifacts, to support High Availability, as a requisite for backup tools, to satisfy the I/O requirements of SAS Grid Manager shared storage, as an RWX Persistent Volume for SAS Viya 4 in Kubernetes, etc...   Here are some of the most common storage solutions used by SAS architects. Managed disks (Yes, the ones described in my previous post).   A first, simple option is to do just as on-premise: attach virtual disks to one VM, then use NFS (Linux) or CIFS (Windows) to export that storage to other machines in the environment.   With this, you are using Azure just to provide the infrastructure, keeping all the traditional software configuration and maintenance for yourself. It is really not a managed solution.   You still have some advantages compared to traditional, on-prem solutions, when you consider that the file server machine and the exported disks are often single points of failure of your environment. On the Azure cloud, a standalone virtual machine can have a guaranteed availability above 99.9%, and managed disks are provisioned from redundant storage. According to Azure documentation: “Managed disks are designed for 99.999% availability. Managed disks achieve this by providing you with three replicas of your data, allowing for high durability.”. A downside of this approach is that you need one dedicated machine to be the file server, and you have to manage it yourself.   In simple environments (dev, test), you may simply delegate this role to one of the SAS hosts. Managed Azure Files This is the “default” way of sharing file storage on Azure. Azure Files offers fully managed file shares in the cloud. Although Azure documentation lists multiple benefits, I think this is the most important: “Azure file shares can be created without the need to manage hardware or an OS. This means you don't have to deal with patching the server OS with critical security upgrades or replacing faulty hard disks.”. It’s a step forward in being cloud-native, compared to the previous option.   Azure file shares use the Server Message Block (SMB) protocol, i.e. they behave as shares created from a Windows server. Although originally not a Linux native protocol, Linux hosts can use Azure file shares by mounting them with the CIFS kernel client.   In using this with SAS, you can encounter some limitations of the CIFS protocol: the most notable is that once you chose a user/group as the owner of the mounted share and a permissions mode, these properties are fixed and you cannot change them without unmounting/remounting.   As an example, suppose you create a share called utils in a storage account named mysasstorage123 (The name must be unique across all existing storage account names in Azure.) After entering the correct credentials in /etc/smb_credentials, you can mount it using this line in /etc/fstab:   //mysasstorage123.file.core.windows.net/utils /mnt/utils cifs rw,vers=3.0,credentials=/etc/smb_credentials,uid=AzureUser,gid=users,file_mode=0775,dir_mode=0775,serverino 0 0   This creates a share owned by the AzureUser, with the file_mode and dir_mode of your choice:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   If you create any subdirectory or file there, it will all be owned by the same user and have the same permissions.   This can be a good fit to share data in some specific use cases: installation artifacts as in the example above, including the SAS 9 depot, SAS Viya mirror, Ansible playbooks, etc. shared vault for SAS Viya backups or central vault for SAS 9 backups shared file system for CAS, required if your deployment includes a secondary CAS controller On the other side, it is not a good fit for home directories, because each user would see everybody else’s content – including secrets such as ssh keys – unless you mount a dedicated share per user.   Another huge limitation, intrinsic to the CIFS protocol, is that it requires communication on port 445; while this is usually not a problem between hosts running in your datacenter, or in the same vnet in the cloud, that port is usually closed by administrators on firewalls and between different networks. This means that, probably, you will not be able to mount these shares on any server outside Azure, including on-prem. Managed Azure NetApp Files (ANF) Although the name sounds similar to the previous option, this is a totally different kind of shared storage. You have to request onboarding to Azure NetApp Files, following the Register for Azure NetApp files instructions. After your subscription has been authorized to use the service and you have registered the Azure Resource Provider for Azure NetApp Files, you can start creating storage artifacts: storage accounts, storage pools, volumes. Finally, you can mount those volumes to multiple hosts using native NFS for Linux or SMB for Windows.   Since ANF uses native protocols, it supports multiple users and permissions on subdirectories. Although more expensive than regular Azure file shares, it is also more performant. You can provision a minimum of 4 Terabytes, up to a few Petabytes.   Just as with Azure Files, a huge benefit is that ANF is fully managed: Azure handles hardware maintenance, updates, and critical issues for you.   It also comes with some specific considerations: It requires specific mount options to avoid performance degradation during metadata updates and cache coherency issues. At the time of writing, there are no workarounds to the NFS limit to the number of groups (16) that a user can belong to. If you use Active Directory – including its Azure variants – as your identity source, Linux will probably only see the first 16 groups users belong to and nothing else. These could even be “random” email groups instead of the desired business units you based your security design upon. Currently, it cannot scale to more than 1 storage volume per environment. This puts a hard cap to the total bandwidth that the storage can provide (about 5000 MiB/s read, 2000 MiB/s write). There are workarounds, but they require manual, ad-hoc work from NetApp engineers, or expensive vnet peering (see the next point). Each volume has to reside in a specific subnet assigned to the same vnet of the compute tier. To mount that volume in another vnet, you have to configure vnet peering, which means Microsoft will charge you a few cents for every gigabyte that crosses the boundary. That can easily sum up to millions of dollars per year. Performance considerations Azure file shares are provisioned from a storage account; each Azure subscription can have multiple storage accounts, and different types of storage accounts. The two types of storage accounts that can provide Azure file shares to SAS environments are: General purpose version 2 (GPv2) storage accounts: these are backed by standard, hard disk-based hardware. In addition to storing Azure file shares, GPv2 storage accounts can store other storage resources such as blob containers, queues, or tables. As an example, the SAS Viya Quickstart Template for Azure recommends using a blob container to host the Viya license and Mirror. This storage can be cheap (no fixed minimum price, only pay per usage), but you can quickly hit performance limits. Utilization from other storage services affects the Azure file shares allocated in the same storage account. For example, if you reach the maximum storage account capacity with Azure Blob storage, you will not be able to create new files on your Azure file share. FileStorage storage accounts: these allow you to deploy Azure file shares on premium, solid-state disk-based hardware. FileStorage accounts can only be used to store Azure file shares; no other storage resources (blob containers, etc.) can be deployed in a FileStorage account. FileStorage provides higher throughput than GPv2, but can be more expensive (you pay per provisioned space/throughput, even when not using it, with a minimum of 100 GiB) Standard Storage Accounts have a fixed maximum bandwidth of 60 MiB/s, that can be upgraded to 300 MiB/s.   Premium Storage Accounts bandwidth is proportional to the allocated storage size; the maximum possible allocation is 100 TiB which delivers 6200 MiB/s (read) and 4130 MiB/s (write), but the limits per single file are much lower: 300 MiB/s (read), 200 MiB/s (write).   Due to these low I/O throughput limits, Azure file shares are not recommended to process SAS datasets across multiple machines.   Azure NetApp Files comes in three service levels: Standard, Premium, Ultra. Ultra is usually the best choice for SAS, both in terms of cost and performance. The throughput limit for a volume is determined by a combination of the quota assigned to the volume and the service level selected; the maximum empirical throughput that has been observed in testing is about 4500 MiB/s (read) and 2000 MiB/s (write).   SAS Grid Manager can run on Azure using Azure NetApp Files as a shared storage for small environments, given proper OS and infrastructure sizing and tuning. The storage can scale to accommodate up to 24 physical cores for the compute nodes (3 nodes with 8 cores each, or 6 with 4 cores each).   As a final consideration, it is important to understand an intrinsic performance limit when using any sharing technology: they all transfer data through the Azure network and are thus subject to virtual machines networking limits. For this reason, SAS requires using accelerated networking. The network bandwidth allocated to each virtual machine is capped on egress (outbound) traffic from the virtual machine, while ingress is not metered or limited directly. However, there are other factors, such as CPU and storage limits, which can impact a virtual machine’s ability to process incoming data. In practice, this means that Azure virtual machines enforce a hard limit to the maximum write throughput towards any shared storage, while read bandwidth is virtually unlimited, up to the maximum that the storage can provide. As an example, E32s_v3 machines have a write limit at 2000 MiB/s.   The following image shows how the performance limits discussed so far can affect the storage performance of a SAS Grid Manager environment:     Bottom line: the most "throttled" pipe or constraint at ANY point in the data path will be the best throughput your workload can achieve! Additional Considerations The options presented in this post are the most common, but not the only possible ones. There can be specific use cases where other shared disk technologies can be used: DDN Whamcloud Cloud Edition for Lustre This is the cloud community version of Lustre, a clustered filesystem that has proved to be a good fit for SAS, including SAS Grid Manager. The biggest consideration for this offer is that it is not managed, i.e. you are responsible for the installation, configuration, management, and maintenance just as on-prem. Not everyone has the skills to do it. Azure file shares using native NFS instead of SMB This can remove some of the Azure files limitations, such as user ownership and permissions, but is still in preview in limited regions, and does not improve the available performance. For this reason, it is still not recommended to share SAS datasets. IBM Spectrum Scale is, traditionally, a good fit to share SAS data, including for SAS Grid Manager. Unfortunately, it is still not available on Azure. The linked GitHub page only provisions the infrastructure and not the actual file system; it also reads: “All templates / modules / resources in this repo are released for use "AS IS" without any warranties of any kind, including, but not limited to their installation, use, or performance.” As already stated in the previous post, Azure shared disks allow you to attach a managed disk to multiple virtual machines simultaneously, but are not suitable for SAS. They require specific operating system support and specialized applications to provide the sharing capability. It may be possible to use them through RedHat GFS2, but this has not been tested nor vetted yet. Conclusion. This post closes with the same consideration as the previous one: there are many resource and configuration choices available within Azure. In order to select the proper shared storage to meet the needs of your SAS environment, you may have to overprovision the storage capacity to provision enough I/O throughput required by SAS.   The Cloud is an ever-evolving environment; as you are reading this post, cloud vendors have already added new capabilities to the technologies presented here, and SAS engineers are further testing them to always highlight the best fit.   Stay tuned to read the results of these performance tests with a special focus on Azure NetApp Files and SAS Grid Manager. ... View more