SAS Viya 3.5 introduces an entirely new approach to configuring network communication. There are several new configuration directives to become familiar with. One part to achieving that is understanding how the various terms all relate to one another. In particular, the configuration directives that identify which network bindings the Viya software should use have an interesting chain of dependencies and assumptions.   This post will dive into the details specific to the BIND parameters, explaining how they relate one to another, and offering suggestions on configuration for different deployment scenarios. Where The network configurations for SAS Viya 3.5 are presented in two places: On the Ansible controller in: .../sas_viya_playbook/host_vars/<deploytarget>.yml This file is processed at install time when the site.yml play is run. Specifying these values is optional for single NIC hosts and required for multi-NIC hosts. Each deploy target defined in the Ansible inventory file of the sas_viya_playbook should have its own YAML file here, if needed.   On the Ansible Controller host, you manually create the network configuration files for each deploy target in the host_vars subdirectory. Here's one of mine from a recent installation for a deploy target with hostname alias "sasviya01":   $ cat /path/to/sas_viya_playbook/host_vars/sasviya01.yml --- network_conf: SAS_HOSTNAME: sasviya01.race.sas.com SAS_BIND_ADDR_IF: "eth0"   On each Viya host in: /opt/sas/viya/config/network.conf This file is created on each host of the Viya deployment with the values provided by associated the host_vars file. And this file is read each time Viya services start up on this host.   Shown here is the file created for my "sasviya01" deploy target on its host machine:   $ cat /opt/sas/viya/config/network.conf # BEGIN Ansible managed network configuration options export SAS_HOSTNAME="sasviya01.race.sas.com" export SAS_BIND_ADDR_IF="eth0" Notice that the files in the host_vars subdirectory are provided in YAML format, but that the resulting network.conf files on each Viya host are in shell syntax. What One of the key objectives of revamping how network configuration works in Viya 3.5 is to provide a set of reference variables that all of the Viya software services can rely on identically.   Of those, the first two you should know about are: SAS_HOSTNAME The hostname used to advertise how Viya services on this machine should be identified and contacted. It should resolve in DNS (or elsewhere) to the same IP address as SAS_BIND_ADDR. If not specified, default value is set equivalent to calling "hostname --fqdn". SAS_BIND_ADDR The IP address used for identifying and contacting Viya services on this machine. If not specified, default value is 0.0.0.0 (meaning listen on all network interfaces). Notice that both of these have self-determining default values. So specifying these configuration directives is optional for host machines with a single network interface. If your machine has more than one network interface (a.k.a. multiple NICs or multi-NIC), then the SAS Configuration Server requires you to identify which network interface it should bind to for communication (it will not allow 0.0.0.0 as an acceptable value).   Making the choice and specifying a particular network interface has consequences to consider. The values of those two configuration parameters are referred to by almost all Viya 3.5 services (or they should be) through internal SAS function calls. This means that your choice of SAS_BIND_ADDR will affect not just SAS Configuration Server, but the rest of the Viya software, too - everything will listen on that one specified network interface, unless…   …you also set these two configuration parameters as well: SAS_EXTERNAL_HOSTNAME The hostname used to advertise how public Viya services on this machine should be identified and contacted. It should resolve in DNS (or elsewhere) to the same IP address as SAS_EXTERNAL_BIND_ADDR. If not specified, default value is set to SAS_HOSTNAME. SAS_EXTERNAL_BIND_ADDR The IP address used for identifying and contacting public Viya services on this machine. If not specified, default value is set to SAS_BIND_ADDR. Be careful with your interpretation of "external" here. It doesn't mean "alternative" or "outside". It's only used by a small, select set of Viya services which present public-facing interfaces. Think of this in comparison to the majority of Viya services communication which is private - not directly shared with any end-user client. For example, the CAS Controller provides a public interface which listens for end-user clients on port 5570 (by default) and is bound to the network specified by SAS_EXTERNAL_BIND_ADDR, but it communicates with the CAS workers extensively through a different internal set of interfaces which are configured by SAS_BIND_ADDR instead.   There's a secondary tier of configuration values which can be specified. These directives are used to deterministically derive and set the value for SAS_BIND_ADDR based on dynamic criteria of your network infrastructure: SAS_BIND_ADDR_IF The name of the network interface that Viya services should bind to. If provided, SAS_BIND_ADDR will be set to the first primary IP address found for that network. If SAS_BIND_ADDR is also defined with a value, then this directive is ignored. SAS_BIND_ADDR_CIDR The range of IP addresses to choose from in CIDR notation. If provided, SAS_BIND_ADDR will be set to the first primary IP address found in that range. If SAS_BIND_ADDR or SAS_BIND_ADDR_IF is also defined with a value, then this directive is ignored. Notice in terms of precedence that specifying a value for SAS_BIND_ADDR overrides SAS_BIND_ADDR_IF and that SAS_BIND_ADDR_IF will override SAS_BIND_ADDR_CIDR due to specificity. My recommendation is only to set the directive you intend to use to avoid any confusion.   These same secondary parameters are also available for _EXTERNAL_ as well. How To show this all as a simple set of rules, take a look at the following pseudocode to see how Viya services will determine which hostname and IP to use in their communications and registration:   For Viya to determine the SAS_HOSTNAME value: if SAS_HOSTNAME is set in network.conf, then use it else use `hostname --fqdn` For Viya to determine the SAS_EXTERNAL_HOSTNAME value: if SAS_EXTERNAL_HOSTNAME is set in network.conf, then use it else use SAS_HOSTNAME value For Viya to determine the SAS_BIND_ADDR value: if SAS_BIND_ADDR is set in network.conf, then use it else if SAS_BIND_ADDR_IF is set in network.conf, then use it else if SAS_BIND_ADDR_CIDR is set in network.conf, then use it else use 0.0.0.0 For Viya to determine the SAS_EXTERNAL_BIND_ADDR value: if SAS_EXTERNAL_BIND_ADDR is set in network.conf, then use it else if SAS_EXTERNAL_ BIND_ADDR_IF is set in network.conf, then use it else if SAS_EXTERNAL_BIND_ADDR_CIDR is set in network.conf, then use it else use SAS_BIND_ADDR value Why There really is no one right way to configure the network that will work for all circumstances. You must evaluate the requirements of a given environment and adjust. With that out of the way, I usually start off with the following and then modify my approach as needed: Avoid setting an explicit value for SAS_BIND_ADDR Defining SAS_BIND_ADDR directly will lock Viya down to static IP address assignment. While that's consistent with the current system requirements, we're optimistic that the new network.conf capabilities will soon allow support of dynamic IP address assignment. Use SAS_BIND_ADDR_IF instead This directive will cause Viya to look up the specified network interface name and automatically assign SAS_BIND_ADDR to the first primary IP address it finds there. Your host can still have static IP address assignment. But if/when Viya supports dynamic IP address assignment, then you've no changes to make. Consider SAS_EXTERNAL_BIND_ADDR (and _IF and _CIDR) If SAS_BIND_ADDR resolves to a subnet that the Viya end users do not have access to, then ensure that the public interfaces (like CAS port 5570) are reachable by specifying a network interface on this host which is accessible to end users. Pro-tip: consider SAS_EXTERNAL_BIND_ADDR=0.0.0.0 to direct the Viya public services to listen on all of this host's network interfaces if that makes sense for your environment. I could go on and on. As you can tell, I'm excited by the new possibilities afforded to this huge upgrade to network identification in Viya. I hope this post helps guide your understanding how the network.conf file is processed when Viya services are installed and started. More Refer to the the SAS documentation for more information about networking configuration in Viya 3.5:  SAS Viya 3.5 for Linux: Deployment Guide > (Optional) Configure Network Settings ... View more

SAS Workload Orchestrator is the new grid provider that ships with SAS Grid Manager 9.4 M6. It presents HTTP RESTful interfaces for ongoing operations and administration. Out of the box, the configuration of this HTTP traffic is somewhat rudimentary, with only plain-text communication (i.e. no encryption) and basic redirection provided from the master candidate hosts to the active master (i.e. limited automatic client discovery of SWO master failover).   Out of the box When SAS Workload Orchestrator is installed and configured by the SAS Deployment Wizard, the minimum failover capability for the discovery of the current SWO master is enabled. Default, out-of-the-box SAS Workload Orchestrator Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page. In a SAS Grid Manager deployment, there are basically two kinds of grid server hosts: Grid master candidates Grid worker nodes Grid worker nodes only perform the work they're assigned. A grid master directs all work on the grid - and it can also perform work as well, if desired. If there's only one host capable of acting as the grid master, then that's a single point of failure - which is antithetical to what we want from a grid. So for SAS Grid Manager, we can identify multiple hosts which have the ability to take on the role of the grid master - and they're called master candidates. The master candidates always know which host among them is currently acting as the grid master. If an HTTP request for the current grid master is inadvertently sent to a grid master candidate, then the candidate will respond with a 300-level HTTP code which re-directs the client to make the request of the current master. This is a nifty trick, but it's not really enough to ensure high availability. The client must know which host(s) are master candidates. If there's a hardware failure, then the SWO master candidate on that host cannot reply with an HTTP 300-level code (or anything else) because it's offline.   Provide automatic client discovery of SWO master failover Instead of requiring the grid client to keep track of all SWO master candidate hosts and then possibly attempt to contact them each one-by-one in a failure situation until it finds the current master, we can configure the SAS Web Server to act as a proxy. We will configure it to keep track of everything automatically, and then just use it as a single point of contact.   SAS Workload Orchestrator with automatic client discovery   For details on this, see my post: New Considerations for SAS Grid Manager 9.4 M6   Provide TLS encryption of SWO communication over HTTP   Have you noticed that so far all communication with the SWO has been shown as plain, unencrypted HTTP? That's not what our customer IT administration teams want to see for the ongoing operations of their critical grid infrastructure. They prefer that communication to be fully encrypted with TLS.   Encrypting SAS Workload Orchestrator communications   My GEL team comrade Ajmal Farzam explains how to enable TLS encryption for HTTPS communication of SWO operations in his blog post, SSL Encryption for SAS Grid Manager. Notice that automatic client discovery of the current SWO master is still enabled as well.   Fully realize failover and encryption   SAS Grid Manager environments often represent a significant investment in time, effort, and money by our customers. Ensuring the availability of that investment is a high priority and so all single points of failure must be addressed. Having just one SAS Web Server to act as a proxy between the grid clients and the SWO master candidates isn't sufficient - we need two of them plus another third-party load-balancer in front of them. And of course, hardening HTTP communication with TLS encryption is absolutely necessary.   Highly available and fully encrypted operation of SAS Workload Orchestrator   SAS Technical Consultant Wade Adkins authored a post to explain how to accomplish this: High Availability and TLS with SAS Workload Orchestrator.   An exercise for the reader   We use the SAS Web Server to act as a proxy in front of the SWO master candidates for one reason: because it's there. But we don't have to. If your customer site enforces a strict separation of web tier from the compute tier, then they might prefer a different approach.   Bypassing the SAS Web Server     The service labeled as "VIP Load Balancer" in the illustrations above is an oblique reference to some piece of kit that your customer must provide, configure, and maintain. It might be hardware. Or software. Or some combination. In any case, it should be implemented as a highly available service, too. Given the examples above explaining how to configure the SAS Web Server (effectively the same as Apache HTTP Server), then your customer's IT team should have sufficient information to proceed.   Coda   The SAS Workload Orchestrator is delivered with a basic configuration out of the box. Sites running production environments SAS Grid Manager 9.4 M6 will usually require configuration to improve both high availability and encrypted communication. In addition to the SAS documentation, refer to the blog posts shown here to address the customer's requirements for availability and encryption of SWO operations. ... View more

Your customer has given requirements for your project which include disaster recovery (DR). Now, followers of this article might recall that disaster recovery is not a feature of SAS software. But you know that many SAS software components do offer options to improve availability through clustering. And perhaps your customer's IT organization is large enough to have data centers in multiple geographic regions. Or maybe they're running SAS in the cloud using AWS or Azure or GCS, all of which offer multiple availability zones (AZ) to protect cloud resources from the loss of a physical data center.   A quick back-of-the-napkin sketch showing SAS software clustered with nodes deployed into different AZ is often looked at as a simple solution to provide DR automatically and on the cheap:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   But this seemingly simple solution is fraught with several challenges. Overcoming these challenges is necessary to deliver the solution the customer expects to have. Not doing so will lead to performance problems and even worse, that the solution won't provide the availability and disaster recovery that was intended all along.   A quick note: In cloud parlance, "availability zone" has a specific meaning (1, 2, 3), usually where two or more AZ will reside in nearby geographic proximity within a "region". A widespread disaster that affects an entire region then might take out multiple AZ. And therefore, some companies will cluster across regions as well. Of course, regions can neighbor each other or be on opposite sides of the planet. To be clear, this post uses "AZ" to simply mean "different physical locations, near or far". Take note that topics discussed here will have varying considerations specific to physical distance. Challenge No. 1 - Latency, not just throughput From a computer networking perspective, latency is the amount of time between sending a request and receiving a response. Assuming no processing, just a simple bounce back, then this time is driven mostly by physical distance between the two communication nodes. We often use the ping utility to determine that amount of time. The further the distance, the longer the ping times meaning the greater the latency.   ping localhost # within the same host 64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.032 ms 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.093 ms   ping sas.com # within 1 mile (1.6 km) 64 bytes from 10.15.1.19: icmp_seq=0 ttl=252 time=2.753 ms 64 bytes from 10.15.1.19: icmp_seq=1 ttl=252 time=2.710 ms   ping weibo.cn # over 7,000 miles away (11,000 km) 64 bytes from 47.88.66.107: icmp_seq=1 ttl=90 time=153.716 ms 64 bytes from 47.88.66.107: icmp_seq=2 ttl=90 time=143.928 ms   So the challenge here is splitting up SAS clustered services into different availability zones. The farther the distance between the AZ, then the longer the ping times will be, sometimes substantially, orders of magnitude larger than when services co-exist in the same physical location. To see what this means for Amazon Web Services, visit cloudping.co for a real-time assessment of latencies between the various AZ.   Latency adds up fast, too. Web pages often make dozens or even hundreds of individual requests for files that make up a single web page… especially web apps like SAS Studio and SAS Visual Analytics. As those requests stack up, so does the impact of longer latency - and this means the user may perceive SAS performance as sluggish. If latency is too high, then some requests will timeout, meaning those resources aren't returned properly, leading to a broken user experience.   The same holds true for SAS Cloud Analytic Services and its component controllers and workers. They communicate frequently and sometimes even shuffle data between them. Higher latency between nodes means that the high-performance analytics engine won't be able to complete coordinated tasks as quickly or efficiently.   Challenge No. 2 - Placement of services Improving availability means eliminating single points of failure. This means that we often cluster our software so that redundant components run on separate hosts. In this way, if a single service instance, disk, power supply, or something else on the host fails, then the overall clustered service can continue operations until the problem is resolved.   The same concept can be extended to data centers, too. If a natural disaster strikes, then the entire data center might go dark. And that’s why cloud providers offer AZ so that it's easy to eliminate their physical infrastructure as a point of failure as well. So eliminating the data center as a single point of failure makes sense, but how many AZ do we need? Two? Three? More?   The answer is it depends on the software being clustered - and how that clustering actually works. Let's look a two quick examples. Cluster Type: Redundant Service Nodes For SAS Viya, consider the SAS Configuration Server (a.k.a. Hashicorp Consul). Each instance of Consul maintains its own copy of data. They can act independently to satisfy requests, but coordinate closely to ensure their data is kept in sync.   To maintain quorum, Consul requires clustering on three or more odd number of nodes. That way, as long as more than half the nodes stay in communication with each other, then Consul will continue in service. If we split those three nodes across two AZ, have we eliminated the data center as a single point of failure?   No. Not really.   With only two AZ, that means two out of three Consul nodes are in the same AZ. If that one AZ goes down for some reason, then Consul loses more than half of its nodes, which means it no longer has quorum. Even though one Consul node is still active, Consul will cease service at that point. To be consistent with Consul's expected operations in support of quorum, then it's necessary to deploy each Consul node to a different AZ.     Make similar considerations for SAS Infrastructure Data Server (Postgres and pgPool), SAS Message Broker (RabbitMQ), SAS Secrets Manager (Hashicorp Vault), SAS Cache Locator and Server (Apache Geode), as well as the various SAS server and microservices components. And keep in mind that each technology employs its own approach for ensuring a single version of the truth - not the same quorum technique as Consul. Cluster Type: Collaborative Service Nodes With the SAS Cloud Analytic Services (CAS), we have a different type of cluster technology. CAS is built using a shared-nothing MPP architecture. CAS employs one (or more) controllers to coordinate workload and a number of workers. The workers each are given a subset of the larger data problem to solve. They work independently on their own assignments, then send the results back to the controller.   If we split MPP CAS across two AZ, have we eliminated the data center as a single point of failure?   Maybe. It's possible.   CAS offers two controllers: a primary (required) and a secondary (optional). The secondary controller does not offer full functionality of the primary - and is only intended as a temporary backup for failover if the primary is suddenly lost. So if you place the primary controller in one AZ and the secondary controller on a different AZ, then split the workers evenly across both AZ, then the CAS service should be resilient if one data center does go dark.     But, keeping the CAS service running is only one problem to solve. More important is the data which CAS maintains in memory. Without data, CAS can't do much. If too many CAS workers go offline at once, then it's likely that the data CAS was keeping in memory distributed across all workers will be incomplete, effectively corrupt, and therefore unusable. So we need to consider what CAS requires to keep in-memory data highly available. Challenge No. 3 - Data availability For many, but not all, data sources and formats, CAS defaults to keeping one additional copy of data available to its workers (see COPIES= option for the PROC CASUTIL SAVE statement). With COPIES=1, then CAS can maintain data integrity if one CAS worker goes out of service.     Now what if a second CAS worker goes down? Without a second copy of the data to reference on the remaining node, then the data table is no longer complete. So if you want CAS to be resilient and maintain data integrity after the loss of multiple CAS workers, then increase the COPIES= value accordingly.   Consider this for an MPP CAS deployment with 10 workers split across two AZ - five workers in each AZ. If one data center goes offline, can CAS maintain data integrity with the remaining five workers?   Yes, but only if COPIES=5 was in effect when the data was loaded.   That's a big if. And it's a big ask. That's a lot of redundant data to copy around. And it requires considerable space for the CAS_DISK_CACHE on each of the workers. That's a large commitment to resources which must be maintained for high availability of data to protect against an event which is unlikely to occur.   An alternative to consider might be to establish a manual recovery process. For example, run two different MPP CAS servers, each in their own AZ with their respective workers together. The idea is that if one AZ goes down, then the CAS server in the other AZ is still available for service.   But what about the data? For this scenario, establish a process that all data for CAS use will first be staged in appropriate infrastructure that provides rapid parallel loading (e.g. SASHDAT format in DNFS). And of course, ensure it also has high availability across AZ.     This way, both instances of CAS will have speedy access to the same data source. If one CAS goes down, then the other could be used to run the same workload. While this requires more manual effort and some coordination, it's likely a simpler and more efficient use of resources over time. Indeed, the process could be setup so that a load action for SASHDAT data in the staging area is always run on both CAS servers every time. Due to how CAS loads SASHDAT from DNFS, the redundant copy of the data on the other CAS server may consume no in-memory resources if it's not actually used. See "Lazy Loading of SASHDAT" in my previous article, 4 Rules to Understand CAS Management of In-Memory Data. Challenge No. 4 - Synchronizing content As described above, clustering across AZ presents some significant challenges. If those can't be overcome, but DR is still desired, then a different approach might be employed. Typically this means deploying the full stack of SAS Viya software twice - each in a different AZ. Active-Active As you can imagine, many customers would like to see both such deployments of Viya online and processing their day-to-day workloads. They reason that if one Viya deployment goes down, then all of the workload could go to the other. This is most difficult to achieve, however, in terms of keeping the data and user-generated content of two different Viya environments perfectly in synchronization. Currently, we are not aware of a good way to do that for separate databases (like Consul, CAS, or even Postgres) which are operating in different physical deployments… unless they're set up as a single service cluster. But then you have to go back to the top of this post and re-read about Challenges 1-3 again. 😉 Active-Passive For active-passive, the idea is that the active Viya deployment handles the normal day-to-day operations while the passive deployment sits idle (or even switched off) until it's needed. In a disaster strikes the active deployment, then the passive one is spun up to take over - usually requiring some minutes or hours to come fully online. For Viya deployed using cloud providers, active-passive should be the default approach because the nature of the cloud makes it possible to easily bring up resources - like a passive environment - on demand when they're needed, costing nearly nothing in the meantime.   The challenge here is keeping the passive deployment up-to-date with data and user-generated content from the active environment. And with an environment like SAS Viya, this is usually best achieved with some form of regularly scheduled process which might involve a number of approaches, including elastic storage options, shared file systems, backup+restore and/or promote as well as manual steps to switch on the passive environment at the time it's needed. Coda Planning for disaster recovery is necessarily a collaborative process with your customer's IT team. When you see clustering across data centers or AZ proposed, then be wary about attempting that as a shortcut to achieve DR. While that might be tempting, educate your customer's IT team about the significant challenges that must be properly addressed to truly ensure performance is adequate and also that availability and DR are actually accomplished. Also remember that manual and/or site-specific actions might be needed (meaning design, build, testing, and evaluation) to ensure the customer can achieve their goals.   SAS publishes our SAS 9.4 Disaster Recovery Policy and our SAS Disaster Recovery Policy for SAS Viya 3.4 online for easy reference. ... View more

The SAS Cloud Analytic Services (CAS) offer lightning fast analytics of in-memory data. However, relying exclusively on RAM for all data management and operations is often a short road with an abrupt end. CAS extends its capabilities by also taking advantage of the persistent storage options available. In particular, CAS can be defined to use one or more local disks as a caching space to help improve fault tolerance, data availability and memory utilization. That disk space is referred to as CAS_DISK_CACHE. Analytics In-Memory CAS operates as a high-performance in-memory analytics engine.   In-memory is emphasized as that mode of operation provides the substantial performance value which is CAS’ trademark. Since CAS is designed for performing operations in memory, then running out of memory, like jail, is a place we don't want to be. But CAS offers us a Get Out of Jail Free card with its CAS_DISK_CACHE.   One of the many new features CAS has over LASR is the ability to use a cache as a backing store for in-memory data. The CAS cache provides flexibility for in-memory operations ensuring that CAS: Maintains availability of tables when more data is loaded than physical RAM available Provides failover protection if a CAS worker goes offline unexpectedly Relies on file handles to memory-mapped SASHDAT blocks to provide the mechanism for multiple CAS sessions to access the same single instance of data What is a backing store for CAS tables in memory? The data in CAS tables in memory are organized as SASHDAT blocks. And CAS will memory-map every SASHDAT block in memory to a location on disk. That disk might be the CAS cache. Or not.   If the original SASHDAT blocks are loaded from a mappable location, then that original source will be the backing store (caslib srctypes corresponding to Path, DNFS, and symmetrically co-located HDFS). Else if the table comes from another place (or a non-SASHDAT source), then CAS will memory-map that table's SASHDAT blocks in memory to the location specified for CAS_DISK_CACHE. Regardless of source, any new in-memory blocks created as output from CAS operations are always mapped to the CAS cache.   CAS relies on the operating system to manage the movement of data to/from the CAS cache via the memory map. This means the OS determines if and when to write (that is, page out) the in-memory SASHDAT data to the cache on disk. Even if the data is on disk, it's possible - ideal, really - that CAS might not ever need to read it back (that is, page in) from the cache. So for a system with plenty of RAM, the SASHDAT blocks may always be available to CAS in memory until the tables themselves are explicitly dropped.   The cache does provide a useful service and yet, as a rule, we prefer CAS to work with in-memory data as much as possible. If data must be fetched (that is, paged in) from disk, then performance should be expected to degrade by orders of magnitude for that operation. The CAS cache therefore is helpful when nominal RAM resources have been exhausted, but is not performance-equivalent to RAM.   The location for the CAS cache ( env.CAS_DISK_CACHE ) can be specified in several places:   File: When: Description: sas_viya_playbook/vars.yml At install The default used by CAS …/cas/casconfig.lua At startup on CAS Controller The default used by CAS Controller notifies Workers …/cas/casconfig_usermods.lua At startup on CAS Controller after casconfig.lua Overrides the default Controller notifies Workers …/cas/node_usermods.lua At startup on each CAS host after casconfig_usermods.lua Overrides the value from Controller per worker Analogy So, in a way, the CAS cache might be thought of as similar to an escape tunnel. It's there if we need it, but hopefully we won't. If we're digging a tunnel, but don't really plan to use it much, then we probably will spend as little time and effort on that tunnel's construction as we can get away with.   CAS is a very flexible product and can be used in many ways. Often, a site will use CAS in ways not originally envisioned at the time of purchase. We want to ensure that kind of flexibility of use is maintained. Also, as a high-performance product, many assumptions about CAS operations are based on the idea that hardware has been dedicated primarily, even exclusively, to CAS. If CAS is sharing its host(s) with other enterprise products, then we may need to adjust our approach.   So let's look at some tunnel - I mean, CAS_DISK_CACHE considerations. Primitive Resources There are some places on the CAS host which we can use as the CAS cache without any modifications at all. With minimal effort, we can get things working, but it's possible that we won't like it for long.   env.CAS_DISK_CACHE using /tmp Out of the box, CAS configuration defaults to using /tmp as the location of CAS_DISK_CACHE.   While /tmp is practically guaranteed to exist on most systems, it is not an ideal location to use for critical software operations. It might be sized too small (as little as 1 GB). And if /tmp fills up, then the operating system won't be able to run correctly, affecting everything else on the host. Furthermore, just because the files in CAS cache are indeed temporary, we want to ensure CAS itself makes the determination as to when they can be released.   ➤ Using /tmp as a short-term solution for working with small volumes of data with limited users is okay. But as a rule, we do not recommend using /tmp – instead, we should specify a different location.   Specified as:   env.CAS_DISK_CACHE='/tmp' env.CAS_DISK_CACHE using /dev/shm The /dev/shm location is a special construct in Linux referring to shared memory: it’s not a physical location on disk. Instead, it provides a mechanism for programs to share data with each other through the use of RAM. By specifying /dev/shm as the location for the CAS cache, the operating system will not need to perform any memory-mapped copying to slow disk.   ➤ Notice that we need to be careful using /dev/shm for the CAS cache because if RAM is exhausted, then the OS will swap out to its paging file. If that occurs, then all of sudden CAS will transition from running very fast to very, very slow - or worse, if CAS is running in a cgroup, then the largest RAM consuming process will be killed.   Specified as:   env.CAS_DISK_CACHE='/dev/shm' Improved Resources With a little bit of effort and planning, we can improve the route to CAS_DISK_CACHE so it can function well in a wider range of scenarios.     "In a hole in the ground there lived a hobbit. Not a nasty, dirty, wet hole, filled with the ends of worms and an oozy smell, nor yet a dry, bare, sandy hole with nothing in it to sit down on or to eat: it was a hobbit-hole, and that means comfort." -- Gratuitous reference to The Hobbit: There and Back Again Physical Disk as Persistent Storage If the site has requirements where CAS must load, manage, and operate on data which combined is larger than the RAM available, then the CAS cache really should rely on physical disk. After all, this concept has been de rigueur for storage considerations in computer science for decades. With physical disk, we can ensure that if circumstances occur where available RAM is not sufficient, then CAS has a dedicated location with plenty of space to store inactive data.   Most of the time, users of enterprise Linux servers aren't concerned with physical disks themselves. They just know about directories. Directories are free. Create them wherever you can. And this approach can work for CAS_DISK_CACHE. But at some point, things get real. If CAS fills up the root or user file systems, causing unexpected problems on the box, then the Linux administrator might want to have a discussion about giving CAS dedicated disk resources for its cache.   ➤ When provisioning physical disk for CAS cache, implement a single file system per disk. We prefer XFS and ext4 is acceptable. Avoid ext3 and any kind of NFS-based storage as the disk metadata and journaling may be overwhelmed by requests. env.CAS_DISK_CACHE using JBOD The acronym JBOD means Just a Bunch of Disks. It's a reference to one or more disks directly attached to the host with a basic intermediary connection. Almost like (but not really), we don't care how it's attached.   We typically want JBOD to be local-attached storage with enough disk space to match the size of RAM or more. Never less. Usually 1 - 3 disks are sufficient. When multiple disks are used, CAS has multiple I/O channels to access data and will create its files (with associated memory mappings) on those disks in round-robin order.   The challenges of working with individual disks are well-known. Each is a single point of failure and throughput is constrained. And the more disks we add to the system, the likelihood of a failure with those disks increases. If CAS loses a disk from its cache and that's the only place where that segment of data was, then the associated table(s) are incomplete - meaning losing a small part of a table is effectively the same as losing the entire table. Those files in CAS cache have a temporary lifespan, but CAS does need them.   ➤ Most enterprise IT teams don't expect to work with JBOD disks in this manner due to these challenges. So if positioning this as an option for CAS cache with your customer, take time to discuss with them. Perhaps a RAID solution (below) is more acceptable.   Specified as colon-delimited list of 1 or more full directory paths:   env.CAS_DISK_CACHE='/disk1:/disk2:/disk3' env.CAS_DISK_CACHE using RAID Having an escape tunnel is a nice feature. Furnishing it with nice appointments when you expect to use it regularly is a good idea. But there are cases where having a motorcycle on rails can help speed things up, too.   A RAID is a Redundant Array of Independent Disks. And there are many RAID levels with different functionality. As is typical for SAS solutions, we're focused here on RAID 5 where we have a group of 3 or more disks which work together as a single logical disk. Two-thirds of the space is for data and one-third is for parity. (For a 9-disk pack, only one-ninth is used for parity.) If one disk breaks, the others can continue operations. This eliminates individual disks as single points of failure and also provides some improvement to throughput as well.   ➤ Enterprise IT teams usually like this RAID approach. It's a standard offering with racked systems; easy to understand and justify. And our needs for CAS cache here are relatively modest. A single RAID array for the CAS cache should be fine.   Avoid shared storage solutions for CAS cache We want to keep the storage for the cache local to the CAS host - so do not cross the line over to shared storage solutions/appliances for CAS cache. Shared storage may appear convenient in situations where capacity is easily available, but the approach CAS uses with its cache negates some of the perceived benefits. Furthermore, some storage solutions, like IBM Spectrum Scale (a.k.a. GPFS), are not compatible with CAS_DISK_CACHE - it just won't work.   CAS cache is a completely separate conversation from using DNFS. DNFS technology used by CAS was explicitly designed for and works great with shared storage solutions.   For more information about RAID and other disk provisioning considerations for CAS, see Tony Brown's paper from SAS Global Forum 2019 titled, Engineering CAS Performance Hardware Network, and Storage Considerations for CAS Servers.   Specified as colon-delimited list of 1 or more full directory paths:   env.CAS_DISK_CACHE='/raidlogicaldisk1:/raidlogicaldisk2' Special Cases There are circumstances where CAS can be directed to operate in a way which (inadvertently) maximizes its reliance on the CAS cache. These situations need to be identified and recommendations made to either operate differently or tweak the configuration to reduce the impact. In-line scoring models The use of an in-line workflow to iteratively process scoring models, where the output from one run is appended with a new column and input into another run, may cause CAS to hit its cache excessively. So if CAS is relying on persistent storage for its cache, then expect a commiserate slowdown in performance since data loading from physical disk is much slower than from RAM.   ➤ If the iterative in-line approach to running scoring models is necessary, then consider: Ensuring there is plenty of RAM provided to CAS for this heavy use-case Configuring CAS to use /dev/shm for its cache Setting cas.MAXTABLEMEM=0 so that CAS won't cache very-temporary process files Using COPIES=0 in program code so CAS won't make any failover copies as its default behavior Super Highway Having an escape tunnel CAS_DISK_CACHE is a fine idea and it should be built correctly. But don't lose sight of the fact that CAS itself is meant to act as a super highway of performance. It is very flexible and extremely scalable. Remember that for CAS, we prefer to invest in scaling out horizontally with more CPU and RAM as problems get larger. So don't go too crazy with effort and resources trying to optimize every bit of performance out of CAS_DISK_CACHE. Often that can be better spent on CAS' primary mode of operation: in memory.     CAS cache offers very useful functionality to help in situations where primary performance resources (notably RAM) have been exhausted. Allowing a process to complete successfully, albeit slowly, is often preferable to outright failure. But slow is not the objective for CAS which is designed to operate at maximum performance and efficiency. Understanding how CAS can be configured for performance and resilience within the spirits of its design objectives is important for proper use. Understand more about how CAS uses its cache Before you can get serious about your decision on where to place the CAS_DISK_CACHE, you need to understand exactly what your processes will be asking CAS to do. With careful planning, a process can be optimized for CAS to reduce its reliance on the cache. On the other hand, some processes may require extensive use of the cache.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   I recommend getting acquainted with the SAS® Cloud Analytic Services 3.4: Fundamentals document for more information about how CAS works with data in-memory and on disk. Specifically look at topics including: Repeated tables for performance Fault tolerance and data redundancy Normal SASHDAT vs. compressed vs. encrypted Data partitioning and indexes Etc.  Thanks Just a quick shout out to Brian Bowman, Tony Brown, Gordon Keener, Steve Krueger, Barbara Walters, and many others for their work to identify, discuss, and explain the concepts here. ... View more

In SAS Viya 3.4, SAS Cloud Analytic Services (CAS) is the star of the show, providing lightning fast analytics of in-memory data for SAS Visual Analytics and other software offerings. However, Foundation SAS which is the long-time workhorse of SAS analytics is also offered, referred to as the SAS Programming Runtime Environment (SPRE). The SPRE provides a user interface and data processing environment for executing classic SAS program code. That is, it offers the Foundation SAS software we're all familiar with, including Base SAS, SAS/ACCESS engines, SAS/CONNECT packages, and more as well as the SAS Studio web application. Under the covers, however, there are two different SPRE implementations. One based on legacy SAS 9 Integration Technologies and the other built using the new Viya approach to software architecture. A Tale of Two SPRE The SAS Programming Runtime Environment is comprised of two different techniques to access Foundation SAS service: 1) legacy SAS Workspace Server relying on SAS Integration Technologies and 2) a new SAS Compute Server built using Viya architecture principles. Each has its own version of the SAS Studio web application to offer a SAS coding interface for users.   The inventory.ini file lists a set of host groups which represent pre-defined groups of SAS software components to deploy to specified hosts, including...   Legacy SAS 9 runtime host group: [programming] : Provides software in support of SAS Studio 4, including legacy SAS/CONNECT, SAS Object Spawner and SAS Workspace Server plus any additional software licensed (SAS/ACCESS, SAS/STAT, etc.). The SAS Workspace Server executes the SAS program code from SAS Studio. These components only come with a the "local" version of the SAS software license, so those components must all exist together on the same one host. Access this SPRE at url: http://[your Viya34 site].com/SASStudio New Viya Architecture SAS runtime host groups: [ComputeServer] : Provides the SAS Compute Server and SAS Launcher Server. The SAS Compute Server is the modern Viya equivalent of the legacy SAS Workspace Server. For SAS Studio 5, it executes SAS program code. [ComputeServices] : Provides the ComputeService and LauncherService microservices in support of the SAS Compute Server and SAS Launcher Server. [StudioViya] : Provides the SAS Studio 5 web app software. Relies on the SAS Compute Server and related services as the SAS backend. Access this SPRE at url: http://[your Viya34 site].com/SASStudioV As you can see, SAS Viya 3.4 effectively ships with two versions of SAS Studio - one for each flavor of Foundation SAS offered. SAS Studio 4 is the legacy offering with tight dependencies between software components, requiring them all together on the same host. And SAS Studio 5 is the modern Viya equivalent. SAS Studio 5 offers more deployment flexibility as implied by relying on three host groups.   See the SAS Viya 3.4 for Linux: Deployment Guide for information about the inventory.ini file and specifying deploy targets for the various host groups. Full vs. Programming-only There are two types of Viya deployment: full and programming-only. Full deployments are typically recommended and are most common. However, programming-only deployments are sometimes a better fit for specific use cases (such as using SAS Data Science).   Among other things, the programming-only deployment provides the [programming] host group in the inventory.ini file to install SAS Studio 4 and the legacy SAS 9 components as well as direct programming interfaces to SAS Cloud Analytic Services (CAS) through RESTful HTTP and languages like Python, R, and Lua.   The programming-only deployment does not provide host groups in the inventory.ini to install visual interfaces like SAS Drive and SAS Visual Analytics, nor does it include SAS Environment Manager or authentication services like SASLogon and Identities which are critical to the rest of Viya. It also does not include SAS Studio 5. The legacy SAS 9 runtime in the [programming] host group is built using pre-Viya software and so they still include their own user authentication capabilities. This means that programming-only deployments will rely on host-based authentication (usually via PAM in Linux hosts).   Interestingly enough, a full deployment of Viya provides host groups to install both SAS Studio 4 [programming] and SAS Studio 5 [StudioViya] along with [ComputeServer] and [ComputeServices]. For a full deployment of Viya, the SASLogon microservice is available to authenticate users with LDAP, Kerberos, OAuth, SAML, PAM, etc.   See "How Deployment Works" in the SAS Viya 3.4 for Linux: Deployment Guide for more explanation about different deployment types. And refer to the SAS Viya 3.4 Administration: Authentication doc to understand authentication options for your deployment. Considerations for SPRE on multiple hosts The SAS runtime - either as a legacy SAS Workspace Server delivered in [programming] or the SAS Viya Compute Server [ComputeServer] - can consume significant system resources like CPU, RAM, and especially disk I/O. If there will be heavy usage of the SPRE, then allocate hardware appropriately. For multi-host systems, avoid placing the SPRE on other mission-critical hosts like CAS servers to prevent resource contention.   In some cases, having just one host for the SPRE isn't sufficient. Identifying and eliminating single points of failure are key to improving system availability. Both types of SPRE can be deployed to run on multiple hosts to improve availability of their service. That said, it's important to understand that this does not provide failover protection of individual user sessions. That is, if a host fails, then any user sessions on that host (and their in-flight activities) are lost. To recover, the user would need to sign on a new SAS Studio session, establish a new SAS instance, and resume work from the last known good point.   With either legacy SAS Workspace Server [programming] or the SAS Viya Compute Server [ComputeServer], it is important to provide a shared file system for common data and other files which must be accessible across multiple hosts: Data Mart: SAS programs run in the SPRE could execute on any deploy target. We must ensure each of those hosts has access to the underlying data that the SAS program expects to work with. User Home Directories: The users' SAS preferences, the SASUSER libref, and other important files must be available for a consistent user experience. Performance: We typically exhort our customers to implement shared file systems which are can transfer data simultaneously to every concurrent execution of Foundation SAS at a rate of 100 MB/sec/core or more! Other: If you're already experienced with SAS Grid Manager implementations, this should all be familiar to you: Viya doesn't offer SAS Grid Manager (yet) and so related concepts like shared SAS binaries and configuration directories, checkpoint-restart, etc. do not apply For more information about shared file system considerations, see my article: Contemplating shared file systems for SAS. Deploy SAS Studio 4 and its SPRE to multiple hosts The SPRE host group for SAS Studio 4 [programming] bundles several pieces of disparate software together: SAS/CONNECT Spawner and Server, SAS Object Spawner, SAS Workspace Server, SAS Studio 4 web app and its own web application server.   By adding multiple deploy targets to the [programming] host group, full copies of all those components are placed on each host. Each deployment of [programming] operates statelessly with no coordination between each other.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   ▶ Side note: The [Operations] host group has a dependency on the [programming] host group (relying on some of those components to do its job). However, [Operations] is currently a singleton-only service - meaning it can only run on one host in the Viya environment and cannot be clustered for improved availability. Specify the deploy target for [Operations] that is the same as the first-listed deploy target for [programming].   To ensure that all [programming] hosts get their fair share of user workload, manual effort to configure load balancing in the Apache HTTP Server (or some equivalent) is required as a post-deployment task for programming-only deployments.     In this setup, each new user session is distributed in basic round-robin fashion to the hosts defined for SAS Studio 4. Each user session in SAS Studio establishes its own SAS Workspace Server instance on the same host. Each user's subsequent in-session requests are automatically routed to the same assigned host.   For full deployments of SAS Viya 3.4, the http-proxy microservice will automatically configure the load balancing when more than one [programming] deploy target is running. Deploy SAS Studio 5 and its SPRE to multiple hosts Much like a multi-tier deployment in SAS 9, we can deploy SAS Studio 5 and its SPRE host groups onto different deploy targets which are optimized for those workloads. In accordance with modern Viya architecture design, each of these host groups can run on multiple deploy targets with clustering configuration provided automatically by the Viya deployment process. So you don't need to muck about with the Apache HTTP Server proxy configuration at all.     The [StudioViya] host group provides a web application server for running the SAS Studio5 web app. Specify two or more deploy targets to improve availability of the SAS Studio 5 web app. Place this host group alongside other SAS Viya web apps on host(s) optimized for web application hosting.   The [ComputeService] host group provides ComputeServices and LauncherServices microservices for coordinating startup and activities of the the Compute Server and Launcher Server. Specify two or more deploy targets to improve availability of these services. As these services are relatively lightweight, they should be placed alongside other Viya infrastructure services.   The [ComputeServer] host group provides the SAS Launcher Server (similar in part to the SAS 9 Object Spawner) and SAS Compute Server (similar to SAS 9 Workspace Server) software. The SAS Compute Server can require significant system resources - CPU and RAM, of course, but is most often constrained by disk I/O. If you're expecting many concurrent users and/or time-sensitive batch jobs, then provision adequate deploy targets for this service. Specify at least two deploy targets for improved availability or more as needs dictate for workload.   For SAS Studio 5, there is no deterministic workload balancing of the SAS Compute Server… not even round-robin. When multiple deploy targets for [ComputeServer] exist, then SAS Compute Server host selection is random. Watch for improvements in this area coming with SAS Viya 3.5 or later. Interesting developments Even though the [programming] components are based on legacy SAS 9 Integration Technologies software with a SAS Object Spawner and SAS Workspace Server included, in SAS Viya 3.4 those components are restricted by a local software license to only work with SAS Studio 4 which resides on the same host machine.   For Viya 3.5 - expected to ship around 19w47 - an enterprise-grade license will be included with the [programming] components which will allow them to accept client connections from other hosts. This should allow SAS clients like SAS Add-in for Microsoft Office and SAS Enterprise Guide to connect to the [programming] SAS Object Spawner and SAS Workspace Server.   That said, the expectation is that the deployment process for the [programming] host group will remain the same in Viya 3.5 - all components included in just one host group. Summary The following table summarizes many of these concepts for easy review.     SAS Studio 4 and its SPRE SAS Studio 5 and its SPRE Available in deployment type In both Full and Programming-only deployments. Only in Full deployments. Foundation SAS software provided by... [programming] > SAS Workspace Server [ComputeServer] > SAS Compute Server Multiple SPRE hosts requires shared file system Yes, for user home directories and common data files. Yes, for user home directories and common data files. Multiple SPRE hosts requires manual configuration of Apache HTTP Server proxy definitions Yes, for Programming-only deployments of SAS Viya.   No, Full deployments of SAS Viya include ability to automatically cluster the SPRE. No, Full deployments of SAS Viya include ability to automatically cluster the SPRE. SAS Foundation is accessible to SAS Add-In for Microsoft Office and SAS Enterprise Guide No, with Viya 3.4.   Yes, with Viya 3.5. No. And as always... Remember to work with the SAS Enterprise Excellence Center to perform a proper workload estimate and hardware sizing prior to provisioning hardware and deploying SAS software. This will help ensure that the SPRE has adequate resources to accommodate the expected workload and number of concurrent users.   ... View more

In my earlier posts, we discussed illustrations of various deployment patterns and anti-patterns for SAS Viya when working with different numbers of server hosts. To refresh your memory, see Deploying SAS Viya on 1, 2, 3 servers and Deploying SAS Viya on more servers.   For this article, we will look at a real world, multi-host deployment of SAS Viya 3.4 which provides higher availability at a major European municipality.   The following diagram illustrates the major architectural components employed:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   This deployment pattern isn't directly represented in my earlier articles. So here's what I would've written… Scenario: A smart compromise ⇒ 6 or more hosts In some situations where the workload for Viya and its implications for the host machines are well understood, then we can further combine roles onto the same hosts. Specifically, we can place the CAS controller(s) alongside other Viya infrastructure on the same machines and still provide improved availability. Only the CAS workers would be deployed to dedicated hosts.   ▷ 3+ hosts for SAS Viya Stateful Services, Microservices, SPRE, and CAS Controllers [Stateful services host groups: consul, httpproxy, pgpoolc, rabbitmq, sasdatasvc]   [Microservices host groups: AdminServices, CASServices, ComputeServices, CoreServices, DataServices, HomeServices, ReportServices, ThemeServices, configuratn, and many more]   [SPRE host groups: Operations, ComputeServer, programming]   [CAS host groups: cas-casserver-primary, sas-casserver-secondary]   Why 3 hosts? The SAS Configuration Server software is a specially packaged version of Consul from Hashicorp. Consul is designed to implement HA through a consensus methodology which relies on an odd number of hosts (that is 1, 3, 5, not 2, 4, 6) to prevent the split-brain problem. The same concept also applies to RabbitMQ, which is the technology behind SAS Message Broker.   Deployment: Place the consul and rabbitmq host groups on all 3 hosts. Place the sasdatasvc (SAS Infrastructure Data Server) host group on 2 hosts. The pgPool-II software currently cannot be clustered yet (to be addressed in Viya 3.5). Place the pgpoolc hostgroup on the primary sasdatasvc host. Place the host groups which provide the SPRE (Operations, ComputeServer, programming) on 2 (or more) hosts – with the exception of the operations microservice. It’s currently not clusterable (yes, single point of failure) and can only deploy to a single host. Add SPRE to more hosts if needed (at time of initial deployment only!). Place 2 sets of the microservice host groups across all 3 hosts, distributed evenly by expected RAM usage. Place sas-casserver-primary (CAS controller) host group on 1 host and sas-casserver-secondary (CAS backup controller) host group on a different host. ▷ 3+ hosts for CAS Workers [CAS host groups: cas-casserver-worker]   In the illustration above 13 hosts for CAS workers are provided.   Deployment: Place the sas-casserver-worker host group on separate physical hosts. Add (or remove) workers at any time. 6+ Hosts Points of Interest The co-existence of Viya services plus SPRE plus CAS controllers may cause contention for limited resources in some situations - yet it provides a minimal footprint with effective improvement to overall availability. The placement of the SPRE must be decided at initial deployment - and may be difficult to change later without a complete redeployment of Viya. So plan this carefully to ensure SPRE usage is inline with this deployment scenario. Consider keeping the SPRE separate from the primary CAS controller. See my post, Deploying the SPRE in SAS Viya 3.4, for more information. Keep in mind that the SPRE is typically licensed to run on the same number of CPU cores as allocated for CAS. If CAS is licensed for 100 cores, then the SPRE can run on another 100 cores. Because a CAS controller and SPRE co-exist on 1 host, it's like those cores are counted twice (from a licensing/usage perspective). The CAS backup controller host does not count towards the number of CPU cores utilized by CAS (until it takes over primary role). The SPRE software as well as the full set of microservices were deployed on to each of the three Viya hosts. The singleton operations microservice was placed on the third Viya host without a CAS controller (since it'd be less busy). Not shown, but also the pgpoolc hostgroup was placed separately from the two sasdatasvc host group instances on the third Viya host without a CAS controller. This required manual post-deploy effort to finalize pgPool-II communication with the PostgreSQL servers - and the team would likely recommend against it in a fresh deployment, preferring instead to keep the pgpoolc host group assigned to the same primary host as the sasdatasvc host group. More Environmental Considerations The great thing about contemplating real-world deployments is that we can see where other considerations are important to ensure success. Data Loading: For this site, data is only loaded into CAS by running it serially through a SAS 9 instance (run as part of SAS Grid Manager). While not the speediest approach for loading data into CAS, it was selected to make implementation simpler. Providing CAS with direct access to other data (both native and external sources) may be a future update. Shared File System: IBM SpectrumScale (GPFS) is a powerful and flexible shared file system technology. And it integrates with SAS technologies very well. It's currently used here to provide shared storage between the primary and secondary CAS controllers as well as for SAS Viya backups.   GPFS is a natural fit as a shared file system to provide high-performance data transfer to multiple CAS workers (as seen with the DNFS type of caslib, and other data sources). Do not use GPFS for CAS_DISK_CACHE. It is not compatible with that use case. Instead, provision local disk resources for the CAS cache. Load Balancer for Web Activity: Multiple instances of the Apache HTTP Server can be deployed to provide improvement in availability and scalability. However, they generally are unaware of each other, operating statelessly. So a third-party balancer is necessary to direct traffic to each HTTP Server (with session stickiness configured). There are many technologies which can fill this role - just make sure it's deployed with high availability, too.   Coda While there are many ways to deploy Viya, not all of them are appropriate for your site and use cases. Plan your Viya deployment carefully taking into account the various workloads and expectations for performance. SAS can help with this effort. Contact the SAS Enterprise Excellence Center for free hardware sizing estimation.  ... View more

SAS software has always been very flexible to meet the needs of different workloads, performance requirements, user expectations, and more. To do so, SAS Viya is flexible to work with deployments ranging from one server to hundreds. Determining the actual number of host machines for the SAS solution is not always easy. In my previous post, we looked at the possibilities when forced to fit Viya within 1, 2, or 3 host machines.   What we learned is that while SAS Viya can scale down to run in that limited environment, we must necessarily give up some functionality to make it all fit. This is a valid choice - but one we must ensure the customer understands properly for best result.   In this post, let's tackle this same concept from a different angle - this time, let's look at a large-scale implementation where hosts provide not just scalability and availability but are tuned specifically for the specialized software roles employed. The cart is still before the horse It's time to reiterate that we should not simply plan a Viya deployment based on an arbitrary number of host machines, regardless if they're few or many. While we can get to a functional deployment that way, it's not going to provide the ideal experience.     Instead, discuss all aspects of requirements and expectations with your customer. And, of course, get a sizing for your customer’s solution performed by the SAS Enterprise Excellence Center. The EEC will ask the questions to determine the workloads involved and return with a recommendation for hardware specifying CPU and RAM to meet typical performance expectations. The shape that the hardware actually takes based on those recommendations as well as other considerations of the business is then something we will need to work with.   With these formalities addressed, then let's look at what we have to work with. Go big or go home Viya - and CAS in particular - are designed to accommodate massive scalaility - running across many hosts as well as enabling increasing compute capacity easily by adding even more hosts. A lot of machines implies significant investment in terms of hardware costs, ongoing operations, administration, and even moreso with software licensing, user training, data management, and so much more.   When customers make that kind of signficant investment, they want to protect it. And so oft times, there will be requirements to ensure the ongoing availability of the system to minimize unexpected outages. High availability considerations therefore go hand-in-hand with scalability considerations.   Furthermore, SAS Viya is comprised of many disparate software technologies. Compare CAS with its massively parallel processing which is highly dependent on all working data available in RAM with the SAS Programming Runtime Environment which uses the more classic model of disk-based data storage and access. Both are considered computation engines, and yet they function very differently. Tuning a host ideally for one may mean that the other won't run as efficiently as it could. Fortunately, the Viya architecture allows us to separate these computation engines to different hosts - if we choose. And we can extend that concept to other aspects of Viya as well. If you're familiar with multi-tier deployments of SAS 9, this is a similar concept applied to new technologies. Scenario: Specialize Everything ⇒ 12 or more hosts Viya offers the ability to deploy the SAS software in preset groupings referred to in Ansible technology as host groups. This allows us to break up the software deployment across a number of host machines. Understanding which software components populate each host group then is necessary to devise a deployment of Viya where hosts can be tuned especially for the software they will run.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   In this scenario, we deploy the SAS Viya software to separate hosts which can be optimized for their specific tasks. ▷ 3 hosts for SAS Viya Stateful Services [Host groups: consul, httpproxy, pgpoolc, rabbitmq, sasdatasvc]   Why 3 hosts? The SAS Configuration Server software is a specially packaged version of Consul from Hashicorp. Consul is designed to implement HA through a consensus methodology which relies on an odd number of hosts (that is 1, 3, 5, not 2, 4, 6) to prevent the split-brain problem. The same concept also applies to RabbitMQ, which is the technology behind SAS Message Broker.   Deployment: Place the Consul and RabbitMQ host groups on all 3 hosts. The other stateful services need only be placed on any 2 of the hosts for their improved availability. So place sasdatasvc on 2 hosts and rabbitmq on 2 hosts (with only 1 overlap). The pgpoolc software currently cannot be clustered yet (yes, it's still a single point of failure - to be addressed in Viya 3.5). Place pgpoolc on the host without sasdatasvc instances. ▷ 2 hosts for SAS Microservices [Host groups: AdminServices, CASServices, ComputeServices, CoreServices, DataServices, HomeServices, ReportServices, ThemeServices, configuratn, and many more]   In general, the SAS microservices are mostly built using Spring Boot technology running on Java. This is not required, but common for Viya right now. Future microservices may be built in Go or any other HTTP RESTful friendly technology.   Why 2 hosts? To ensure availability in case one physical host goes down.   Deployment: Place all microservice host groups on both hosts. ▷ 2 hosts for SAS Programming Runtime Environment [Host groups: Operations, ComputeServer, programming]   The SPRE provides a runtime environment for execution of classic SAS program code.   Why 2 hosts? To ensure availability in case one physical host goes down.   Deployment: Place all SPRE host groups on both hosts - with the exception of the operations microservice. It's currently not clusterable (again, single point of failure) and can only deploy to a single host.. Add more hosts if needed at time of initial deployment. ▷ 5 hosts for SAS Cloud Analytic Services [Host groups: sas-casserver-primary, sas-casserver-secondary, sas-casserver-worker]   CAS is our flagship product for massively scalable processing of huge-volumes of in-memory data.   Why 5 hosts? We need 2 hosts to provide improved availability of the CAS controller role (primary and secondary). While MPP CAS will function with a single worker, that's inefficient and doesn't provide any worker failover. Two workers gives us that minimum redundancy - however, in my opinion, we should always have more worker hosts than controller hosts - so 3 workers then. Add more if/when needed.   Deployment: Place each CAS Controller on separate physical hosts. Place each CAS Worker on a separate physical host. Add more (or remove) hosts at any time. 12+ Hosts Points of interest Each group of hosts (stateful, microservices, SPRE, CAS) can be optimized for their specialized workloads: CAS: network, memory, and CPU intensive SPRE: disk and CPU intensive Microservices: memory and CPU intensive Stateful: network, disk, and CPU intensive If one Consul server process goes down, then any microservices which are placed on the same host machine are automatically de-registered even if they're running okay. So separating Consul from all microservices improves the effective availability of the microservices even more. Keep in mind that the SPRE is typically licensed to run on the same number of CPU cores as allocated for CAS. If CAS is licensed for 100 cores, then the SPRE can run on another 100 cores. The CAS 2nd Controller host does not count towards the number of CPU cores utilized by CAS (until it takes over primary role). Scenario: Let's Compromise ⇒ 8 or more hosts Perhaps your customer doesn't need that much specialization and optimization of hosts. Maybe they don't plan to use the SPRE very much. Or it may be that they're new to using Viya and understand needing some specialization as well as accommodating scalability and availability, but want to keep things conceptually simplified. So let's compromise.     In this scenario, we've placed infrastructure services (stateful and microservices) together on one set of hosts along with the computational services (SPRE and CAS) on another set of hosts. ▷ 3 hosts for infrastructure services This is a very common deployment for the Viya infrastructure services.   Deployment: Place the Consul host group on all 3 hosts. The other infrastructure services need only be placed on any 2 of the hosts for their improved availability. For the stateful services, follow the guidance shown above for 12 hosts. For the microservices, try to spread them evenly based on their memory-usage. ▷ 5 hosts for SAS Cloud Analytic Services and SAS Programming Runtime Environment Both CAS and the SPRE are considered runtime environments. But they work very differently in actual operations.   Deployment: Place the SPRE components on the CAS Controller hosts. The rationale here is that the CAS Controller hosts are not typically used as heavily as the CAS Workers. Assuming identical host sizes between controllers and workers, then the free overhead might be sufficient room for the SPRE. 8+ Hosts Points of Interest This approach is a compromise allowing Viya to run on fewer hosts, but with more potential conflict as services compete for resources. If this kind of compromise is sufficient to meet your customer's requirements, then great. Else look at where changes must be made. To ensure uniform efficiency, we try to keep the CAS Workers running on similar hardware with similar workloads. That means that not just the hardware (CPU, RAM, disk, etc.) is the same for all workers, but also the OS, background processes, and anything else as well. So avoid placing the SPRE on CAS Worker hosts. Even if you place the SPRE on all of the CAS Workers, the actual SPRE workload will vary significantly with each user, each job, etc. Scenario: Constrained Can Work ⇒ 5 or more hosts Your customer wants a scalable CAS deployment with rudimentary availability improvements. And they don't expect a large number of users as much as they expect a few users to perform work on large volumes of data. Then we can tackle that, too.   ▷ 5 hosts for CAS, SPRE, and all of the Viya infrastructure Running everything together - but with some careful placement decisions to smooth out the kinks.   Deployment: Place two instances of the Viya infrastructure services on the same hosts alongside the SPRE and the CAS Controllers - except for Consul. In a crazy twist, let's place the Consul host group on 3 of the CAS Workers. 5+ Hosts Points of Interest This approach is best for a sites with only a few users who perform heavy work in CAS, but which don't stress the CAS controllers with tasks. The controller hosts are overloaded even further with more work as well as RAM-hungry microservices. Ensure those machines are beefy enough with enough resources to do all of that. The really unusual bit here is that idea of placing Consul on 3 of the CAS Workers. The idea is that Consul is relatively lightweight - not consuming much RAM, CPU, or network in normal operations. Placing all 3 Consul Servers on the CAS Workers keeps those worker hosts similar in terms of workload. And with Consul separate from the microservices, it helps improve their effective availability slightly as well. Which scenario is right for me? To be frank, I don't know. There's a good chance none of those shown here or in my previous post are exactly right for your customer's requirements and expectations. The point of these posts is to convey some of the architectural concepts behind different deployment decisions so that you can work with your customer to design your own scenario.   Even within a single customer project, you may need to make use of multiple scenarios at the same time. Production environments vs. Dev/Test environments. Or environments optimized for data and analytic processing as compared to environments which specialize in report delivery and consumption. Throw in ESP, MAS, other SAS solutions, and the possibilities are nigh endless. Coordination of SAS Viya service operations Contrary to popular belief, SAS Viya does not really handle random startup or shutdown order of its constituent services very well. When everything is installed on a single host machine, then the sas-viya-all-services script will correctly handle startup and shutdown order. That specific use-case works so well, that the default is to configure the operating system to automatically start Viya services at host startup.   But if you distribute the Viya software components across multiple host machines, then there's a real problem. First is that the sas-viya-all-services script does not talk across machines. So it's not for "all services on all machines", but for "all services on this machine". Furthermore, there are some dependencies between Viya software services - some weak (they'll figure it out on their own) and some very strong (startup order matters! Else it just breaks). So we need to disable the automatic startup of Viya services on those hosts and then work to ensure each of the Viya services are started in proper order to meet those dependencies.   To help ensure proper startup and shutdown order of SAS Viya services, we recommend implementing the VIRK's Viya Multi-Machine Services Utilities Playbooks. In that Github repository, you can find a set of playbooks to start or stop the SAS Viya services gracefully across the 1 - n machines that are identified in the inventory.ini file. Share and enjoy! Decision time When deploying Viya, some decisions must be made at the time of initial deployment. For example, if your customer might want multi-tenancy later, then you must decide whether to enable multi-tenancy right now at initial deployment -or- deploy Viya as single-tenant now, but then re-deploy from scratch as multi-tenant later. Some aspects of Viya infrastructure clustering have similar considerations - what you decide at initial deployment can have an impact later.   Some quick examples of Viya software which needs clustering decisions made at initial deployment:   Viya service: Decide now or later: Notes: RabbitMQ Now Clustering RabbitMQ after the initial deployment is a manual process with constraints SAS Cloud Analytic Services Later Scalable after initial deployment, even from SMP to MPP Apache HTTP Server Later Remember running multiple httpproxy requires a 3rd-party load balancer, not provided with Viya SAS Programming Runtime Environment Now For multi-tenant deployments of SAS Viya, adding ComputeServer hosts after initial deployment is not yet supported. Later Adding more hosts to the ComputeServer after the initial deployment is supported for single tenant deployments of SAS Viya   To keep a simple rule in mind, the Viya stateful services are critical and they typically are less forgiving of mis-configuration or other challenges. So try to determine their ultimate deployment topology as early as possible and then stick with it. The microservices are generally more accommodating of changes since they're designed to be resilient in that way already. And then CAS has been designed with future scalability in mind, so scaling it out is relatively easy. As a matter of fact, hosts can be added to (or removed from) an existing MPP CAS deployment without any interruption in service.   The SPRE is a a little complicated as it's not a single component, but many spread across different host groups in the inventory.ini file. And its deployment considerations vary on several key factors. See my article, Deploying the SPRE in SAS Viya 3.4 for more details.   Coda SAS Viya offers a dizzying range of options for architecture, deployment, and operation. A couple of articles are not sufficient to address all of the options. However, we can all improve our knowledge our how the components of Viya work together and then use that to benefit our customers with a deployment plan which is tailored to their specific needs. My next blog post will describe the considerations weighed for an actual customer implementation. See Deploying SAS Viya in the real world. ... View more

Our customers come to SAS for unique and powerful software solutions to address their data, analytics, and reporting challenges. Our software has to run on hardware which the customer is responsible for providing whether that's on-premise, in the cloud, or through a third-party provider. Sometimes the business environment can dictate how that hardware is provisioned in a such a way that we must adapt the SAS solution software to make the best fit possible. Fortunately, SAS software is flexible to a very wide range of deployment scenarios.   This means that sometimes, often for proof-of-concepts or budget-constrained implementations, we find that a SAS Viya deployment project may come with a pre-determined number of server machines. This is usually not ideal. And so it's important to understand what this can mean. So let's try to answer the questions presented by, "My customer gave me X machine(s) for Viya. How do I deploy the software?" Don't let the cart get before the horse You know what I'm gonna say here:  get a sizing for your customer's solution performed by the SAS Enterprise Excellence Center. The EEC will ask the questions to determine the workloads involved and return with a recommendation for hardware specifying CPU and RAM to meet typical performance expectations. The shape that the hardware actually takes based on those recommendations as well as other considerations of the business is then something we will need to work with.   Whenever possible, let's ensure we've done our due diligence in advance of the customer buying hardware. Once they've committed to a specific layout of server hosts, it can be difficult to make changes later. Work on identifying all requirements for your solution before designing a hardware solution to support it.   So now that we've acknowledged how the hardware situation should be determined in typical implementation, then let's look at what happens when we put the cart before the horse - trying to fit Viya software on an arbitrary number of machines.     The scenarios described below do not cover all possible use cases. This post focuses primarily on deployment patterns which meet original design intent as well as some common anti-patterns. CAS deployments and tenants With SAS Viya 3.3, we gained the ability to configure multi-tenant environments. Each tenant runs with their own CAS configuration, dedicated memory space, alongside a shared set of Viya infrastructure services. There is no limit on the number of tenants (or users within those tenants).   With SAS Viya 3.4, we also gained the option to install multiple CAS deployments. Each CAS deployment is physically separate from the others (meaning each is installed on its own dedicated host(s)). Keep in mind that each CAS deployment counts toward the total number of CPU cores which have been licensed for CAS overall at the site.   If we choose, we can combine these concepts so that each CAS deployment provides analytic services to multiple tenants. One server With a single server machine for Viya, we will place all of the Viya infrastructure services as well as one deployment of SMP Cloud Analytic Server together on one host. The requirements of production workload running well in a single machine environment will often necessitate some beefy server specifications.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page. Scenario: 1-1-a = All on one host machine For all-on-one host deployments of SAS Viya 3.4, both Linux and Windows Server operating systems are supported by SAS. With all of the Viya software together on one host, this means SMP Cloud Analytic Server is the only option.   In the future, SAS expects to offer the ability to run Viya infrastructure services across multiple Windows Server hosts. So Viya infrastructure services may be clusterable and SMP CAS may also be deployed to separate, dedicated host(s). MPP CAS on Windows Server isn't currently on the roadmap.   Given a choice, we almost always will prefer Linux over Windows Server. Most importantly, running Viya on Linux gives the option of scaling up from SMP CAS to MPP CAS without having to redeploy Viya from scratch. Two servers Compared to a single host, we can expect that two servers will perform twice as much work, handle twice as many users, process twice as much data, right? Well, maybe… mapping functional operations to physical resources is more nuanced than that. But having two servers does have usable benefits beyond straight performance. One CAS deployment Even with just one deployment of SAS Cloud Analytic Server, we have multiple options on how to install it.   Scenarios: 2-1-a = Separate hosts for Viya infrastructure and SMP CAS Place all of the Viya infrastructure services together on one host. Install SMP CAS on the other host. Optimize each host for its purpose. 2-1-b = Install MPP CAS on 2 hosts We can deploy MPP CAS to run on two host machines: one host dedicated to run as a CAS Worker and the other as CAS Controller alongside all of the Viya infrastructure services. The problem with this is that a two host MPP CAS deployment is the least efficient possible. Running in MPP mode with only two hosts involves unnecessary orchestration overhead for what is a single server's workload. Avoid this scenario. Two CAS deployments With two servers now we also have the option of two CAS deployments: installing an SMP CAS on each host. One SMP CAS will live alongside the Viya infrastructure services and the other on a dedicated machine.   Scenario: 2-2-c = Two deployments of SMP CAS With one host dedicated to SMP CAS already, we could also deploy a second SMP CAS to the Viya infrastructure host. Obviously, that SMP CAS would share physical RAM and CPU when running alongside the Viya infrastructure services. Smaller jobs for fewer users is best there. The deployment of SMP CAS on the dedicated host could be optimized for larger workloads. Three servers and beyond As we add more servers to these scenarios, the possibilities multiply. With each additional host, we have options to consider which can significantly improve analytics processing as well as service isolation and availability. One CAS deployment   With three hosts, we've reached the minimum cluster size for running MPP Cloud Analytic Server. Scenarios: 3-1-a = Viya infrastructure on one host with MPP CAS on all three hosts This is typical for a small multi-host deployment of Viya. The MPP CAS Controller doesn't typically consume as much RAM and CPU as the CAS Workers, so it should reside comfortably enough alongside the Viya infrastructure services on the same host. 3-1-b = Attempt high availability of Viya infrastructure and CAS Workers Enable clustering of Viya infrastructure services and rely on built-in data availability offered by the CAS Workers. For Viya 3.3, you could make the case that this would be the smallest footprint possible to achieve high availability. But remember that with Viya 3.4, we now have the CAS Secondary Controller to improve availability of that role - and it's not shown in this illustration because it would require a fourth host. Regardless, this approach prioritizes availability over performance - which is not typical. Avoid this scenario. Two (or more) CAS deployments   If your site requires more than one CAS deployment using only three servers, then SMP CAS is the only option. Scenarios: 3-2-d = Viya infrastructure on one host with SMP CAS on two hosts Dedicate one host for Viya infrastructure services and deploy two SMP CAS. Optimize each host for workload. 3-2-e = Viya infrastructure on one host with SMP CAS on all three hosts This could be functional, but it's getting out of hand. If your site has this much CAS computation requirement, then revisit deploying MPP CAS and consider adding more hosts, multi-tenancy, etc. Avoid this scenario. 3-2-f = Attempt high availability of Viya infrastructure and SMP CAS Enable clustering of Viya infrastructure services and manually manage analytics and data availability across multiple SMP CAS. While automatic failover of most Viya infrastructure services would be provided (exceptions: pgPool-II and the operations microservice), ensuring availability of CAS resources to end users would require significant manual effort. Instead, consider deploying MPP CAS, adding more hosts, etc. if availability is a significant requirement. Avoid this scenario. What about other SAS computational engines? That's a great question. This post has so far ignored considerations for the implementation of the SAS Programming Runtime Environment, the SAS Event Stream Processing Server, the SAS Micro-Analytics Service, the SAS Embedded Process, and more. They each present their own factors and requirements to contemplate in the overall solution architecture. Let's take a quick look at the SPRE...   Scenario: 3-1-c = Viya infrastructure on one host, SMP CAS on one host, and SPRE on one host In the other illustrations for this post, the SAS Programming Runtime Environment (consisting of SAS Launcher Server and Service as well as the SAS Compute Server and Service) is not shown, but assumed to be deployed to the same host machine as the rest of the Viya infrastructure services. However, if you expect significant workload for the SPRE, then its components can be deployed to one or more additional servers. The number of licensed CPU cores the SPRE Compute Server can run on is the same count as for CAS. Optimize each host for its workload. Ready to add more servers The promise of Viya is that it utilizes many modern approaches to software architecture. A key tenet of this is the ability to scale the solution easily (with "easily" being a relative term best compared to SAS 9 equivalents). In particular, it is possible to scale up CAS from SMP mode on a single host to MPP running as a cluster across multiple hosts with a planned, short-term outage simply by running the necessary Ansible playbook. If you already have MPP CAS and want to add more Workers, this can be accomplished with no outage at all.   And don't forget that if growing the Viya deployment from one host to many is a future goal, then ensure your customer opts to deploy using Linux-based hosts, not Windows Server. Other takeaways The SAS Cloud Analytic Server was designed from its inception to handle the largest analytic workloads. It's fast and extremely scalable. Performance therefore, is its raison d'être. For best performance, consider starting with at least four server machines with one host for the Viya infrastructure services and three hosts for MPP CAS. This provides separation of Viya infrastructure services from the CAS workload and allows optimization of hosts for their respective purposes.   For smaller deployments of Viya - three or fewer machines - then we acknowledge that huge data processing performance is really no longer the primary driving factor. Accommodating cost effectiveness, constrained budget, limited data volume, few users, short project lifespan, or similar is the goal. It's great that Viya is flexible to these other business drivers… we just need to ensure our customers understand the tradeoffs for the selected approach.   High availability and disaster recovery considerations will again drive up the number of necessary server hosts. When scaling out and increasing the number of hosts in the cluster for performance, then we often can gain advantages in terms of availability as well.   My next post illustrates SAS Viya deployments with four or more host machines. See Deploying SAS Viya on more servers. Coda SAS Viya offers so much flexibility in terms of architecture, deployment, and operation that it's a real challenge to run down all of the possible options. This post covers only a select few of the possible combinations offered when deploying on up to only three hosts - and we typically see most multi-machine deployments starting with at least four hosts, with more to come later.   It's important to understand the crucial driving factors of your customer implementation of SAS Viya. Hardware choices will have a significant impact on Viya's ability to accomplish the goals desired. ... View more

The way that the SAS Viya Cloud Analytic Services (CAS) Server manages data in memory is a complicated topic. There are many twists and turns to cover all of the nuances and possibilities. However, most of that behavior can be distilled down to a few simple rules which adequately describe the majority of situations. This post will provide a brief but illustrative guide to the inner workings of how data is managed in memory by CAS. The Rules Let's start with what you came here for: № 1. CAS analyzes data in SASHDAT format, regardless of the original source Data loaded into CAS can come from a variety of sources and formats thanks to the SAS Data Connector technology. And of course, CAS can write that data back out into a myriad of other formats as well. Refer to the SAS documentation about Working with SAS Data Connectors for a lot more explanation.   However, the data that CAS has in-memory is exclusively in SASHDAT format, regardless of its original source or ultimate destination. The SASHDAT structure is optimized for distributed analytic processing. № 2. All in-memory data is memory mapped to a locally-accessible backing store Memory mapping associates an address in RAM with a location in locally-accessible storage (hard disk, file system, etc.).   Most of the time, CAS will memory-map any data loaded into memory to the CAS_DISK_CACHE. But, CAS_DISK_CACHE is not the only backing store for in-memory data.   If the original source data is already unencrypted SASHDAT which is accessible via local storage, then CAS will memory-map to that location as the backing store instead - but just for the source data.   All changes to in-memory data are memory mapped exclusively to the CAS_DISK_CACHE. № 3. The OS, not CAS, moves data to/from the backing store Here's an over-simplified explanation: CAS initially places the data either in RAM or on disk and then creates the memory map. CAS then references the place in memory for the data it wants. If the data is already there in RAM, then great, it's instantly available. Else the OS will "page in" the data from disk at the other end of the memory map. Similarly, if data in RAM is unused and the OS determines that it needs to free that space for other work, then the OS will automatically "page out" that data to the disk at the other end of the memory mapping. There are other situations not covered here, but the gist of it is that the OS is very efficient at managing data access with memory mappings.   This approach effectively provides a virtual memory scheme which increases the amount of data CAS can bring on-board and work with, reducing the need for manual administration of in-memory table availability. № 4. CAS_DISK_CACHE is where all replicated copies of data are stored Besides acting as a virtual memory storage backend, CAS_DISK_CACHE also provides data protection for CAS workers. Additional copies of SASHDAT blocks can be stored in CAS_DISK_CACHE so that a complete SASHDAT table is still available if a CAS worker goes down (taking its allotment of data blocks with it). Remember, these rules are just to describe a basic understanding of how CAS operates with respect to data in memory. There are many configuration and programming considerations available which can be used to address a wide spectrum of different scenarios that do not adhere to these rules. But I find that these rules provide a good starting point from which to explore those variations. Implications There are many implications which result from how CAS manages the data in memory. Let's take a quick look at a few of them. CAS_DISK_CACHE is virtual memory... to a point As a virtual memory scheme, the idea is that CAS_DISK_CACHE provides resilience on occasions when RAM is over-pressured with too much data. If needed, in-memory tables can be paged out to the CAS cache via the memory mappings. After all it's better for CAS to run slowly while accessing an infrequently used table than to fail for out-of-memory conditions. However, there are some CAS items for which there is no memory mapping - they must remain in memory for CAS to work. The point here is that CAS_DISK_CACHE is not sufficient for all virtual memory operations - so take care when relying on it.   It is important to realize that CAS_DISK_CACHE is a critical consideration in host machine setup. It must be accounted for in terms of size and performance with real-world hardware when properly sizing a SAS Viya solution. Read more about these considerations in my blog post, Provisioning CAS_DISK_CACHE For SAS Viya.   But virtual memory does not provide performance equivalence with RAM. It's much, much slower. To keep things snappy, it is more important to ensure the CAS hosts have sufficient RAM first and foremost. Prioritize your money on RAM first. CAS_DISK_CACHE offers protection for CAS worker failure When CAS employs COPIES=1 (or greater), then additional block replication of CAS data is performed with duplicates kept in CAS_DISK_CACHE. Hopefully, they're never needed, but if a CAS worker fails, and the users can continue working with uninterrupted access to their in-memory tables, then that's a great feature. Lazy loading of SASHDAT When CAS is pointed at a data source with data in SASHDAT format which is accessible on local disk, then it behaves differently by memory mapping to the source location directly (and so it doesn't use CAS_DISK_CACHE to back those blocks). Building the memory maps is a minor and quick action. So when the loadTable directive is executed, it seems as if CAS loads that SASHDAT table extremely fast - way faster than physically possible. That's because it just builds those memory maps at that time. It's when the data is first accessed by CAS for analytics work that the OS realizes that the data isn't yet in RAM, so it pages in the data from the source disk (meaning the first analytics job must wait longer than any subsequent ones). For really large SASHDAT tables, consider running a preliminary job that will force the OS to page in the data before real end users try to hit it. Default COPIES=1, except when it's 0 By default for most external data sources (i.e. not locally-accessible SASHDAT), CAS uses COPIES=1 to place redundant blocks of data in the CAS_DISK_CACHE to guard against failure of a single CAS worker. But there's one exception to that rule: loading SASHDAT data from remote HDFS. CAS cannot memory map to remote HDFS blocks of SASHDAT, so if CAS loses a worker in this scenario, then the in-memory table will be corrupted, requiring a reload. The assumption here, however, is that remote HDFS will be fast enough to reload the data easily enough that it's worth the tradeoff and avoid using CAS_DISK_CACHE. It is possible to manually specify COPIES=1 or any other preferred value. DNFS and CAS_DISK_CACHE are not the same Both DNFS and CAS_DISK_CACHE can be considered as locally-accessible storage which can work with CAS' use of memory maps to associate data in RAM with locations on storage devices. But their actual implementations will likely be very different.   DNFS is a SAS-term simply meaning shared file system. There are a wide ranges of offerings and options which work perfectly fine for DNFS (assuming their performance meets expectations). For example, these offerings can range from plain old NFS to high-performance clustered file systems. One such CFS is IBM SpectrumScale (a.k.a. GPFS) which is often a go-to technology used for many high-performance SAS solutions.   CAS_DISK_CACHE, on the other hand, will definitely not be any kind of shared file system. It must use local disk(s) which are ideally provisioned exclusively for the CAS host. CAS_DISK_CACHE supports specifying multiple directory paths and so high-performance and high-throughput are possible here, too. But the techniques that CAS uses to work with files in its cache isn't compatible with many shared file system technologies. Never use an NFS-based solution for CAS_DISK_CACHE. And GPFS simply won't work.   Point is, plan and provision storage space for DNFS and CAS_DISK_CACHE separate from each other. They are not interchangeable. This concept also carries over to Viya deployments where CAS is symmetrically colocated alongside Apache HDFS. The challenge there can be ensuring enough disk space is available for CAS_DISK_CACHE (i.e. that it's not all given to HDFS).   For a lot more detail about performance of storage performance, see Tony Brown's paper from SAS Global Forum, Engineering CAS Performance Hardware Network, and Storage Considerations for CAS Servers. Monitoring There are several tools which can provide insights into how CAS is managing its data in memory. For a normal SAS user, this is likely more detail than is necessary. However for Viya administration and other IT personnel, there are details which might come in handy in certain situations.   To get started, take a look at the SAS documentation for Monitoring: How to (CAS Server Monitor).  ... View more