We all know that SAS Cloud Analytics Services (CAS) is an in-memory analytics engine. Ideally, all of the data will reside in RAM on the CAS host(s) which gives it the fastest possible route for processing in the CPU. To improve its enterprise flexibility however, CAS can employ disk-based storage for caching data as well. And that leads to benefits such as improved data availability and a more robust memory utilization scheme.   The space set aside for this purpose is CAS_DISK_CACHE. But the CAS_DISK_CACHE isn't always used when CAS is working with in-memory data. There are circumstances that determine when it's used and when it's not. Most of the time, CAS will use its cache in a way that benefits your process objectives. But occasionally, there's a need to direct CAS to behave differently.   Let's take a look at the situations where you might want to override CAS' default caching behavior and how to accomplish it. Default behaviors The way that the SAS Viya Cloud Analytic Services (CAS) Server manages data in memory is a complicated topic. There are many twists and turns to cover all of the nuances and possibilities. However, most of that behavior can be distilled down to a few simple rules which adequately describe the majority of situations. See 4 Rules to Understand CAS Management of In-Memory Data for a fuller explanation, but for now, let's look at: Rule № 2: All CAS in-memory data is memory mapped to a locally-accessible backing store. Very briefly, this means that the data loaded into CAS is typically backed with a memory-map to local disk storage - and usually, that's CAS_DISK_CACHE. But there's one notable distinction, if you're loading data into CAS which is already in unencrypted SASHDAT format on disk that appears local to the CAS host, then it memory-maps to the source (and not to cache). Why change the behavior? To ensure robust data availability in spite of a hardware failure, CAS normally defines a backing store for all in-memory data. For PATH and DNFS types of caslib, CAS makes efficient use of the existing SASHDAT file as the backing store instead relying on its own cache location. The problem There's one use-case in particular where overriding CAS' default caching behavior can be helpful. When you have multiple, concurrent users of a PATH or DNFS sourced, unencrypted SASHDAT file - and one (or more) of those users need to make changes to the data. This kind of situation often becomes a classic computer science challenge referred to as a race condition. If you've ever tried to edit a file alongside other people on a shared disk, you've experienced a similar race condition yourself.   Let's say you and I both want to work with an unencrypted SASHDAT file in a DNFS caslib. And then we both start our own CAS sessions, load the table, and get to work.   By default, when the first user (you) runs the loadTable action, their CAS session will memory-map to the SASHDAT file's source (not CAS_DISK_CACHE). Then when the second user (me) runs the loadTable action, the second CAS session will use the same memory-map handles. This makes CAS very efficient - it only effectively loads the table into RAM once.   Knock, knock. A race condition. Who's there?   But now imagine that you make changes to the in-memory table, save them back to source, and quit your session. And then I make some different set of changes and write those back to the same source. I'll be the jerk who overwrote your changes and you won't know until later (that's the race condition).   The reality is slightly more gritty than this, though. Turns out in real life that I wouldn't get a chance to save my changes. After you pushed your changes to the source, that then effectively adds (and/or deletes) some of the memory maps CAS was using. Your session knows about those changes (of course), but my CAS session doesn't. And we've seen that when this happens, it can cause my and other CAS sessions to hang or otherwise become unresponsive. That's not the kind of experience we want to provide to our users. The solution Normally if you have a table in CAS for multiple people to see, it should be promoted to the global caslib. Updates can be made and the other consumers will see those as well. But in the scenario I'm describing here, we've got multiple users attempting to update the table at its source.   You should protect your PATH and DNFS sources of unencrypted SASHDAT to only have a single writer and for best results, keep it as read-only for general use. But if you cannot for some reason, then direct CAS to always rely on its cache as the backing store for unencrypted SASDHAT from PATH or DNFS caslibs.   Let's return to our scenario where you and I both have CAS sessions which loaded the same table from source into memory and this time, we've directed CAS to use its cache (not memory mapping to source).   This now means that we both have our own instance of the table in memory. Your instance is backed to the cache with its own memory maps. And so is mine. When you make changes to your in-memory table, it has no effect on mine. That's great! And better, eliminates that nasty hanging of CAS sessions.   But to be clear, we haven't really eliminated the race condition if we both intend to save our final changes back to the same source. To do that, we must coordinate with each other and employ other common data management strategies to be good stewards of the data and system. Make it happen Specific to loading SASHDAT data using either PATH or DNFS type of caslibs, we are able to override CAS' default behavior with its cache through the use of three parameters.   Those parameters rely on two values in particular: INPLACEPREFERRED: Directs the CAS server to follow its default behavior as described by Rule № 2 above.   CASDISKCACHE: Directs the CAS server to always relies on the CAS_DISK_CACHE as a backing store for the in-memory data loaded from unencrypted SASHDAT files using either PATH or DNFS type of caslibs. The three parameters where these values are used employ a hierarchical relationship which you can override depending on the level of specificity desired: entire CAS server > per CAS library > each CAS table. First To globally set the CAS server's behavior, modify the casconfig_usermods.lua file on the CAS Controller host in directory /opt/sas/viya/config/etc/cas/default:   env.CAS_LOADTABLEBACKINGSTORE="INPLACEPREFERRED" | "CASDISKCACHE" Specifies the backing store to use for all SASHDAT tables in PATH and DNFS caslibs on this CAS server.   If not specified, then the default is INPLACEPREFERRED.   For reference, see Controlling Use of CAS_DISK_CACHE in the SAS® Viya® 3.5: System Programming Guide. Second When adding a caslib with a srcType of either PATH or DNFS, provide this source-specific option:   loadTableBackingStore="CASDISKCACHE" | "INHERIT" | "INPLACEPREFERRED" Specifies the backing store to use for all SASHDAT tables in this caslib.   If not specified, then the default is INHERIT which means this caslib will follow the global setting indicated by env.CAS_LOADBACKINGSTORE above.   Here's an example:   caslib mydata sessref=mysess datasource=(srctype="DNFS", path="/path/to/data" loadTableBackingStore="CASDISKCACHE");   For reference, see addCaslib Action in the SAS® Viya® 3.5: System Programming Guide. Third When loading an unencrypted SASHDAT table in a PATH or DNFS type of caslib, provide this importOption:   backingStore="CASDISKCACHE" | "INHERIT" | "INPLACEPREFERRED" Specifies the backing store to use for this SASHDAT table.   If not specified, then the default is INHERIT which means this table load will follow the caslib setting indicated by the loadTableBackingStore caslib option above.   Here's an example:   proc casutil;   load casdata="ImportantData.sashdat"        importOptions=(filetype="HDAT", backingStore="CASDISKCACHE")        casout="ImportantData" ; run;   For reference, see Common Parameter: importOptions in the SAS® Viya® 3.5: System Programming Guide. Other considerations There are a few other considerations to deal with. SASHDAT exceptions There are a couple of exceptions where the value of the backingStore parameter has no effect when a SASHDAT file is loaded: Encrypted SASHDAT: Decryption must take place in memory, and so the unencrypted data is backed to the CAS cache   Filtered data with a where parameter: The subset results of the where processing after the load are placed in the CAS cache. HDFS When the Apache Hadoop Distributed File System is installed symmetrically alongside CAS, then unencrypted SASHDAT is locally accessible and so CAS will memory-map to the source blocks directly. It will not use its cache.   As a matter of fact, specifying the backingStore option for any caslib type other than PATH or DNFS will be silently ignored (i.e. no syntax errors in the log). Block copies When CAS memory-maps to SASHDAT directly at the source, then COPIES=0 (i.e. no replicated blocks for failover are placed in the CAS cache). However, when you specify the CAS cache as the backing store, then the usual default for external data source COPIES=1 will go into effect. This means CAS maps in-memory data to the cache *and* also places duplicate copies of the blocks in cache as well to protect against the unexpected loss of a worker. Coda Designing your system and processes for multi-user concurrent access to tables which change requires planning, effort, and communication. CAS offers several options for dealing with this in how it operates, the procedures for working with tables, and configuration parameters.   H/T Thanks to Andy Bouts, Principal Pre-Sales Solutions Architect for sharing his hard-won experience and insights with these use-cases in the field. ... View more

SAS Viya 3.5 introduced a new approach for defining network configuration for SAS host machines. The new approach improves the consistency of how networking is applied which in turn leads to substantial benefits including more flexibility for different network topologies and better resiliency in the face of network changes. See my previous post, Understanding Network Binding Variables in Viya 3.5, for more details.   The idea is that your Viya host(s) will often have more than one network interface card (NIC). Depending on how your customer IT has configured their use, those cards could be used to separate traffic into and out of the machine, essentially providing a dedicated i/o path for traffic that's high priority, requires low latency, needs to be isolated from other networks, or is of large volume to prevent it from competing with general communication of other activity.   As it turns out, SAS Cloud Analytic Services (CAS) often fits the profile of network usage for all of those reasons. Let's look at configuring CAS to optimize its communication with other services and its movement of data. Directing CAS with new configuration By default, CAS relies on the same set of network configuration variables defined by the new network.conf technique (i.e. SAS_BIND_ADDR, SAS_EXTERNAL_BIND_ADDR, and related). Often these are sufficient for nominal usage. But sometimes, it might be necessary to split out data traffic because it can be so voluminous that it saturates the network, preventing timely response to other requests and activity. CAS offers some additional parameters to make it happen.   There are a few files to be familiar with when making changes to the configuration. vars.yml The vars.yml file is provided in the SAS Viya Playbook which is placed on the Ansible Controller host machine. It's referenced when running the site.yml file which performs the deployment of Viya software. The parameters discussed in this post can all be defined in the CAS_CONFIGURATION section of vars.yml.   It is considered a proven practice to keep vars.yml up-to-date with configuration changes after deployment as well.   Changes made to vars.yml require re-running the site.yml deployment for them to go into effect - which means a full Viya outage while it runs.   There are times when you might need to make targeted configuration changes. And so there are two more files to consider to help with that: casconfig_usermods.lua This file exists on the CAS controller host and is read by the controller at start up. The values it finds here override the standard configuration files and are then shared identically with all workers.   So if you have a universal parameter that works for all CAS nodes, then place it here.   Changes made to this file require a restart of CAS to go into effect. The rest of Viya is unaffected. node_usermods.lua Each CAS node has an instance of this file that it reads at start up. These values override anything provided by the CAS controller (as specified in casconfig_usermods.lua).   So if you have a parameter with a value unique to this CAS host, then place it here.   Changes made to this file require a restart of CAS to go into effect. The rest of Viya is unaffected. Selecting network identity At deployment, CAS is configured by the Viya 3.5 network configuration. The hostnames and IP addresses it references are originally driven by the SAS_HOSTNAME, SAS_BIND_ADDR and other parameters as well as their _EXTERNAL_ equivalents for public interfaces. So, for example, the cas.hosts file should populate with hostnames derived from the Viya 3.5 network config.   There are situations when the hostnames and/or IP addresses that CAS uses to reference its own nodes are not compatible with external data providers which might reside on a different subnet (or otherwise find CAS' hostnames and IP addresses unroutable). So CAS provides configuration parameters to override those default values. cas.dcHostnameResolution="<ep | ep_fqdn | cas | cas_ipv6>" This parameter directs which software service will resolve hostnames into IP addresses for the SAS Embedded Process to use.   Default is "cas". This directs the CAS controller to resolve its nodes' names into IPv4 addresses and send the list over to the EP when requesting data. The EP will attempt to connect to those IP addresses directly. The "cas_ipv6" value works the same, but the CAS controller sends over IPv6 addresses for its nodes instead.   However, if the EP is in a subnet where those IP addresses aren't routable to CAS, then we can direct CAS to send over hostnames as listed in its cas.hosts file for the workers with cas.dcHostnameResolution="ep". Or specify "ep_fqdn" which directs CAS to send its workers' fully-qualified domain names over to the EP.   By using one of the the "ep" values, it might mean that the names the EP resolves point to a different set of IP addresses than CAS knows about. Same name, but different IP addresses is somewhat unusual and assumes separate DNS, virtualized networking, local hostname resolution files, or some other extra bit of mapping outside of Viya.   To be clear, it's certainly possible that the hostname/FQDN resolution on the EP would yield the exact same set of IP addresses that CAS would provide when cas.dcHostnameResolution="cas". And if so, that's fine, too. cas.extHostKnownBy="<cas-node-id.site.com>" This parameter provides an additional FQDN for the CAS host. If needed, be sure to specify this parameter in node_usermods.lua for each CAS host.   Default is unspecified. This means that CAS relies on the values from its cas.hosts file to advertise its hosts names, FQDN, and/or IP addresses.   Use this parameter when you want to provide a substitute name for the CAS controller to provide for this host, instead of how it's listed in cas.hosts file, when it sends the list of hosts over to the external data provider. This allows the external data provider to resolve the hostname/FQDN to an IP address using its own local lookup service (DNS, /etc/hosts file, or similar). Selecting network ports Within a single machine, network services each listen at their own port. Some services, like CAS, can listen on several ports simultaneously for different tasks. Depending on what communication needs further tweaking, CAS offers a selection of port configurations to help ensure traffic is routed the best way. The parameters shown here can be defined at initial install in the CAS_CONFIGURATION section of vars.yml or after install in either casconfig_usermods.lua or node_usermods.lua files, depending on specificity. cas.port="<0 - 65535>" Default is 5570 (TLS encrypted). 0 requests an ephemeral port assignment from the OS.   This is the port that the CAS controller listens to for client requests. No CAS workers use this value. Avoid using a value of zero for this parameter as this port number is one that you need to know in order to advertise it to non-Viya clients. cas.httpport="<0 - 65535>" Default is 8777 (TLS encrypted). 0 requests an ephemeral port assignment from the OS.   Similar to cas.port, this is another port that the CAS controller listens to for client requests over HTTP. No CAS workers use this value. Avoid using a value of zero for this parameter as this port number is one that you need to know in order to advertise it to non-Viya clients. cas.gcPort="<0 - 65535>" Default is 0. 0 requests an ephemeral port assignment from the OS with the resulting assignment in the range of 32678–61000. If set explicitly, the typical value used is 5580.   The gcPort is CAS' network channel for general communications between the CAS hosts. It's what the controller(s) and workers use to talk to each other. cas.dcPort="<-1,0 - 65535>" Default is -1 which means use the same value as cas.gcPort. 0 requests an ephemeral port assignment from the OS.   This is the data communication port. Specifying a separate port for data transfer i/o from external data providers can improve performance by reducing contention for a single port (i.e. the gcPort). cas.dcExtPort="<0 - 65535>" Default is 0. Unlike the other port parameters, zero doesn't request an ephemeral port, instead it directs CAS to specify the same value as dcPort (similar to how dcPort="-1" directs CAS to use the gcPort for data traffic).   Note that this directive doesn't define a new port for CAS to listen on. This is the port number that CAS will inform an external data provider, like the EP, to use when it initiates data transfer communication to the CAS workers. This is useful for situations when the dcPort isn't accessible from the outside - like when CAS is running in a containerized environment. The dcExtPort value will direct CAS to advertise the mapped port instead. That's why there's no ephemeral option. For non-containerized environments, expect to leave this with default value.   Summarizing CAS port parameters: Parameter Purpose Applies to Default value Value's meaning Encryption cas.port Client communication Controller only 5570   Yes by default cas.httpPort Client communication Controller only 8777   Yes by default cas.gcPort General intra-node communication Any CAS node 0 Use ephemeral port Optional cas.dcPort Data connector communication Any CAS node -1 Use gcPort Optional cas.dcExtPort Data connector communication Any CAS node 0 Advertise dcPort's value to external data providers Optional Scenarios Okay, I've explained the documentation and theory a bit. Let's look at some common scenarios and how to deal with them. Default Without specifying any new values to the parameters described above, then you can expect an MPP deployment of CAS will behave as: CAS controller listens for TLS encrypted, native client traffic on TCP port 5570. CAS controller listens for TLS encrypted, HTTP RESTful traffic on TCP port 8777. All CAS intra-node traffic is not encrypted - between controller(s) and workers - on ephemeral ports (cas.gcPort) designated when CAS processes start up. As a point of interest, CAS intra-node traffic includes: Controller directing workers on tasks Controller distributing data to workers from a data load Workers sharing data between them (rare) Workers returning their interim results to the controller And so on. Encryption of CAS intra-node traffic (i.e. cas.gcPort) can be enabled - be sure to allow for an accompanying slowdown of throughput performance - with the env.CAS_Internode_Data_SSL parameter. Significant data transfer to/from data sources Change cas.dcPort in casconfig_usermods.lua and/or node_usermods.lua.   CAS is a powerful, multi-threaded runtime engine which can support many concurrent users, sessions, and operations. Data movement of large volumes can consume considerable network resources and potentially slow other communications traffic between the CAS hosts.   For an environment where data transfer from external sources is a substantial (or high latency) workload, consider setting the cas.dcPort value to a value other than -1 to split the communication from outside data providers (through the SAS data connectors) to a different port interface.   Encryption of cas.dcPort traffic can be enabled if needed using available caslib options. Separate data transfer from data sources to a different I/O channel Change cas.dcExtHostname in node_usermods.lua.   Assuming your CAS host(s) has more than one NIC, then we can configure CAS to direct an external data provider (like the EP) to send data over a different network interface than CAS uses for its normal, intra-node communication. This further separates traffic onto a different physical path for improved throughput performance.   When CAS for Viya 3.5 is deployed, the cas.hosts file is built using values derived from SAS_HOSTNAME and SAS_BIND_ADDR in the new network configuration. And those specify the normal route (that is, which NIC card) that CAS will use for its general communications.   For CAS hosts with more than one NIC, consider setting cas.dcExtHostname to a value which resolves to specify the CAS host's alternate NIC. Take care with this approach to ensure that the alternate NIC is intended for this use (not an admin-only interface, on the correct subnet, speed-rated sufficiently high to feed CAS with large data volumes, etc.) SAS Embedded Process on a different subnet For SAS Data Connector Accelerators, change cas.dcHostnameResolution in casconfig_usermods.lua.   It's fairly common for the SAS In-Database Embedded Process to reside in a data provider (Hadoop, Teradata) placed on a subnet which isn't directly routable to the IP addresses used by CAS.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   When a load action of data from the EP is executed, the default behavior of CAS is to send the EP a list of the workers' IP addresses. Then each instance of the EP distributes its allotment of data across all of the CAS workers at those IP addresses. If those IP addresses aren't routable, then modify the cas.dcHostnameResolution parameter to "ep" or "ep_fqdn". This causes CAS to send a list of hostnames (or their equivalent FQDN) as referenced by the cas.hosts file over to the EP instead of IP addresses. Then the EP itself can resolve them to IP addresses which would routable from that end.   This approach necessitates some additional OS or DNS network configuration by your customer's IT team, similar in concept to dedicated /etc/hosts files (or accomplished in DNS or other way).     Encryption of traffic between CAS and the EP can be enabled with the cas.dcTCPMencrypt parameter on both CAS and EP hosts. CAS running in containers Viya 2020 is due to be released a little later this year. And as we all know, it'll employ a wholly new approach to deployment, running inside Docker containers and orchestrated by Kubernetes. In Viya 3.5, it is also possible to employ a containerized approach to running CAS.   One (of many) key things to understand about containers is that, by default, they are completely isolated from the outside world, even their host OS. So when you reference things like network ports inside a container, they only exist there, and aren't directly accessible to the outside world. Of course, that's not very useful in most cases, so you can define mappings to resources likes network ports to make them accessible.   As we established earlier, the CAS controller typically listens for HTTP-based requests on port 8777. If we place CAS inside a container, then we must define a port mapping to allow requests from outside the container to reach that port. The simplest approach is to define a mapping where Docker listens on the host OS port 8777 and forwards those to CAS listening on 8777 inside the container.     But what if there's something else in the host OS already using port 8777? Well, we can define Docker to listen at some other port number, like 7888 (or whatever) and anything sent there will forward to CAS' port 8777 inside its container. In that case, CAS does not know about port 7888 - as that's outside its containerized world.     For data from external providers: Specify static port values for cas.gcPort and cas.dcPort If needed, specify cas.dcExtPort and cas.dcExtHostname in node_usermods.lua. Now change gears to consider cas.gcPort (default is ephemeral) and cas.dcPort (default is to use gcPort). So when CAS informs external data providers about the dcPort it's listening on for data connections, it'll advertise the current port number inside its container, but that will only work if a port mapping exists where Docker is listening for that traffic on the exact same port number in the host OS.   In containerized environment, specify a static (not ephemeral) value for cas.gcPort and cas.dcPort to use for the Docker port mappings. And then, if needed, specify cas.dcExtPort to tell CAS what that external port mapping value is that it should advertise to an external data provider when sending data.   A similar concept applies to CAS' hostname values inside the container as well. If they differ from what an external data provider would use to route traffic, then specify cas.dcExtHostname on each CAS node to inform CAS of the external hostname (or IP address) that should be advertised outside of the containerized environment.     Remember also that cas.dcHostnameResolution="ep" causes CAS to send over hostnames based on the contents of the cas.hosts file for the EP to resolve into IP addresses. When cas.dcExtHostname is provided for a CAS node, then that value will override the entry from the cas.hosts file which gets sent to the EP. Coda As you can see, the combination of CAS roles, various ports to listen for different kinds of traffic, deployment topologies, and virtualization technologies requires numerous configuration parameters to ensure everything gets to where it needs to go. Furthermore, the technique CAS relies on when transferring data from the SAS In-Database Embedded Process differs significantly from most typical client-server interactions... and this necessitates additional considerations as well.   For more information about the related topics in this article, refer to: Raphaël Poumarede's post, Working with multi-homed CAS clusters and the Embedded Process Erwan Granger's many GEL posts about SAS Viya running in containers SAS documentation: SAS® 9.4 and SAS® Viya® 3.5 Programming Documentation > setServOpt Action SAS® Viya® 3.5 Administration: SAS® Cloud Analytic Services > Configuration File Options Encryption in SAS® Viya® 3.5: Data in Motion > Configure CAS Internode TLS SAS Support Site > SAS® for Containers Focus Area   ... View more

In this SAS demo, @RobCollum of SAS walks through the procedure and considerations for scaling out the SAS Cloud Analytic Services in SAS Viya 3.5 by adding more worker hosts without interrupting ongoing system operations. He explains the impact that this change means for an existing production environment and the benefits to users and administrators of the system.     If YouTube is blocked at your site use this link   @RobCollum has been at SAS for over 20 years and focuses on high-performance architecture of SAS software.   Watch more videos like this:  SAS Technical Insights & Expertise Series   ... View more

SAS Cloud Analytic Services is - first and foremost - a high-performance in-memory analytics engine. In the ideal, when operating with everything it needs in memory, CAS operates as fast as the system's electrons can pass through it. However, we don't often enjoy the ideal work environment. SAS Viya and CAS must often support uncoordinated workloads as a mix of users perform ad hoc tasks of varying complexity on data of undefined sizes. In such scenarios, having a safety mechanism to fall back on when RAM alone isn't sufficient can mean the difference between a getting the job done versus a crashed system. That's where the CAS_DISK_CACHE comes in…   CAS_DISK_CACHE is disk storage space primarily relied upon to act as part of a virtual memory scheme for CAS tables. It has other uses as well, such as block replication to ensure data availability if a CAS worker goes offline. Effectively, it acts as a buffer to relieve pressure on RAM and improve CAS's resiliency and availability in dealing with unanticipated short-term challenges.   Because CAS_DISK_CACHE is an important part of CAS operations, provision it adequately and then monitor and maintain it over time. And in Viya 3.5, we have a new tool available to help with monitoring: the builtins.getCacheInfo action.   SAS Program Code The builtins.getCacheInfo action is, as the name implies, built into CAS as of Viya 3.5. You don't need to define or load it. It's already there, just waiting for you to use it. So, let's see it in action: /* if needed */ *options cashost="cashost.sas.com" casport=5570 authinfo="/path/to/.authinfo";  *title2 "<cas server instance> -- cashost.sas.com:5570"; cas sess1; proc cas; session sess1; accessControl.assumeRole / adminRole="superuser"; builtins.getCacheInfo result=results; describe results; run; print results.diskCacheInfo; run; quit; cas sess1 terminate;   This code example shows how to call builtins.getCacheInfo : If needed, define which host the CAS controller is running on, the port it's listening at, and the location of an .authinfo file with administrator-level credentials that have the ability to assume the superuser role.   Connect to a new CAS session.   Run  PROC CAS , assuming the superuser privilege, invoking the builtins.getCacheInfo action, and capturing the resulting data.   Print out the results   The results that are returned in the SAS listing output will look similar to: diskCacheInfo: Results from builtins.getCacheInfo Node NodePath Capacity Free_Mem %_Used cashost.sas.com /cas/cctemp01 100 Gb 86 Gb 13.9 cashost.sas.com /cas/cctemp02 100 Gb 86 Gb 13.9 casworker01.sas.com /cas/cache01 191 Gb 161 Gb 15.6 casworker01.sas.com /cas/cache02 191 Gb 161 Gb 15.6 casworker01.sas.com /cas/cache03 191 Gb 161 Gb 15.6 casworker02.sas.com /cas/cache01 191 Gb 162 Gb 15.2 casworker02.sas.com /cas/cache02 191 Gb 162 Gb 15.2 casworker02.sas.com /cas/cache03 191 Gb 162 Gb 15.2   Of immediate note: The disk space usage of CAS_CONTOLLER_TEMP is also reported. It only applies to the MPP CAS Controller host (here, that's cashost.sas.com).   CAS_DISK_CACHE (and CAS_CONTROLLER_TEMP) can be defined to include multiple directory paths.   Results show the incorrect units. A lowercase "b" would indicate "bits", but these numbers are indeed for "bytes" which should be shown as an uppercase "B". That is, GB not Gb. You can confirm this for yourself by querying the OS directly. This is a known issue with a scheduled fix.  The usage numbers are for the entire file system(s) where CAS_DISK_CACHE is located - not only the data blocks actually being stored there by CAS unless the file system(s) is dedicated exclusively to CAS cache (which is considered the ideal approach).  The results are arranged by CAS Controller's CAS_CONTROLLER_TEMP space followed by the CAS Workers' CAS_DISK_CACHE space as defined per directory. The %_Used column is what you're most likely to want to monitor closely as an indicator if the cache space is becoming overloaded. I recommend watching very closely at 70% or more. But keep in mind that depending on the available space, you might just be one large table load away from filling it all the way.   Note that the builtins.getCacheInfo action makes no assumptions about any underlying file systems or storage arrangements. It's up to you bring that intimate underlying physical system knowledge to bear in reading these results. For example, my test environment is very basic and very cheap. It does not provide high-performance storage with multiple redundant i/o channels to an array of many physical disks striped to act as a single logical volume. Instead I just created some simple subdirectories. And so the cache space on each of the CAS workers is just 191 GB total - not 573 GB (=3 × 191 GB). That is, casworker01 has 191 GB of file system space where the CAS_DISK_CACHE is specified to use three sub-directories. And the same is true for casworker02. So then the workers of my small MPP CAS Server have a combined space of 382 GB for CAS_DISK_CACHE.   Let's consider what this might look like for a real, production environment. To that end, imagine each CAS Worker has 16 physical CPU cores (not virtual cores or hyper threading). Depending on the storage technology available, then there might be 16 physical disks (1 per CPU core) for CAS_DISK_CACHE to achieve good performance, with each disk formatted as a single file system (xfs or ext4) and directly attached to the host (not via NFS, NAS, or SAN). And then the paths to each of those listed in env.CAS_DISK_CACHE . In that scenario, we'd expect to see 16 different lines for each CAS worker's cache in the report. And because each one would represent a unique file system space, then you could add them all up together (unlike my very basic illustration).   But wait, we can get really hardcore about this. For a high number of concurrent users in SAS Visual Analytics (100's or more), then there is evidence that defining two sub-directories per vCPU can provide some benefit as well (i.e. 4 sub-directories per disk/filesystem for CAS_DISK_CACHE). In that case, each CAS worker would have 64 directories for its cache. The total cache space wouldn't have changed from the 16 first imagined, but you'd have to take care in reading the report results to understand that.   SPECIAL NOTE: Instead of pointing CAS_DISK_CACHE to physical disks, there is an alternative: /dev/shm . That's a shared memory space utility provided by Linux systems - effectively placing the contents of CAS_DISK_CACHE in RAM.   The key benefit to this approach is that CAS_DISK_CACHE can perform at RAM speeds - something you're unlikely to approximate with any physical disk-based solution. The drawback to this approach is that you might consume RAM capacity much faster compared to using physical disk if COPIES=1 (the default) or more. The idea here is that replicating blocks will ensure table availability in case one or more CAS workers goes offline. You could set COPIES=0 to prevent that replication, but then you're disabling a significant enterprise feature of CAS.   So watch out - if you specify env.CAS_DISK_CACHE=/dev/shm and inadvertently overcommit the RAM capacity, then the OS will revert to its system paging file and everything on the host will slow e x c e s s i v e l y.   My recommendation is to use this approach carefully. It's ideal for well-behaved systems which consistently run within well-known parameters so that they can perform at the fastest speed possible. But for systems where the activities are mostly ad-hoc and/or require CAS' availability features, then look to physical disk for CAS_DISK_CACHE instead.   pyViyaTools Viya supports having more than just one CAS server in the environment. You can have multiple CAS servers of different sizes, SMP or MPP, dedicated to specific tasks/users, etc. if needed. So then you might want to create multiple copies of that SAS program to get the cache statistics for each one. You could simply copy & paste the code over and over, changing the parameters to specify each CAS server. Or you could write some SAS Macro code to read from a parameter list, then generate and submit the code for each automatically.   But I thought it'd be nice to have an approach which can get those cache statistics for all CAS servers … and for it do so automatically without having to enter any manual parameter information.   My GEL colleague Gerry Nelson introduced the GEL pyViyaTools to leverage Viya's REST APIs and extend them to glean new information to help with system monitoring and management. So I looked at the utilities already there, opened up a couple program files to get familiar with their coding practices, and then wrote my own pyViyaTool called listallcasservercachestatus.py .   The code to do this in Python follows many of the same steps as in SAS: # Other preliminary tasks not shown here # Identifies CAS servers and loops to connect to each # connect to each CAS server s = swat.CAS(serverhost, serverport) # get CAS_DISK_CACHE usage s.accessControl.assumeRole(adminRole="superuser") # superuser role reqd results = s.builtins.getCacheInfo # returns dictionary, table # display table with CAS_DISK_CACHE usage stats print(tabulate(results,headers='firstrow'))   When you run listallcasservercachestatus.py , you get familiar looking results: [cloud-user@viyahost pyviyatools]$ ./loginviauthinfo.py Logging in with profile: rocoll Enter credentials for https://viyahost.sas.com: Login succeeded. Token saved. [cloud-user@viyahost pyviyatools]$ ./listallcasservercachestatus.py server,host-or-ip,port,restPort cas-shared-default,cashost.sas.com,5570,8777 (u'diskCacheInfo', Result table containing CAS_DISK_CACHE information                    node          path   FS_size   FS_free FS_usage 0   cashost.sas.com  /cas/cache01    191 Gb    144 Gb     24.5 1   cashost.sas.com  /cas/cache02    191 Gb    144 Gb     24.5 2   cashost.sas.com  /cas/cache03    191 Gb    144 Gb     24.5 3   casworker01.sas.com  /cas/cache01    191 Gb    160 Gb     15.9 4   casworker01.sas.com  /cas/cache02    191 Gb    160 Gb     15.9 5   casworker01.sas.com  /cas/cache03    191 Gb    160 Gb     15.9 6  casworker02.sas.com  /cas/cache01    191 Gb    161 Gb     15.6 7  casworker02.sas.com  /cas/cache02    191 Gb    161 Gb     15.6 8   casworker02.sas.com  /cas/cache03    191 Gb    161 Gb     15.6 cas-shared-2NDSERVER,cas2nd.sas.com,5570,8777 (u'diskCacheInfo', Result table containing CAS_DISK_CACHE information                    node          path   FS_size   FS_free FS_usage 0   cas2nd.sas.com  /cas/cache01    101 Gb    94 Gb     6.9 1   cas2nd.sas.com  /cas/cache02    101 Gb    94 Gb     6.9 2   cas2worker01.sas.com  /cas/cache01    101 Gb    89 Gb     11.9 3   cas2worker01.sas.com  /cas/cache02    101 Gb    89 Gb     11.9 4  cas2worker02.sas.com  /cas/cache01    101 Gb    91 Gb     10.0 5  cas2worker02.sas.com  /cas/cache02    101 Gb    91 Gb     10.0   Notice now that there are two different CAS servers listed, each with their own set of CAS_DISK_CACHE utilization statistics. That's because my Viya deployment has a second CAS server that I installed and the listallcasservercachestatus.py utility is coded get the list of CAS servers from the Viya REST API and then it loops through connecting to all of them to get their cache usage statistics.   As an aside, I think it's interesting that the Python-based code yields slightly different formatting than the SAS-based invocation of the builtins.getCacheInfo action, showing an obs count (but weirdly zero-based) and different column labels.   Running the builtins.getCacheInfo action In the SAS program code sample shown above, I was able to run the builtins.getCacheInfo action after successfully establishing a user session on the CAS server. Depending on your CAS client, you might need to establish the antecedent authentication and server identification parameters yourself (e.g. from batch code submission) or it might already be done for you (e.g. submitting code interactively from SAS Studio). So, from a SAS programmer's perspective, this is well understood and the example given should be sufficient to get you going.   Running the pyViyaTools, however, takes just a little more effort and consideration.   First of all, read the INSTALL.md file for instructions on how to clone (i.e. download) the pyViyaTools files to your Viya installation followed by the steps required to authenticate to Viya and get a login token. This is a crucial step so don't move on until you're successful.   At this point, you might be able to successfully run the listallcasservercachestatus.py tool. Or you might not. It depends on how your environment was setup and the topology of the Viya software deployment.   So if you can't, I've got a couple of troubleshooting tips to help:   If you see a complaint about a package not being available, then use pip to install it. In particular, the listallcasservercachestatus.py tool relies on "tabulate" and "swat" packages for Python which are not included by default: pip install tabulate pip install swat Another failure you might see depending on your Viya deployment topology is an error about being unable to connect to the CAS controller host, specifically stating, " ERROR: SSL Error: Missing CA trust list ". To correct this, there's a third environment variable you should define (similar to what was necessary for loginviauthinfo.py ): export CAS_CLIENT_SSL_CA_LIST= /opt/sas/spre/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem # See SAS documentation for more information.   The pip install steps should only be necessary once. But the environment variables and acquiring a login token are steps that you should expect to perform pretty regularly when working with the pyViyaTools, so consider placing those in a shell script to run all together.   Under the covers If you've administered UNIX or Linux environments before, then the output from the builtins.getCacheInfo action might look familiar to you. It's very similar to the output provided from the df command-line utility. That's because they both use the statvfs() system call to get their information.   The point to this is just to say that builtins.getCacheInfo is returning the statistics for the entire file system where the CAS_DISK_CACHE is located ... not only the files that CAS has cached in its own sub-directories. This is a Very Good Thing because it means you're provided with information about the file system's total capacity and utilization, which is the significant bit to monitor.   Wrapping up Keeping CAS well maintained and humming along is vital to ensuring happy user experiences with Viya. One task in doing so is ensuring that the file system(s) hosting CAS_DISK_CACHE are adequate in size (as well as I/O throughput). In addition to OS utilities, SAS provides internal tools, like the builtins action set and the getCacheInfo action, to aid in reporting on that space. And we can extend on that using the pyViyaTools to automate that monitoring task across multiple CAS servers.   For more information about the related topics in this blog, refer to: 4 Rules to Understand CAS Management of In-Memory Data by Rob Collum A New Path for CAS_DISK_CACHE by Rob Collum Introducing the GEL pyviyatools by Gerry Nelson And SAS documentation: SAS Viya 3.5 System Programming Guide > List Cache Information for a CAS Session (CASL)   ... View more

SAS Viya 3.5 introduces an entirely new approach to configuring network communication. There are several new configuration directives to become familiar with. One part to achieving that is understanding how the various terms all relate to one another. In particular, the configuration directives that identify which network bindings the Viya software should use have an interesting chain of dependencies and assumptions.   This post will dive into the details specific to the BIND parameters, explaining how they relate one to another, and offering suggestions on configuration for different deployment scenarios. Where The network configurations for SAS Viya 3.5 are presented in two places: On the Ansible controller in: .../sas_viya_playbook/host_vars/<deploytarget>.yml This file is processed at install time when the site.yml play is run. Specifying these values is optional for single NIC hosts and required for multi-NIC hosts. Each deploy target defined in the Ansible inventory file of the sas_viya_playbook should have its own YAML file here, if needed.   On the Ansible Controller host, you manually create the network configuration files for each deploy target in the host_vars subdirectory. Here's one of mine from a recent installation for a deploy target with hostname alias "sasviya01":   $ cat /path/to/sas_viya_playbook/host_vars/sasviya01.yml --- network_conf: SAS_HOSTNAME: sasviya01.race.sas.com SAS_BIND_ADDR_IF: "eth0"   On each Viya host in: /opt/sas/viya/config/network.conf This file is created on each host of the Viya deployment with the values provided by associated the host_vars file. And this file is read each time Viya services start up on this host.   Shown here is the file created for my "sasviya01" deploy target on its host machine:   $ cat /opt/sas/viya/config/network.conf # BEGIN Ansible managed network configuration options export SAS_HOSTNAME="sasviya01.race.sas.com" export SAS_BIND_ADDR_IF="eth0" Notice that the files in the host_vars subdirectory are provided in YAML format, but that the resulting network.conf files on each Viya host are in shell syntax. What One of the key objectives of revamping how network configuration works in Viya 3.5 is to provide a set of reference variables that all of the Viya software services can rely on identically.   Of those, the first two you should know about are: SAS_HOSTNAME The hostname used to advertise how Viya services on this machine should be identified and contacted. It should resolve in DNS (or elsewhere) to the same IP address as SAS_BIND_ADDR. If not specified, default value is set equivalent to calling "hostname --fqdn". SAS_BIND_ADDR The IP address used for identifying and contacting Viya services on this machine. If not specified, default value is 0.0.0.0 (meaning listen on all network interfaces). Notice that both of these have self-determining default values. So specifying these configuration directives is optional for host machines with a single network interface. If your machine has more than one network interface (a.k.a. multiple NICs or multi-NIC), then the SAS Configuration Server requires you to identify which network interface it should bind to for communication (it will not allow 0.0.0.0 as an acceptable value).   Making the choice and specifying a particular network interface has consequences to consider. The values of those two configuration parameters are referred to by almost all Viya 3.5 services (or they should be) through internal SAS function calls. This means that your choice of SAS_BIND_ADDR will affect not just SAS Configuration Server, but the rest of the Viya software, too - everything will listen on that one specified network interface, unless…   …you also set these two configuration parameters as well: SAS_EXTERNAL_HOSTNAME The hostname used to advertise how public Viya services on this machine should be identified and contacted. It should resolve in DNS (or elsewhere) to the same IP address as SAS_EXTERNAL_BIND_ADDR. If not specified, default value is set to SAS_HOSTNAME. SAS_EXTERNAL_BIND_ADDR The IP address used for identifying and contacting public Viya services on this machine. If not specified, default value is set to SAS_BIND_ADDR. Be careful with your interpretation of "external" here. It doesn't mean "alternative" or "outside". It's only used by a small, select set of Viya services which present public-facing interfaces. Think of this in comparison to the majority of Viya services communication which is private - not directly shared with any end-user client. For example, the CAS Controller provides a public interface which listens for end-user clients on port 5570 (by default) and is bound to the network specified by SAS_EXTERNAL_BIND_ADDR, but it communicates with the CAS workers extensively through a different internal set of interfaces which are configured by SAS_BIND_ADDR instead.   There's a secondary tier of configuration values which can be specified. These directives are used to deterministically derive and set the value for SAS_BIND_ADDR based on dynamic criteria of your network infrastructure: SAS_BIND_ADDR_IF The name of the network interface that Viya services should bind to. If provided, SAS_BIND_ADDR will be set to the first primary IP address found for that network. If SAS_BIND_ADDR is also defined with a value, then this directive is ignored. SAS_BIND_ADDR_CIDR The range of IP addresses to choose from in CIDR notation. If provided, SAS_BIND_ADDR will be set to the first primary IP address found in that range. If SAS_BIND_ADDR or SAS_BIND_ADDR_IF is also defined with a value, then this directive is ignored. Notice in terms of precedence that specifying a value for SAS_BIND_ADDR overrides SAS_BIND_ADDR_IF and that SAS_BIND_ADDR_IF will override SAS_BIND_ADDR_CIDR due to specificity. My recommendation is only to set the directive you intend to use to avoid any confusion.   These same secondary parameters are also available for _EXTERNAL_ as well. How To show this all as a simple set of rules, take a look at the following pseudocode to see how Viya services will determine which hostname and IP to use in their communications and registration:   For Viya to determine the SAS_HOSTNAME value: if SAS_HOSTNAME is set in network.conf, then use it else use `hostname --fqdn` For Viya to determine the SAS_EXTERNAL_HOSTNAME value: if SAS_EXTERNAL_HOSTNAME is set in network.conf, then use it else use SAS_HOSTNAME value For Viya to determine the SAS_BIND_ADDR value: if SAS_BIND_ADDR is set in network.conf, then use it else if SAS_BIND_ADDR_IF is set in network.conf, then use it else if SAS_BIND_ADDR_CIDR is set in network.conf, then use it else use 0.0.0.0 For Viya to determine the SAS_EXTERNAL_BIND_ADDR value: if SAS_EXTERNAL_BIND_ADDR is set in network.conf, then use it else if SAS_EXTERNAL_ BIND_ADDR_IF is set in network.conf, then use it else if SAS_EXTERNAL_BIND_ADDR_CIDR is set in network.conf, then use it else use SAS_BIND_ADDR value Why There really is no one right way to configure the network that will work for all circumstances. You must evaluate the requirements of a given environment and adjust. With that out of the way, I usually start off with the following and then modify my approach as needed: Avoid setting an explicit value for SAS_BIND_ADDR Defining SAS_BIND_ADDR directly will lock Viya down to static IP address assignment. While that's consistent with the current system requirements, we're optimistic that the new network.conf capabilities will soon allow support of dynamic IP address assignment. Use SAS_BIND_ADDR_IF instead This directive will cause Viya to look up the specified network interface name and automatically assign SAS_BIND_ADDR to the first primary IP address it finds there. Your host can still have static IP address assignment. But if/when Viya supports dynamic IP address assignment, then you've no changes to make. Consider SAS_EXTERNAL_BIND_ADDR (and _IF and _CIDR) If SAS_BIND_ADDR resolves to a subnet that the Viya end users do not have access to, then ensure that the public interfaces (like CAS port 5570) are reachable by specifying a network interface on this host which is accessible to end users. Pro-tip: consider SAS_EXTERNAL_BIND_ADDR=0.0.0.0 to direct the Viya public services to listen on all of this host's network interfaces if that makes sense for your environment. I could go on and on. As you can tell, I'm excited by the new possibilities afforded to this huge upgrade to network identification in Viya. I hope this post helps guide your understanding how the network.conf file is processed when Viya services are installed and started. More Refer to the the SAS documentation for more information about networking configuration in Viya 3.5:  SAS Viya 3.5 for Linux: Deployment Guide > (Optional) Configure Network Settings ... View more

SAS Workload Orchestrator is the new grid provider that ships with SAS Grid Manager 9.4 M6. It presents HTTP RESTful interfaces for ongoing operations and administration. Out of the box, the configuration of this HTTP traffic is somewhat rudimentary, with only plain-text communication (i.e. no encryption) and basic redirection provided from the master candidate hosts to the active master (i.e. limited automatic client discovery of SWO master failover).   Out of the box When SAS Workload Orchestrator is installed and configured by the SAS Deployment Wizard, the minimum failover capability for the discovery of the current SWO master is enabled. Default, out-of-the-box SAS Workload Orchestrator Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page. In a SAS Grid Manager deployment, there are basically two kinds of grid server hosts: Grid master candidates Grid worker nodes Grid worker nodes only perform the work they're assigned. A grid master directs all work on the grid - and it can also perform work as well, if desired. If there's only one host capable of acting as the grid master, then that's a single point of failure - which is antithetical to what we want from a grid. So for SAS Grid Manager, we can identify multiple hosts which have the ability to take on the role of the grid master - and they're called master candidates. The master candidates always know which host among them is currently acting as the grid master. If an HTTP request for the current grid master is inadvertently sent to a grid master candidate, then the candidate will respond with a 300-level HTTP code which re-directs the client to make the request of the current master. This is a nifty trick, but it's not really enough to ensure high availability. The client must know which host(s) are master candidates. If there's a hardware failure, then the SWO master candidate on that host cannot reply with an HTTP 300-level code (or anything else) because it's offline.   Provide automatic client discovery of SWO master failover Instead of requiring the grid client to keep track of all SWO master candidate hosts and then possibly attempt to contact them each one-by-one in a failure situation until it finds the current master, we can configure the SAS Web Server to act as a proxy. We will configure it to keep track of everything automatically, and then just use it as a single point of contact.   SAS Workload Orchestrator with automatic client discovery   For details on this, see my post: New Considerations for SAS Grid Manager 9.4 M6   Provide TLS encryption of SWO communication over HTTP   Have you noticed that so far all communication with the SWO has been shown as plain, unencrypted HTTP? That's not what our customer IT administration teams want to see for the ongoing operations of their critical grid infrastructure. They prefer that communication to be fully encrypted with TLS.   Encrypting SAS Workload Orchestrator communications   My GEL team comrade Ajmal Farzam explains how to enable TLS encryption for HTTPS communication of SWO operations in his blog post, SSL Encryption for SAS Grid Manager. Notice that automatic client discovery of the current SWO master is still enabled as well.   Fully realize failover and encryption   SAS Grid Manager environments often represent a significant investment in time, effort, and money by our customers. Ensuring the availability of that investment is a high priority and so all single points of failure must be addressed. Having just one SAS Web Server to act as a proxy between the grid clients and the SWO master candidates isn't sufficient - we need two of them plus another third-party load-balancer in front of them. And of course, hardening HTTP communication with TLS encryption is absolutely necessary.   Highly available and fully encrypted operation of SAS Workload Orchestrator   SAS Technical Consultant Wade Adkins authored a post to explain how to accomplish this: High Availability and TLS with SAS Workload Orchestrator.   An exercise for the reader   We use the SAS Web Server to act as a proxy in front of the SWO master candidates for one reason: because it's there. But we don't have to. If your customer site enforces a strict separation of web tier from the compute tier, then they might prefer a different approach.   Bypassing the SAS Web Server     The service labeled as "VIP Load Balancer" in the illustrations above is an oblique reference to some piece of kit that your customer must provide, configure, and maintain. It might be hardware. Or software. Or some combination. In any case, it should be implemented as a highly available service, too. Given the examples above explaining how to configure the SAS Web Server (effectively the same as Apache HTTP Server), then your customer's IT team should have sufficient information to proceed.   Coda   The SAS Workload Orchestrator is delivered with a basic configuration out of the box. Sites running production environments SAS Grid Manager 9.4 M6 will usually require configuration to improve both high availability and encrypted communication. In addition to the SAS documentation, refer to the blog posts shown here to address the customer's requirements for availability and encryption of SWO operations. ... View more

Your customer has given requirements for your project which include disaster recovery (DR). Now, followers of this article might recall that disaster recovery is not a feature of SAS software. But you know that many SAS software components do offer options to improve availability through clustering. And perhaps your customer's IT organization is large enough to have data centers in multiple geographic regions. Or maybe they're running SAS in the cloud using AWS or Azure or GCS, all of which offer multiple availability zones (AZ) to protect cloud resources from the loss of a physical data center.   A quick back-of-the-napkin sketch showing SAS software clustered with nodes deployed into different AZ is often looked at as a simple solution to provide DR automatically and on the cheap:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   But this seemingly simple solution is fraught with several challenges. Overcoming these challenges is necessary to deliver the solution the customer expects to have. Not doing so will lead to performance problems and even worse, that the solution won't provide the availability and disaster recovery that was intended all along.   A quick note: In cloud parlance, "availability zone" has a specific meaning (1, 2, 3), usually where two or more AZ will reside in nearby geographic proximity within a "region". A widespread disaster that affects an entire region then might take out multiple AZ. And therefore, some companies will cluster across regions as well. Of course, regions can neighbor each other or be on opposite sides of the planet. To be clear, this post uses "AZ" to simply mean "different physical locations, near or far". Take note that topics discussed here will have varying considerations specific to physical distance. Challenge No. 1 - Latency, not just throughput From a computer networking perspective, latency is the amount of time between sending a request and receiving a response. Assuming no processing, just a simple bounce back, then this time is driven mostly by physical distance between the two communication nodes. We often use the ping utility to determine that amount of time. The further the distance, the longer the ping times meaning the greater the latency.   ping localhost # within the same host 64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.032 ms 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.093 ms   ping sas.com # within 1 mile (1.6 km) 64 bytes from 10.15.1.19: icmp_seq=0 ttl=252 time=2.753 ms 64 bytes from 10.15.1.19: icmp_seq=1 ttl=252 time=2.710 ms   ping weibo.cn # over 7,000 miles away (11,000 km) 64 bytes from 47.88.66.107: icmp_seq=1 ttl=90 time=153.716 ms 64 bytes from 47.88.66.107: icmp_seq=2 ttl=90 time=143.928 ms   So the challenge here is splitting up SAS clustered services into different availability zones. The farther the distance between the AZ, then the longer the ping times will be, sometimes substantially, orders of magnitude larger than when services co-exist in the same physical location. To see what this means for Amazon Web Services, visit cloudping.co for a real-time assessment of latencies between the various AZ.   Latency adds up fast, too. Web pages often make dozens or even hundreds of individual requests for files that make up a single web page… especially web apps like SAS Studio and SAS Visual Analytics. As those requests stack up, so does the impact of longer latency - and this means the user may perceive SAS performance as sluggish. If latency is too high, then some requests will timeout, meaning those resources aren't returned properly, leading to a broken user experience.   The same holds true for SAS Cloud Analytic Services and its component controllers and workers. They communicate frequently and sometimes even shuffle data between them. Higher latency between nodes means that the high-performance analytics engine won't be able to complete coordinated tasks as quickly or efficiently.   Challenge No. 2 - Placement of services Improving availability means eliminating single points of failure. This means that we often cluster our software so that redundant components run on separate hosts. In this way, if a single service instance, disk, power supply, or something else on the host fails, then the overall clustered service can continue operations until the problem is resolved.   The same concept can be extended to data centers, too. If a natural disaster strikes, then the entire data center might go dark. And that’s why cloud providers offer AZ so that it's easy to eliminate their physical infrastructure as a point of failure as well. So eliminating the data center as a single point of failure makes sense, but how many AZ do we need? Two? Three? More?   The answer is it depends on the software being clustered - and how that clustering actually works. Let's look a two quick examples. Cluster Type: Redundant Service Nodes For SAS Viya, consider the SAS Configuration Server (a.k.a. Hashicorp Consul). Each instance of Consul maintains its own copy of data. They can act independently to satisfy requests, but coordinate closely to ensure their data is kept in sync.   To maintain quorum, Consul requires clustering on three or more odd number of nodes. That way, as long as more than half the nodes stay in communication with each other, then Consul will continue in service. If we split those three nodes across two AZ, have we eliminated the data center as a single point of failure?   No. Not really.   With only two AZ, that means two out of three Consul nodes are in the same AZ. If that one AZ goes down for some reason, then Consul loses more than half of its nodes, which means it no longer has quorum. Even though one Consul node is still active, Consul will cease service at that point. To be consistent with Consul's expected operations in support of quorum, then it's necessary to deploy each Consul node to a different AZ.     Make similar considerations for SAS Infrastructure Data Server (Postgres and pgPool), SAS Message Broker (RabbitMQ), SAS Secrets Manager (Hashicorp Vault), SAS Cache Locator and Server (Apache Geode), as well as the various SAS server and microservices components. And keep in mind that each technology employs its own approach for ensuring a single version of the truth - not the same quorum technique as Consul. Cluster Type: Collaborative Service Nodes With the SAS Cloud Analytic Services (CAS), we have a different type of cluster technology. CAS is built using a shared-nothing MPP architecture. CAS employs one (or more) controllers to coordinate workload and a number of workers. The workers each are given a subset of the larger data problem to solve. They work independently on their own assignments, then send the results back to the controller.   If we split MPP CAS across two AZ, have we eliminated the data center as a single point of failure?   Maybe. It's possible.   CAS offers two controllers: a primary (required) and a secondary (optional). The secondary controller does not offer full functionality of the primary - and is only intended as a temporary backup for failover if the primary is suddenly lost. So if you place the primary controller in one AZ and the secondary controller on a different AZ, then split the workers evenly across both AZ, then the CAS service should be resilient if one data center does go dark.     But, keeping the CAS service running is only one problem to solve. More important is the data which CAS maintains in memory. Without data, CAS can't do much. If too many CAS workers go offline at once, then it's likely that the data CAS was keeping in memory distributed across all workers will be incomplete, effectively corrupt, and therefore unusable. So we need to consider what CAS requires to keep in-memory data highly available. Challenge No. 3 - Data availability For many, but not all, data sources and formats, CAS defaults to keeping one additional copy of data available to its workers (see COPIES= option for the PROC CASUTIL SAVE statement). With COPIES=1, then CAS can maintain data integrity if one CAS worker goes out of service.     Now what if a second CAS worker goes down? Without a second copy of the data to reference on the remaining node, then the data table is no longer complete. So if you want CAS to be resilient and maintain data integrity after the loss of multiple CAS workers, then increase the COPIES= value accordingly.   Consider this for an MPP CAS deployment with 10 workers split across two AZ - five workers in each AZ. If one data center goes offline, can CAS maintain data integrity with the remaining five workers?   Yes, but only if COPIES=5 was in effect when the data was loaded.   That's a big if. And it's a big ask. That's a lot of redundant data to copy around. And it requires considerable space for the CAS_DISK_CACHE on each of the workers. That's a large commitment to resources which must be maintained for high availability of data to protect against an event which is unlikely to occur.   An alternative to consider might be to establish a manual recovery process. For example, run two different MPP CAS servers, each in their own AZ with their respective workers together. The idea is that if one AZ goes down, then the CAS server in the other AZ is still available for service.   But what about the data? For this scenario, establish a process that all data for CAS use will first be staged in appropriate infrastructure that provides rapid parallel loading (e.g. SASHDAT format in DNFS). And of course, ensure it also has high availability across AZ.     This way, both instances of CAS will have speedy access to the same data source. If one CAS goes down, then the other could be used to run the same workload. While this requires more manual effort and some coordination, it's likely a simpler and more efficient use of resources over time. Indeed, the process could be setup so that a load action for SASHDAT data in the staging area is always run on both CAS servers every time. Due to how CAS loads SASHDAT from DNFS, the redundant copy of the data on the other CAS server may consume no in-memory resources if it's not actually used. See "Lazy Loading of SASHDAT" in my previous article, 4 Rules to Understand CAS Management of In-Memory Data. Challenge No. 4 - Synchronizing content As described above, clustering across AZ presents some significant challenges. If those can't be overcome, but DR is still desired, then a different approach might be employed. Typically this means deploying the full stack of SAS Viya software twice - each in a different AZ. Active-Active As you can imagine, many customers would like to see both such deployments of Viya online and processing their day-to-day workloads. They reason that if one Viya deployment goes down, then all of the workload could go to the other. This is most difficult to achieve, however, in terms of keeping the data and user-generated content of two different Viya environments perfectly in synchronization. Currently, we are not aware of a good way to do that for separate databases (like Consul, CAS, or even Postgres) which are operating in different physical deployments… unless they're set up as a single service cluster. But then you have to go back to the top of this post and re-read about Challenges 1-3 again. 😉 Active-Passive For active-passive, the idea is that the active Viya deployment handles the normal day-to-day operations while the passive deployment sits idle (or even switched off) until it's needed. In a disaster strikes the active deployment, then the passive one is spun up to take over - usually requiring some minutes or hours to come fully online. For Viya deployed using cloud providers, active-passive should be the default approach because the nature of the cloud makes it possible to easily bring up resources - like a passive environment - on demand when they're needed, costing nearly nothing in the meantime.   The challenge here is keeping the passive deployment up-to-date with data and user-generated content from the active environment. And with an environment like SAS Viya, this is usually best achieved with some form of regularly scheduled process which might involve a number of approaches, including elastic storage options, shared file systems, backup+restore and/or promote as well as manual steps to switch on the passive environment at the time it's needed. Coda Planning for disaster recovery is necessarily a collaborative process with your customer's IT team. When you see clustering across data centers or AZ proposed, then be wary about attempting that as a shortcut to achieve DR. While that might be tempting, educate your customer's IT team about the significant challenges that must be properly addressed to truly ensure performance is adequate and also that availability and DR are actually accomplished. Also remember that manual and/or site-specific actions might be needed (meaning design, build, testing, and evaluation) to ensure the customer can achieve their goals.   SAS publishes our SAS 9.4 Disaster Recovery Policy and our SAS Disaster Recovery Policy for SAS Viya 3.4 online for easy reference. ... View more

The SAS Cloud Analytic Services (CAS) offer lightning fast analytics of in-memory data. However, relying exclusively on RAM for all data management and operations is often a short road with an abrupt end. CAS extends its capabilities by also taking advantage of the persistent storage options available. In particular, CAS can be defined to use one or more local disks as a caching space to help improve fault tolerance, data availability and memory utilization. That disk space is referred to as CAS_DISK_CACHE. Analytics In-Memory CAS operates as a high-performance in-memory analytics engine.   In-memory is emphasized as that mode of operation provides the substantial performance value which is CAS’ trademark. Since CAS is designed for performing operations in memory, then running out of memory, like jail, is a place we don't want to be. But CAS offers us a Get Out of Jail Free card with its CAS_DISK_CACHE.   One of the many new features CAS has over LASR is the ability to use a cache as a backing store for in-memory data. The CAS cache provides flexibility for in-memory operations ensuring that CAS: Maintains availability of tables when more data is loaded than physical RAM available Provides failover protection if a CAS worker goes offline unexpectedly Relies on file handles to memory-mapped SASHDAT blocks to provide the mechanism for multiple CAS sessions to access the same single instance of data What is a backing store for CAS tables in memory? The data in CAS tables in memory are organized as SASHDAT blocks. And CAS will memory-map every SASHDAT block in memory to a location on disk. That disk might be the CAS cache. Or not.   If the original SASHDAT blocks are loaded from a mappable location, then that original source will be the backing store (caslib srctypes corresponding to Path, DNFS, and symmetrically co-located HDFS). Else if the table comes from another place (or a non-SASHDAT source), then CAS will memory-map that table's SASHDAT blocks in memory to the location specified for CAS_DISK_CACHE. Regardless of source, any new in-memory blocks created as output from CAS operations are always mapped to the CAS cache.   CAS relies on the operating system to manage the movement of data to/from the CAS cache via the memory map. This means the OS determines if and when to write (that is, page out) the in-memory SASHDAT data to the cache on disk. Even if the data is on disk, it's possible - ideal, really - that CAS might not ever need to read it back (that is, page in) from the cache. So for a system with plenty of RAM, the SASHDAT blocks may always be available to CAS in memory until the tables themselves are explicitly dropped.   The cache does provide a useful service and yet, as a rule, we prefer CAS to work with in-memory data as much as possible. If data must be fetched (that is, paged in) from disk, then performance should be expected to degrade by orders of magnitude for that operation. The CAS cache therefore is helpful when nominal RAM resources have been exhausted, but is not performance-equivalent to RAM.   The location for the CAS cache ( env.CAS_DISK_CACHE ) can be specified in several places:   File: When: Description: sas_viya_playbook/vars.yml At install The default used by CAS …/cas/casconfig.lua At startup on CAS Controller The default used by CAS Controller notifies Workers …/cas/casconfig_usermods.lua At startup on CAS Controller after casconfig.lua Overrides the default Controller notifies Workers …/cas/node_usermods.lua At startup on each CAS host after casconfig_usermods.lua Overrides the value from Controller per worker Analogy So, in a way, the CAS cache might be thought of as similar to an escape tunnel. It's there if we need it, but hopefully we won't. If we're digging a tunnel, but don't really plan to use it much, then we probably will spend as little time and effort on that tunnel's construction as we can get away with.   CAS is a very flexible product and can be used in many ways. Often, a site will use CAS in ways not originally envisioned at the time of purchase. We want to ensure that kind of flexibility of use is maintained. Also, as a high-performance product, many assumptions about CAS operations are based on the idea that hardware has been dedicated primarily, even exclusively, to CAS. If CAS is sharing its host(s) with other enterprise products, then we may need to adjust our approach.   So let's look at some tunnel - I mean, CAS_DISK_CACHE considerations. Primitive Resources There are some places on the CAS host which we can use as the CAS cache without any modifications at all. With minimal effort, we can get things working, but it's possible that we won't like it for long.   env.CAS_DISK_CACHE using /tmp Out of the box, CAS configuration defaults to using /tmp as the location of CAS_DISK_CACHE.   While /tmp is practically guaranteed to exist on most systems, it is not an ideal location to use for critical software operations. It might be sized too small (as little as 1 GB). And if /tmp fills up, then the operating system won't be able to run correctly, affecting everything else on the host. Furthermore, just because the files in CAS cache are indeed temporary, we want to ensure CAS itself makes the determination as to when they can be released.   ➤ Using /tmp as a short-term solution for working with small volumes of data with limited users is okay. But as a rule, we do not recommend using /tmp – instead, we should specify a different location.   Specified as:   env.CAS_DISK_CACHE='/tmp' env.CAS_DISK_CACHE using /dev/shm The /dev/shm location is a special construct in Linux referring to shared memory: it’s not a physical location on disk. Instead, it provides a mechanism for programs to share data with each other through the use of RAM. By specifying /dev/shm as the location for the CAS cache, the operating system will not need to perform any memory-mapped copying to slow disk.   ➤ Notice that we need to be careful using /dev/shm for the CAS cache because if RAM is exhausted, then the OS will swap out to its paging file. If that occurs, then all of sudden CAS will transition from running very fast to very, very slow - or worse, if CAS is running in a cgroup, then the largest RAM consuming process will be killed.   Specified as:   env.CAS_DISK_CACHE='/dev/shm' Improved Resources With a little bit of effort and planning, we can improve the route to CAS_DISK_CACHE so it can function well in a wider range of scenarios.     "In a hole in the ground there lived a hobbit. Not a nasty, dirty, wet hole, filled with the ends of worms and an oozy smell, nor yet a dry, bare, sandy hole with nothing in it to sit down on or to eat: it was a hobbit-hole, and that means comfort." -- Gratuitous reference to The Hobbit: There and Back Again Physical Disk as Persistent Storage If the site has requirements where CAS must load, manage, and operate on data which combined is larger than the RAM available, then the CAS cache really should rely on physical disk. After all, this concept has been de rigueur for storage considerations in computer science for decades. With physical disk, we can ensure that if circumstances occur where available RAM is not sufficient, then CAS has a dedicated location with plenty of space to store inactive data.   Most of the time, users of enterprise Linux servers aren't concerned with physical disks themselves. They just know about directories. Directories are free. Create them wherever you can. And this approach can work for CAS_DISK_CACHE. But at some point, things get real. If CAS fills up the root or user file systems, causing unexpected problems on the box, then the Linux administrator might want to have a discussion about giving CAS dedicated disk resources for its cache.   ➤ When provisioning physical disk for CAS cache, implement a single file system per disk. We prefer XFS and ext4 is acceptable. Avoid ext3 and any kind of NFS-based storage as the disk metadata and journaling may be overwhelmed by requests. env.CAS_DISK_CACHE using JBOD The acronym JBOD means Just a Bunch of Disks. It's a reference to one or more disks directly attached to the host with a basic intermediary connection. Almost like (but not really), we don't care how it's attached.   We typically want JBOD to be local-attached storage with enough disk space to match the size of RAM or more. Never less. Usually 1 - 3 disks are sufficient. When multiple disks are used, CAS has multiple I/O channels to access data and will create its files (with associated memory mappings) on those disks in round-robin order.   The challenges of working with individual disks are well-known. Each is a single point of failure and throughput is constrained. And the more disks we add to the system, the likelihood of a failure with those disks increases. If CAS loses a disk from its cache and that's the only place where that segment of data was, then the associated table(s) are incomplete - meaning losing a small part of a table is effectively the same as losing the entire table. Those files in CAS cache have a temporary lifespan, but CAS does need them.   ➤ Most enterprise IT teams don't expect to work with JBOD disks in this manner due to these challenges. So if positioning this as an option for CAS cache with your customer, take time to discuss with them. Perhaps a RAID solution (below) is more acceptable.   Specified as colon-delimited list of 1 or more full directory paths:   env.CAS_DISK_CACHE='/disk1:/disk2:/disk3' env.CAS_DISK_CACHE using RAID Having an escape tunnel is a nice feature. Furnishing it with nice appointments when you expect to use it regularly is a good idea. But there are cases where having a motorcycle on rails can help speed things up, too.   A RAID is a Redundant Array of Independent Disks. And there are many RAID levels with different functionality. As is typical for SAS solutions, we're focused here on RAID 5 where we have a group of 3 or more disks which work together as a single logical disk. Two-thirds of the space is for data and one-third is for parity. (For a 9-disk pack, only one-ninth is used for parity.) If one disk breaks, the others can continue operations. This eliminates individual disks as single points of failure and also provides some improvement to throughput as well.   ➤ Enterprise IT teams usually like this RAID approach. It's a standard offering with racked systems; easy to understand and justify. And our needs for CAS cache here are relatively modest. A single RAID array for the CAS cache should be fine.   Avoid shared storage solutions for CAS cache We want to keep the storage for the cache local to the CAS host - so do not cross the line over to shared storage solutions/appliances for CAS cache. Shared storage may appear convenient in situations where capacity is easily available, but the approach CAS uses with its cache negates some of the perceived benefits. Furthermore, some storage solutions, like IBM Spectrum Scale (a.k.a. GPFS), are not compatible with CAS_DISK_CACHE - it just won't work.   CAS cache is a completely separate conversation from using DNFS. DNFS technology used by CAS was explicitly designed for and works great with shared storage solutions.   For more information about RAID and other disk provisioning considerations for CAS, see Tony Brown's paper from SAS Global Forum 2019 titled, Engineering CAS Performance Hardware Network, and Storage Considerations for CAS Servers.   Specified as colon-delimited list of 1 or more full directory paths:   env.CAS_DISK_CACHE='/raidlogicaldisk1:/raidlogicaldisk2' Special Cases There are circumstances where CAS can be directed to operate in a way which (inadvertently) maximizes its reliance on the CAS cache. These situations need to be identified and recommendations made to either operate differently or tweak the configuration to reduce the impact. In-line scoring models The use of an in-line workflow to iteratively process scoring models, where the output from one run is appended with a new column and input into another run, may cause CAS to hit its cache excessively. So if CAS is relying on persistent storage for its cache, then expect a commiserate slowdown in performance since data loading from physical disk is much slower than from RAM.   ➤ If the iterative in-line approach to running scoring models is necessary, then consider: Ensuring there is plenty of RAM provided to CAS for this heavy use-case Configuring CAS to use /dev/shm for its cache Setting cas.MAXTABLEMEM=0 so that CAS won't cache very-temporary process files Using COPIES=0 in program code so CAS won't make any failover copies as its default behavior Super Highway Having an escape tunnel CAS_DISK_CACHE is a fine idea and it should be built correctly. But don't lose sight of the fact that CAS itself is meant to act as a super highway of performance. It is very flexible and extremely scalable. Remember that for CAS, we prefer to invest in scaling out horizontally with more CPU and RAM as problems get larger. So don't go too crazy with effort and resources trying to optimize every bit of performance out of CAS_DISK_CACHE. Often that can be better spent on CAS' primary mode of operation: in memory.     CAS cache offers very useful functionality to help in situations where primary performance resources (notably RAM) have been exhausted. Allowing a process to complete successfully, albeit slowly, is often preferable to outright failure. But slow is not the objective for CAS which is designed to operate at maximum performance and efficiency. Understanding how CAS can be configured for performance and resilience within the spirits of its design objectives is important for proper use. Understand more about how CAS uses its cache Before you can get serious about your decision on where to place the CAS_DISK_CACHE, you need to understand exactly what your processes will be asking CAS to do. With careful planning, a process can be optimized for CAS to reduce its reliance on the cache. On the other hand, some processes may require extensive use of the cache.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   I recommend getting acquainted with the SAS® Cloud Analytic Services 3.4: Fundamentals document for more information about how CAS works with data in-memory and on disk. Specifically look at topics including: Repeated tables for performance Fault tolerance and data redundancy Normal SASHDAT vs. compressed vs. encrypted Data partitioning and indexes Etc.  Thanks Just a quick shout out to Brian Bowman, Tony Brown, Gordon Keener, Steve Krueger, Barbara Walters, and many others for their work to identify, discuss, and explain the concepts here. ... View more

In SAS Viya 3.4, SAS Cloud Analytic Services (CAS) is the star of the show, providing lightning fast analytics of in-memory data for SAS Visual Analytics and other software offerings. However, Foundation SAS which is the long-time workhorse of SAS analytics is also offered, referred to as the SAS Programming Runtime Environment (SPRE). The SPRE provides a user interface and data processing environment for executing classic SAS program code. That is, it offers the Foundation SAS software we're all familiar with, including Base SAS, SAS/ACCESS engines, SAS/CONNECT packages, and more as well as the SAS Studio web application. Under the covers, however, there are two different SPRE implementations. One based on legacy SAS 9 Integration Technologies and the other built using the new Viya approach to software architecture. A Tale of Two SPRE The SAS Programming Runtime Environment is comprised of two different techniques to access Foundation SAS service: 1) legacy SAS Workspace Server relying on SAS Integration Technologies and 2) a new SAS Compute Server built using Viya architecture principles. Each has its own version of the SAS Studio web application to offer a SAS coding interface for users.   The inventory.ini file lists a set of host groups which represent pre-defined groups of SAS software components to deploy to specified hosts, including...   Legacy SAS 9 runtime host group: [programming] : Provides software in support of SAS Studio 4, including legacy SAS/CONNECT, SAS Object Spawner and SAS Workspace Server plus any additional software licensed (SAS/ACCESS, SAS/STAT, etc.). The SAS Workspace Server executes the SAS program code from SAS Studio. These components only come with a the "local" version of the SAS software license, so those components must all exist together on the same one host. Access this SPRE at url: http://[your Viya34 site].com/SASStudio New Viya Architecture SAS runtime host groups: [ComputeServer] : Provides the SAS Compute Server and SAS Launcher Server. The SAS Compute Server is the modern Viya equivalent of the legacy SAS Workspace Server. For SAS Studio 5, it executes SAS program code. [ComputeServices] : Provides the ComputeService and LauncherService microservices in support of the SAS Compute Server and SAS Launcher Server. [StudioViya] : Provides the SAS Studio 5 web app software. Relies on the SAS Compute Server and related services as the SAS backend. Access this SPRE at url: http://[your Viya34 site].com/SASStudioV As you can see, SAS Viya 3.4 effectively ships with two versions of SAS Studio - one for each flavor of Foundation SAS offered. SAS Studio 4 is the legacy offering with tight dependencies between software components, requiring them all together on the same host. And SAS Studio 5 is the modern Viya equivalent. SAS Studio 5 offers more deployment flexibility as implied by relying on three host groups.   See the SAS Viya 3.4 for Linux: Deployment Guide for information about the inventory.ini file and specifying deploy targets for the various host groups. Full vs. Programming-only There are two types of Viya deployment: full and programming-only. Full deployments are typically recommended and are most common. However, programming-only deployments are sometimes a better fit for specific use cases (such as using SAS Data Science).   Among other things, the programming-only deployment provides the [programming] host group in the inventory.ini file to install SAS Studio 4 and the legacy SAS 9 components as well as direct programming interfaces to SAS Cloud Analytic Services (CAS) through RESTful HTTP and languages like Python, R, and Lua.   The programming-only deployment does not provide host groups in the inventory.ini to install visual interfaces like SAS Drive and SAS Visual Analytics, nor does it include SAS Environment Manager or authentication services like SASLogon and Identities which are critical to the rest of Viya. It also does not include SAS Studio 5. The legacy SAS 9 runtime in the [programming] host group is built using pre-Viya software and so they still include their own user authentication capabilities. This means that programming-only deployments will rely on host-based authentication (usually via PAM in Linux hosts).   Interestingly enough, a full deployment of Viya provides host groups to install both SAS Studio 4 [programming] and SAS Studio 5 [StudioViya] along with [ComputeServer] and [ComputeServices]. For a full deployment of Viya, the SASLogon microservice is available to authenticate users with LDAP, Kerberos, OAuth, SAML, PAM, etc.   See "How Deployment Works" in the SAS Viya 3.4 for Linux: Deployment Guide for more explanation about different deployment types. And refer to the SAS Viya 3.4 Administration: Authentication doc to understand authentication options for your deployment. Considerations for SPRE on multiple hosts The SAS runtime - either as a legacy SAS Workspace Server delivered in [programming] or the SAS Viya Compute Server [ComputeServer] - can consume significant system resources like CPU, RAM, and especially disk I/O. If there will be heavy usage of the SPRE, then allocate hardware appropriately. For multi-host systems, avoid placing the SPRE on other mission-critical hosts like CAS servers to prevent resource contention.   In some cases, having just one host for the SPRE isn't sufficient. Identifying and eliminating single points of failure are key to improving system availability. Both types of SPRE can be deployed to run on multiple hosts to improve availability of their service. That said, it's important to understand that this does not provide failover protection of individual user sessions. That is, if a host fails, then any user sessions on that host (and their in-flight activities) are lost. To recover, the user would need to sign on a new SAS Studio session, establish a new SAS instance, and resume work from the last known good point.   With either legacy SAS Workspace Server [programming] or the SAS Viya Compute Server [ComputeServer], it is important to provide a shared file system for common data and other files which must be accessible across multiple hosts: Data Mart: SAS programs run in the SPRE could execute on any deploy target. We must ensure each of those hosts has access to the underlying data that the SAS program expects to work with. User Home Directories: The users' SAS preferences, the SASUSER libref, and other important files must be available for a consistent user experience. Performance: We typically exhort our customers to implement shared file systems which are can transfer data simultaneously to every concurrent execution of Foundation SAS at a rate of 100 MB/sec/core or more! Other: If you're already experienced with SAS Grid Manager implementations, this should all be familiar to you: Viya doesn't offer SAS Grid Manager (yet) and so related concepts like shared SAS binaries and configuration directories, checkpoint-restart, etc. do not apply For more information about shared file system considerations, see my article: Contemplating shared file systems for SAS. Deploy SAS Studio 4 and its SPRE to multiple hosts The SPRE host group for SAS Studio 4 [programming] bundles several pieces of disparate software together: SAS/CONNECT Spawner and Server, SAS Object Spawner, SAS Workspace Server, SAS Studio 4 web app and its own web application server.   By adding multiple deploy targets to the [programming] host group, full copies of all those components are placed on each host. Each deployment of [programming] operates statelessly with no coordination between each other.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   ▶ Side note: The [Operations] host group has a dependency on the [programming] host group (relying on some of those components to do its job). However, [Operations] is currently a singleton-only service - meaning it can only run on one host in the Viya environment and cannot be clustered for improved availability. Specify the deploy target for [Operations] that is the same as the first-listed deploy target for [programming].   To ensure that all [programming] hosts get their fair share of user workload, manual effort to configure load balancing in the Apache HTTP Server (or some equivalent) is required as a post-deployment task for programming-only deployments.     In this setup, each new user session is distributed in basic round-robin fashion to the hosts defined for SAS Studio 4. Each user session in SAS Studio establishes its own SAS Workspace Server instance on the same host. Each user's subsequent in-session requests are automatically routed to the same assigned host.   For full deployments of SAS Viya 3.4, the http-proxy microservice will automatically configure the load balancing when more than one [programming] deploy target is running. Deploy SAS Studio 5 and its SPRE to multiple hosts Much like a multi-tier deployment in SAS 9, we can deploy SAS Studio 5 and its SPRE host groups onto different deploy targets which are optimized for those workloads. In accordance with modern Viya architecture design, each of these host groups can run on multiple deploy targets with clustering configuration provided automatically by the Viya deployment process. So you don't need to muck about with the Apache HTTP Server proxy configuration at all.     The [StudioViya] host group provides a web application server for running the SAS Studio5 web app. Specify two or more deploy targets to improve availability of the SAS Studio 5 web app. Place this host group alongside other SAS Viya web apps on host(s) optimized for web application hosting.   The [ComputeService] host group provides ComputeServices and LauncherServices microservices for coordinating startup and activities of the the Compute Server and Launcher Server. Specify two or more deploy targets to improve availability of these services. As these services are relatively lightweight, they should be placed alongside other Viya infrastructure services.   The [ComputeServer] host group provides the SAS Launcher Server (similar in part to the SAS 9 Object Spawner) and SAS Compute Server (similar to SAS 9 Workspace Server) software. The SAS Compute Server can require significant system resources - CPU and RAM, of course, but is most often constrained by disk I/O. If you're expecting many concurrent users and/or time-sensitive batch jobs, then provision adequate deploy targets for this service. Specify at least two deploy targets for improved availability or more as needs dictate for workload.   For SAS Studio 5, there is no deterministic workload balancing of the SAS Compute Server… not even round-robin. When multiple deploy targets for [ComputeServer] exist, then SAS Compute Server host selection is random. Watch for improvements in this area coming with SAS Viya 3.5 or later. Interesting developments Even though the [programming] components are based on legacy SAS 9 Integration Technologies software with a SAS Object Spawner and SAS Workspace Server included, in SAS Viya 3.4 those components are restricted by a local software license to only work with SAS Studio 4 which resides on the same host machine.   For Viya 3.5 - expected to ship around 19w47 - an enterprise-grade license will be included with the [programming] components which will allow them to accept client connections from other hosts. This should allow SAS clients like SAS Add-in for Microsoft Office and SAS Enterprise Guide to connect to the [programming] SAS Object Spawner and SAS Workspace Server.   That said, the expectation is that the deployment process for the [programming] host group will remain the same in Viya 3.5 - all components included in just one host group. Summary The following table summarizes many of these concepts for easy review.     SAS Studio 4 and its SPRE SAS Studio 5 and its SPRE Available in deployment type In both Full and Programming-only deployments. Only in Full deployments. Foundation SAS software provided by... [programming] > SAS Workspace Server [ComputeServer] > SAS Compute Server Multiple SPRE hosts requires shared file system Yes, for user home directories and common data files. Yes, for user home directories and common data files. Multiple SPRE hosts requires manual configuration of Apache HTTP Server proxy definitions Yes, for Programming-only deployments of SAS Viya.   No, Full deployments of SAS Viya include ability to automatically cluster the SPRE. No, Full deployments of SAS Viya include ability to automatically cluster the SPRE. SAS Foundation is accessible to SAS Add-In for Microsoft Office and SAS Enterprise Guide No, with Viya 3.4.   Yes, with Viya 3.5. No. And as always... Remember to work with the SAS Enterprise Excellence Center to perform a proper workload estimate and hardware sizing prior to provisioning hardware and deploying SAS software. This will help ensure that the SPRE has adequate resources to accommodate the expected workload and number of concurrent users.   ... View more