The SAS Cloud Analytic Services (CAS) offer lightning fast analytics of in-memory data. However, relying exclusively on RAM for all data management and operations is often a short road with an abrupt end. CAS extends its capabilities by also taking advantage of the persistent storage options available. In particular, CAS can be defined to use one or more local disks as a caching space to help improve fault tolerance, data availability and memory utilization. That disk space is referred to as CAS_DISK_CACHE.
CAS operates as a high-performance in-memory analytics engine.
In-memory is emphasized as that mode of operation provides the substantial performance value which is CAS’ trademark. Since CAS is designed for performing operations in memory, then running out of memory, like jail, is a place we don't want to be. But CAS offers us a Get Out of Jail Free card with its CAS_DISK_CACHE.
One of the many new features CAS has over LASR is the ability to use a cache as a backing store for in-memory data. The CAS cache provides flexibility for in-memory operations ensuring that CAS:
What is a backing store for CAS tables in memory?
The data in CAS tables in memory are organized as SASHDAT blocks. And CAS will memory-map every SASHDAT block in memory to a location on disk. That disk might be the CAS cache. Or not.
If the original SASHDAT blocks are loaded from a mappable location, then that original source will be the backing store (caslib srctypes corresponding to Path, DNFS, and symmetrically co-located HDFS). Else if the table comes from another place (or a non-SASHDAT source), then CAS will memory-map that table's SASHDAT blocks in memory to the location specified for CAS_DISK_CACHE. Regardless of source, any new in-memory blocks created as output from CAS operations are always mapped to the CAS cache.
CAS relies on the operating system to manage the movement of data to/from the CAS cache via the memory map. This means the OS determines if and when to write (that is, page out) the in-memory SASHDAT data to the cache on disk. Even if the data is on disk, it's possible - ideal, really - that CAS might not ever need to read it back (that is, page in) from the cache. So for a system with plenty of RAM, the SASHDAT blocks may always be available to CAS in memory until the tables themselves are explicitly dropped.
The cache does provide a useful service and yet, as a rule, we prefer CAS to work with in-memory data as much as possible. If data must be fetched (that is, paged in) from disk, then performance should be expected to degrade by orders of magnitude for that operation. The CAS cache therefore is helpful when nominal RAM resources have been exhausted, but is not performance-equivalent to RAM.
The location for the CAS cache (
env.CAS_DISK_CACHE) can be specified in several places:
||At install||The default used by CAS|
||At startup on CAS Controller||The default used by CAS
Controller notifies Workers
||At startup on CAS Controller
|Overrides the default
Controller notifies Workers
||At startup on each CAS host
|Overrides the value from Controller per worker|
So, in a way, the CAS cache might be thought of as similar to an escape tunnel. It's there if we need it, but hopefully we won't. If we're digging a tunnel, but don't really plan to use it much, then we probably will spend as little time and effort on that tunnel's construction as we can get away with.
CAS is a very flexible product and can be used in many ways. Often, a site will use CAS in ways not originally envisioned at the time of purchase. We want to ensure that kind of flexibility of use is maintained. Also, as a high-performance product, many assumptions about CAS operations are based on the idea that hardware has been dedicated primarily, even exclusively, to CAS. If CAS is sharing its host(s) with other enterprise products, then we may need to adjust our approach.
So let's look at some tunnel - I mean, CAS_DISK_CACHE considerations.
There are some places on the CAS host which we can use as the CAS cache without any modifications at all. With minimal effort, we can get things working, but it's possible that we won't like it for long.
Out of the box, CAS configuration defaults to using
/tmp as the location of CAS_DISK_CACHE.
/tmp is practically guaranteed to exist on most systems, it is not an ideal location to use for critical software operations. It might be sized too small (as little as 1 GB). And if
/tmp fills up, then the operating system won't be able to run correctly, affecting everything else on the host. Furthermore, just because the files in CAS cache are indeed temporary, we want to ensure CAS itself makes the determination as to when they can be released.
/tmp as a short-term solution for working with small volumes of data with limited users is okay. But as a rule, we do not recommend using
/tmp – instead, we should specify a different location.
/dev/shm location is a special construct in Linux referring to shared memory: it’s not a physical location on disk. Instead, it provides a mechanism for programs to share data with each other through the use of RAM. By specifying
/dev/shm as the location for the CAS cache, the operating system will not need to perform any memory-mapped copying to slow disk.
➤ Notice that we need to be careful using
/dev/shm for the CAS cache because if RAM is exhausted, then the OS will swap out to its paging file. If that occurs, then all of sudden CAS will transition from running very fast to very, very slow - or worse, if CAS is running in a cgroup, then the largest RAM consuming process will be killed.
With a little bit of effort and planning, we can improve the route to CAS_DISK_CACHE so it can function well in a wider range of scenarios.
"In a hole in the ground there lived a hobbit. Not a nasty, dirty, wet hole, filled with the ends of worms and an oozy smell, nor yet a dry, bare, sandy hole with nothing in it to sit down on or to eat: it was a hobbit-hole, and that means comfort."
-- Gratuitous reference to The Hobbit: There and Back Again
If the site has requirements where CAS must load, manage, and operate on data which combined is larger than the RAM available, then the CAS cache really should rely on physical disk. After all, this concept has been de rigueur for storage considerations in computer science for decades. With physical disk, we can ensure that if circumstances occur where available RAM is not sufficient, then CAS has a dedicated location with plenty of space to store inactive data.
Most of the time, users of enterprise Linux servers aren't concerned with physical disks themselves. They just know about directories. Directories are free. Create them wherever you can. And this approach can work for CAS_DISK_CACHE. But at some point, things get real. If CAS fills up the root or user file systems, causing unexpected problems on the box, then the Linux administrator might want to have a discussion about giving CAS dedicated disk resources for its cache.
➤ When provisioning physical disk for CAS cache, implement a single file system per disk. We prefer XFS and ext4 is acceptable. Avoid ext3 and any kind of NFS-based storage as the disk metadata and journaling may be overwhelmed by requests.
The acronym JBOD means Just a Bunch of Disks. It's a reference to one or more disks directly attached to the host with a basic intermediary connection. Almost like (but not really), we don't care how it's attached.
We typically want JBOD to be local-attached storage with enough disk space to match the size of RAM or more. Never less. Usually 1 - 3 disks are sufficient. When multiple disks are used, CAS has multiple I/O channels to access data and will create its files (with associated memory mappings) on those disks in round-robin order.
The challenges of working with individual disks are well-known. Each is a single point of failure and throughput is constrained. And the more disks we add to the system, the likelihood of a failure with those disks increases. If CAS loses a disk from its cache and that's the only place where that segment of data was, then the associated table(s) are incomplete - meaning losing a small part of a table is effectively the same as losing the entire table. Those files in CAS cache have a temporary lifespan, but CAS does need them.
➤ Most enterprise IT teams don't expect to work with JBOD disks in this manner due to these challenges. So if positioning this as an option for CAS cache with your customer, take time to discuss with them. Perhaps a RAID solution (below) is more acceptable.
Specified as colon-delimited list of 1 or more full directory paths:
Having an escape tunnel is a nice feature. Furnishing it with nice appointments when you expect to use it regularly is a good idea. But there are cases where having a motorcycle on rails can help speed things up, too.
A RAID is a Redundant Array of Independent Disks. And there are many RAID levels with different functionality. As is typical for SAS solutions, we're focused here on RAID 5 where we have a group of 3 or more disks which work together as a single logical disk. Two-thirds of the space is for data and one-third is for parity. (For a 9-disk pack, only one-ninth is used for parity.) If one disk breaks, the others can continue operations. This eliminates individual disks as single points of failure and also provides some improvement to throughput as well.
➤ Enterprise IT teams usually like this RAID approach. It's a standard offering with racked systems; easy to understand and justify. And our needs for CAS cache here are relatively modest. A single RAID array for the CAS cache should be fine.
Avoid shared storage solutions for CAS cache
We want to keep the storage for the cache local to the CAS host - so do not cross the line over to shared storage solutions/appliances for CAS cache. Shared storage may appear convenient in situations where capacity is easily available, but the approach CAS uses with its cache negates some of the perceived benefits. Furthermore, some storage solutions, like IBM Spectrum Scale (a.k.a. GPFS), are not compatible with CAS_DISK_CACHE - it just won't work.
CAS cache is a completely separate conversation from using DNFS. DNFS technology used by CAS was explicitly designed for and works great with shared storage solutions.
For more information about RAID and other disk provisioning considerations for CAS, see Tony Brown's paper from SAS Global Forum 2019 titled, Engineering CAS Performance Hardware Network, and Storage Considerations for CAS Servers.
Specified as colon-delimited list of 1 or more full directory paths:
There are circumstances where CAS can be directed to operate in a way which (inadvertently) maximizes its reliance on the CAS cache. These situations need to be identified and recommendations made to either operate differently or tweak the configuration to reduce the impact.
The use of an in-line workflow to iteratively process scoring models, where the output from one run is appended with a new column and input into another run, may cause CAS to hit its cache excessively. So if CAS is relying on persistent storage for its cache, then expect a commiserate slowdown in performance since data loading from physical disk is much slower than from RAM.
➤ If the iterative in-line approach to running scoring models is necessary, then consider:
/dev/shmfor its cache
cas.MAXTABLEMEM=0so that CAS won't cache very-temporary process files
COPIES=0in program code so CAS won't make any failover copies as its default behavior
Having an escape tunnel CAS_DISK_CACHE is a fine idea and it should be built correctly. But don't lose sight of the fact that CAS itself is meant to act as a super highway of performance. It is very flexible and extremely scalable. Remember that for CAS, we prefer to invest in scaling out horizontally with more CPU and RAM as problems get larger. So don't go too crazy with effort and resources trying to optimize every bit of performance out of CAS_DISK_CACHE. Often that can be better spent on CAS' primary mode of operation: in memory.
CAS cache offers very useful functionality to help in situations where primary performance resources (notably RAM) have been exhausted. Allowing a process to complete successfully, albeit slowly, is often preferable to outright failure. But slow is not the objective for CAS which is designed to operate at maximum performance and efficiency. Understanding how CAS can be configured for performance and resilience within the spirits of its design objectives is important for proper use.
Before you can get serious about your decision on where to place the CAS_DISK_CACHE, you need to understand exactly what your processes will be asking CAS to do. With careful planning, a process can be optimized for CAS to reduce its reliance on the cache. On the other hand, some processes may require extensive use of the cache.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
I recommend getting acquainted with the SAS® Cloud Analytic Services 3.4: Fundamentals document for more information about how CAS works with data in-memory and on disk. Specifically look at topics including:
Just a quick shout out to Brian Bowman, Tony Brown, Gordon Keener, Steve Krueger, Barbara Walters, and many others for their work to identify, discuss, and explain the concepts here.
Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.
If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website.
Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning and boost your career prospects.