BookmarkSubscribeRSS Feed

Scalavailability of CAS

Started ‎08-26-2021 by
Modified ‎08-26-2021 by
Views 4,317

Yeah, I made that word up. As you've probably already surmised, it's a portmanteau of "scalability" and "availability". And their consideration within the architecture and deployment of SAS Cloud Analytic Services are frequent topics of interest. And like most things SAS, we have a wide variety of options regarding its deployment and operation.

 

CAS has its roots based on previous highly scalable in-memory analytic engines. It's optimized to run fast in a wide-range of scenarios and to scale up and out to tackle the largest analytic problems at incredible speed. Beyond scalability, it also provides a number of features to provide improved availability of data and service over its predecessors.

 

Where do scalability and availability of CAS overlap and reinforce each other? Let's talk about it.

Foundation Concepts

This is a quick run down of the features of CAS relevant to this post:

 

Scalability:

  • Two modes of operation: SMP (single machine) and MPP (multiple machines providing a single service)
  • The minimal MPP configuration is 1 Controller and 1 Worker - but that's inefficient and should be avoided (use SMP instead). Real world minimum should be 1 Controller and 2 Workers.
  • Add more Workers at any time without interrupting the system. Converting from SMP to MPP requires an outage.

Availability:

  • Blocks of data can be copied amongst the workers to protect against the loss of a node (more copies == toleration to support loss of more nodes)
  • A second CAS Controller can be enabled to improve its availability - but it's not a full peer and recovery to full service status requires an outage.

Now there are caveats and provisos in certain situations to many of these aspects of CAS… but they convey the attributes we want to look at in broad strokes.

Costs

We should never lose sight of costs. When it comes to scalability, the costs are often easier to justify. If a particular analytics problem takes too much time, needs more resources, or is otherwise too large, there's a good chance that it can be corrected by throwing more money CPU and RAM at it for CAS to work with. Getting necessary work done is often a sufficient justification for increased cost.

 

Availability is sometimes harder to justify cost-wise. Effectively it's an insurance policy. You hope you won't need it and you don't want to pay for it, but at some point, you have been (or will be) glad you did.

 

Remember above where we said that we can keep extra copies of data in CAS to help survive the loss of a Worker? Where does that data actually go? On local disk of each CAS Worker. Which means each CAS Worker must have local disk to use for that purpose (not always a given in today's cloud-provisioned infrastructure). More copies = more protection for more Workers = more disk space = more cost.

 

And then there's the Secondary CAS Controller. When it's operating in its secondary role, it doesn't do much but acknowledge the activities of the primary Controller, ready to take over when needed. And from that perspective, it's an ongoing cost with a possible future benefit that might not be directly realized. Cloud costs for services add up by the minute (or even second), so when does it make sense to standup a secondary Controller for CAS? Usually the decision balances on the cost of loss of work, which often times is correlated to the size of the CAS cluster (i.e. bigger clusters do more work for more people).

Scalavailability

So I've been kicking this idea of scalavailability regarding CAS around for a while. I want a simple set of rules that I can follow as a happy path to achieve the right balance of scalability and availability when they're most likely to complement each other.

 

We can literally deploy CAS almost any way we can think of - including in configurations which we shouldn't because they're inefficient, unwieldly, contradictory to its design, and so on. Of course we want to avoid those, but what about others that, while they are technically correct, don't quite satisfy the idea of scalavailability?

 

My premise here is that as the number of nodes in the CAS cluster increases, the likelihood is that the loss of its service would be a greater impact and we should have a standard approach to reconcile that.

Scalavailability of the CAS Controller

When should you employ a Secondary CAS Controller? It's optional after all. You could just ignore it and never include one. Or you might reason to include it in all of your CAS deployments.

 

For CAS, my scalavailability rule is this:

 

(number of CAS Controllers) < (number of CAS Workers) / 2

 

Let's apply this rule to CAS deployed in scenarios for 1 - n hosts:

 

Hosts Controller(s) Worker(s) Mode Recommend? Reason
1 - - SMP YES For smaller implementations
2 1 1 MPP NO No, this is inefficient, use SMP
3 1 2 MPP YES  
4 1 3 MPP YES  
5 1 4 MPP YES  
6 1 5 MPP YES Borderline - but using 2 controllers with 4 workers would be "equal to" not "less than" as per the proposed scalavailability rule
7 2 5 MPP YES  
n 2 (n-2) MPP YES  

 

So, based on this rule, when you approach the point of running 5 Workers, that's when you should consider adding a Secondary CAS Controller if availability concerns warrant and budget allows.

 

Of course you can come up with your own calculation. Think of this one as a line in the sand. Now that you know where it is, you can decide which side you want to be on.

 

And of course, there will be some circumstances where the need for a Secondary CAS Controller will be obvious (or the lack thereof) regardless of the actual number of Workers. So accommodate that in your approach as well.

Scalavailabilty of Data in CAS

Did you know that the more Workers you add to MPP CAS, the greater the likelihood that one will fail? Each Worker has some small chance of failing due to hardware, software, or other issues. Additional Workers multiply that chance. This isn't just a CAS thing… it's a known consideration for RAID arrays and why a large number of disks in a RAID-0 striped array isn't really a good idea if the data kept there is important, even for a limited lifespan. *looks sideways as SASWORK*

 

Data in CAS is distributed across the Workers in blocks. If there's just one instance of the data, and one of your CAS Workers goes offline (taking its allotment of data with it), then you no longer have a complete data set.

 

The COPIES= parameter directs CAS to make copies of blocks of data. With COPIES=1, there's the original in-memory (hopefully) instance of the data plus one more copy stashed away in the CAS disk cache. The copy is distributed such that a complete instance of the data will be available if one Worker goes offline. Increase the number of copies and CAS can tolerate the loss of more Workers.

 

But really, how likely are we to deal with the loss of multiple Workers? Well remember that we're not just talking about CAS Worker containers in their pods in a Kubernetes cluster, but also with those running in virtual machine instances that reside on physical computer systems racked together with power supplies, storage systems, network hubs, etc. in a data center. A failure at any of those points could take out one or more CAS Workers. But don't freak out yet.

 

Generally, the default is for COPIES=1 when CAS is working with non-native data from remote sources (or COPIES=0 for memory-mappable unencrypted SASHDAT). And, in the real world, for the most part, COPIES=1 is very sustainable adding a relatively low overhead and providing a huge benefit. It's a good default regardless how many CAS Workers you're running.

 

Even so, for data in CAS, I do have a scalavailabilty rule I'm considering:

 

copies = ROUNDUP [ ( number of CAS Workers ) / 10 ]

 

Simply put, for every 10 Workers, increase the number of copies by 1. This should keep the overhead of storing additional copies of data acceptably low and still provide improved availability of larger CAS deployments.

 

Workers Mode COPIES=
1 SMP 0
2-10 MPP 1
11-20 MPP 2
21-30 MPP 3

 

Like before, your mileage may vary here. It's certainly possible to have scenarios where you might want zero copies regardless of how many CAS Workers there are. Or that you might need more copies. The point is to start somewhere with considering the possibility and your customer's requirement, then acting accordingly.

Coda

When do we reach the tipping point that drives us to make a decision in one direction or another? When it comes to the architecture and deployment of SAS Viya, there are many decisions - and often many such tipping points. The simple rules I've illustrated here are meant to be examples, not to be taken as The Right and Only Way. But I find often that codifying common scenarios with such rules can help navigate the complex set of interdependent decisions necessary to achieve success with SAS Viya. I hope this works for you, too.

Comments

Very useful article, thank you for sharing it. 

Version history
Last update:
‎08-26-2021 10:19 AM
Updated by:
Contributors

SAS INNOVATE 2024

Innovate_SAS_Blue.png

Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.

If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website. 

Register now!

Free course: Data Literacy Essentials

Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning  and boost your career prospects.

Get Started

Article Tags