How to implement a highly available cloud deployment for SAS Viya 3.5

[#1]

SAS Viya 3.5 introduced support for clustering the last key component of the architecture that was still a single-point-of-failure, PGPool. The initial design only covered traditional on-premise environments. SAS has recently released detailed sample instructions to implement a highly available environment for each of three major cloud environments: Azure, AWS, and Google Cloud.

Before spending any additional word, let's place front and foremost the actual content: here is the link to the Github project covering the 3 major clouds:

 https://github.com/sassoftware/sas-viya-3.5-ha-deployment

As the readme clearly states, and each subproject reinforces, these are sample instructions based on internal testing by SAS QA. They should not be interpreted as "official instructions" or "mandatory architecture".

You may wonder, why only provide examples, and not "the officially supported way"?

Simply stated, cloud environments provide so much freedom and flexibility to your architectural design and implementation, that we neither can nor we want to mandate a unique way to do it. Rather, we want to show that it can be done, and how we did it.

Architecture diagram of a sample HA deployment of SAS Viya 3.5 on Azure.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

What's the big deal?

With SAS Viya, many components can be clustered to provide High Availability. SAS provides Pgpool-II open-source software to manage the PostgreSQL clusters that implement the SAS Infrastructure Data Server: Pgpool resides and operates between SAS Infrastructure Data Servers and clients, and also monitors the PostgreSQL cluster to react to failures. Until SAS Viya 3.4, Pgpool itself is a single point of failure. This was a conscious decision because the software release that was initially deployed with SAS Viya did not support any reliable way to establish quorum in case of network issues (split-brain problem). SAS Viya 3.5 removes this limitation, and Pgpool can be clustered.

The approach for BareOS uses a virtual IP that watchdog (a process included with PGPool) dynamically assigns to the current cluster master. Watchdog assigns and removes this IP using normal OS commands (sudo ip addr add / sudo ip addr delete), then advertises the change in IP ownership using the ARP protocol (sudo arping).

We soon found out that this approach does not work in cloud environments. Google explains the challenges with trying to use floating IPs on the Google Cloud Platform: "the VPC network handles ARP requests based on the configured routing topology, and ignores gratuitous ARP frames. In addition, it's impossible to directly modify the VPC network routing table with standard routing protocols".

Azure and AWS behave similarly, requiring a different solution to support clustering.

That's only part of the story.

While PGPool clustering is required, there are other critical infrastructure requirements to consider when designing a “Highly Available” SAS Viya environment – such as a shared file system and an HTTPS Load-balancer (with associated TLS certificates). The choice of an appropriate shared file system in a public cloud environment requires careful considerations, depending on the licensed products and customer needs. Also, the architecture should be designed considering typical cloud tenets such as nodes placement, availability zones, network and security design, and more.

As you can imagine, these considerations span such a wide array of topics and implementation details that required an ad-hoc project to be tackled.

Enter the members of the village: representatives from GEL, DevOps Engineering, Customer Lifecycle Testing, Technical Writing, with direction from Product Management, all worked together to bring clarity to our customers. As one of our managers stated: "with commitment, collaboration and perseverance we were able to get this very valuable documentation into the hands of our customers. "

Links to additional resources

To get additional information about PGPool clustering with SAS Viya 3.5, please consult the official documentation available at: https://go.documentation.sas.com/?cdcId=calcdc&cdcVersion=3.5&docsetId=dplyml0phy0lax&docsetTarget=p...

 

Kudos

I’d especially like to thank Raphael Poumarede, Joseph Cho, Susan Pearsall, Prity Shah for being my co-villagers in this adventure! And the final kick to place the ball in the goal came from Joe Furbee, who helped publish the projects on Github.

Conclusion

Providing enterprise-class SAS Viya deployments is a must for our customer's cloud environments. This project demonstrates that it is possible on the major cloud platforms; it provides enhanced availability and scalability, with integration with native cloud technologies.

Find more articles from SAS Global Enablement and Learning here.