Running SAS Viya in Upstream Open Source Kubernetes – part 1
The June 2022 stable release of SAS Viya (2022.1.2) brought the support of upstream open source Kubernetes.
This adds a second and very significant option for on-premises deployment of the Viya platform (in addition to Red Hat Openshift on vSphere).
Customers can choose among many ways to install K8s, many available K8s distributions (Rancher, Mirantis, etc…), many manual or automated deployment methods (kubeadm, kops, kubespray, etc).
In the first part of this blog, we’ll explain what this “upstream” term means and discuss what can indeed be considered as supported open source Kubernetes environments for SAS Viya. We will also present the new associated IaC (Infrastructure as Code) project and explain how it is organized and can be used to provision a supported Kubernetes environment.
What is supported and what is not supported
The scope of support
Before reading on please familiarize yourself with SAS Technical Support statements in regards to Viya.
First of all, we need to make the distinction between the "Standard” support" - provided by the SAS Technical Support and R&D teams for fully tested and documented Kubernetes implementations, such as Azure AKS (and others listed here) - and the "Limited" support policy that is applicable for Alternative Kubernetes Distributions.
In order for your Viya deployment to benefit from the “Standard” support in an open source Kubernetes implementation, there are some strict requirements that correspond to:
- “selective” configuration choices in the Kubernetes setup.
- the way the open-source Kubernetes project is installed and managed.
In Kubernetes implementation there are really a lot of possibilities in terms of network, storage and other infrastructure components. The "selective" configuration choices corresponds to fully tested environments and we will discuss them with more details in the next section.
In addition, SAS only support installations of the official “upstream” Open-Source K8s repository code (https://github.com/kubernetes/kubernetes), and the cluster must be created with one of the official documented ways from the kubernetes.io web page.
Many "packaged" distributions, even if they claim to be Kubernetes CNCF Certified, and even if they rely ultimately on the same upstream official project, are bringing their own “sauce” and features and as such are considered as “downstream” open source Kubernetes implementations.
Consequently, SAS does not provide "Standard" support for any kind of vendor-packaged solutions of open source Kubernetes (at least with the current stable release of SAS Viya).
However manually installing “upstream” Open source Kubernetes in production mode (with a highly available Control Plane for example), using one of the officially supported tools, is not something which is very easy to do...You might have heard about the “Kubernetes the hard way” project from Kelsey Hightower. That’ll give you an idea of what it takes and help you understand why IT admins are looking for packaged and easy-to-deploy Kubernetes implementations 😊
But the good news is that SAS provides the SAS Viya 4 Infrastructure as Code (IaC) for Open Source Kubernetes project (aka sassoftware/viya4-iac-k8s) to help IT and Kubernetes administrators to create and configure a cluster that meets SAS Viya system requirements.
To summarize, what falls under the standard support policy for open-source Kubernetes are clusters that were installed:
- with the Viya 4 IaC project, or
- with officially supported installation tools for the upstream K8s project, which are: kubeadm, kops or kubespray AND that follow the documented selective Kubernetes configuration choices (Calico CNI for the network, containerd as the container runtime, Ingress Nginx as the Ingress controller, etc…).
“OK but my customers are using Rancher for their Kubernetes deployments…”
While CNCF-certified and sometimes open source, packaged distributions like Rancher, Mirantis, or VMware Tanzu are very popular, they fall under the SAS “limited support” policy.
The reason is that those vendor packaged offerings usually include value-added features that often conflict with our deployment or at least need to be accounted for by our deployment. While the Kubernetes bits under the covers are the same, the control plane often imposes additional requirements that have an impact on whether or not things will actually work.
If we look at the landscape of Certified Kubernetes, we can see that there were 69 distributions (at the time of this write-up), and the list keeps growing.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Even if a Viya 4deployment could be successful in one of these distributions, such deployment would also fall under the SAS “limited support” policy.
SAS will continue to research the possibilities in providing standard official support for other certified Kubernetes distributions.
Should my customer use an alternative Kubernetes distributions?
Although not part of the standard supported platforms, various Kubernetes implementations have deployed successfully. SAS Teams have deployed Viya 4 using tools like RKE, Tanzu, ARO, Oracle, OpenShift on IBM Cloud, etc... .
This information, as well as running a successful viya4 ARK pre-installation report, increases the level of confidence for customers who want to continue using such distributions.
So yes, for customers wanting the choice of Kubernetes platform such as RKE Rancher, Mirantis or VMWare Tanzu, (assuming it is a CNCF certified platform) they can proceed. The caveat being that they are aware and are in agreement with the SAS Support for Alternative Kubernetes Distributions policy (which includes the definition of “limited support”).
Selective configuration choices
SAS provides standard support for environments that have been internally tested. As it would not be realistic to test every possible combinations of the Kubernetes infrastructure options, there are some key requirements to limit the scope of the possible.
Here is the link to the official documentation.
The new open source Kubernetes support is subject to most of the standard and documented Virtual Infrastructure requirements that are also true for cloud-managed Kubernetes, such as AKS, GKE or EKS.
These requirements includes things like : Kubernetes version, separate subnets for pods and services, dedicated namespace for Viya, Ingress Nginx, node labels, openSSL or cert-manager for TLS certification generation, etc.
But a few specific requirements for the “upstream” open source Kubernetes are explicitly called out in the documentation:
- “At this time, Calico 3.23.1 or later is the only software-defined networking layer for Kubernetes that is supported for a SAS Viya deployment in open source Kubernetes.”
- “To manage the required load balancer IP addresses, a utility like kube-vip or MetalLB is required.”
- “For a deployment on open source Kubernetes, only a containerd runtime is supported.”
- “For open source Kubernetes, you must have an A record with the FQDN associated with the load balancer and a CNAME record consisting of a wildcard entry pointing to that A”
So let’s review and explain these specific requirements a little bit more.
Calico CNI
As explained on this page, “CNI stands for Container Network Interface, a standard designed to make it easy to configure container networking when containers are created or destroyed. These plugins do the work of making sure that Kubernetes’ networking requirements are satisfied and providing the networking features that cluster administrators require.”
Various CNI plugins (Flannel, Calico, Weave, Canal, etc ) have been released to address specific environments and requirements.
Why do we require the Calico CNI and not something else?
Well, it was important to be selective/opiniated on the CNI (to restrict the scope of support) and the two images below shows the performance, security benefits, and popularity of Calico.
Technically Calico allows “inter-node” communications between pods and services inside the cluster by creating a layer-3 network to route packets between hosts and assigning a fully routable IP address to every pod.
In addition to networking connectivity, Calico is well-known for its advanced network features. As an example, Calico allows you to define network policies.
When we deploy with the IaC tool for opensource Kubernetes, we can specify distinct CIDR IP ranges for pods and services, such as:
kubernetes_pod_subnet : "10.42.0.0/16" kubernetes_service_subnet : "10.43.0.0/16"
and Calico will create virtual and internal (within the Kubernetes cluster) routable IP addresses from these ranges for the Kubernetes pods and services.
Load-balancer/VIP utility
In a Viya 4 environment, load-balancers with external IP addresses are required to expose Viya services and web applications outside of the cluster. In addition, if we install open source Kubernetes in HA mode with 3 control plane nodes, we also need a load-balancer to route our requests across the 3 nodes.
When Kubernetes runs in the Cloud (Azure, AWS or GCP for example), a control plane component called the “cloud-controller-manager” links your cluster into the cloud provider's API.
So when you create a Kubernetes "LoadBalancer" type of service to expose your applications, the cloud-controller-manager interacts with the Cloud provider to create the load-balancer and external IP resources in the Cloud.
But, if you are running Kubernetes on your own physical or virtual machines (on premises), the cluster does not have a cloud controller manager…
Several solutions exist to solve this issue, such as kube-vip or MetalLB, which can leverage network protocols (such as ARP or BGP) to simulate similar behavior using a pool of pre-defined floating IP addresses (IP addresses that must be part of the network but are not assigned).
As an example, kube-vip is used by the SAS Viya 4 IaC for Open Source Kubernetes project to use a pre-determined Virtual IP address as the external IP address for the LoadBalancer service.
Container Runtime Interface (CRI)
Another selective/opiniated choice is the Container Runtime (Kubernetes cannot run any container unless it has access to a Container Runtime, such as Docker engine, containerd or CRI-O).
In theory the container runtime does not matter for Kubernetes, but we have seen containerd progressively replace Docker as the standard default container runtime in modern Kubernetes Managed Services (AKS, GCP, etc).
While you can still use Docker to build your standard container images and run them with other container runtimes, using the Docker engine as the container runtime has been deprecated by the Kubernetes project after v1.20, in favor of runtimes that use the Container Runtime Interface (CRI) created for Kubernetes.
So the requirement here is that the customer has installed and configured upstream Open Source Kubernetes with “containerd” as the container runtime. (See this page for details when deploying with kubeadm.)
What else?
Finally, in the official requirements table below:
We can also notice that 4 node pools are required (instead of the 3 node pools that we require for other cloud platforms).
The requirement differs because in the Cloud managed Kubernetes services, the control plane is under the Cloud provider’s control, and you don’t have access to it (the Cloud provider is in charge of the maintenance, security, and regular upgrades of the control plane).
But in an on-premises implementation of open source Kubernetes, the customers manage the whole cluster infrastructure (including the control plane) – which also means that the they are responsible for maintaining, securing and keeping up to date the Kubernetes environment.
Introducing the new IaC project for open source Kubernetes
The SAS IaC tool for open source Kubernetes allows IT administrators to create a Kubernetes cluster and the associated infrastructure components where customers can then deploy a standard and supported Viya environment.
The diagram below (from the GitHub repository) shows the infrastructure components that can be automatically provisioned by the tool.
The tool support two types of deployments: bare-metal or vSphere and offers two deployment methods: either bash or docker.
In the bare-metal deployment, the customer must provide their own VMs or physical machines.
In the vSphere deployment, the customer can use terraform to automate the provisioning of the VMware VMs in the vSphere environment.
Then the project’s Ansible playbooks are run to configure the nodes at the system level and to install the Kubernetes Open Source software (using kubeadm behind the scene).
A script (oss-k8s.sh) calls the Ansible playbooks to perform these tasks.
The diagram, below summarizes the process.
So, as with other IaC projects, you can run the tool in standard mode (bash) or inside a Docker container.
Using the Docker container requires you to build the Docker image of the tool, and then call it to perform the Terraform provisioning (for a vSphere deployment), node configuration, and K8s installation.
The benefits of using the docker container method is that you don’t have to install all the required third-party tools (such as Ansible, Helm, Terraform, or kubectl) because they are included inside the Docker image.
Conclusion
In this first part of the blog, we discussed the scope of support, opiniated requirements and provided a high-level description of the Viya IaC tool for open source Kubernetes.
In the next part of the blog, we will look into the specific requirements for the IaC tool, see how to configure it for a "bare metal" type deployment (with the Ansible inventory and variables) and share some first "hands-on" experience feedback from our utilization of the tool.
Finally, we’ll show how it can be combined with the Deployment as Code project to prepare the "baseline" and deploy Viya in our open source Kubernetes cluster in an automated way.
Find more articles from SAS Global Enablement and Learning here.
