This wraps up a three-part series where we investigate the number of SAS analytics pods that can run in the Kubernetes cluster.   For this post, we will outline the process and considerations for determining the number of SAS Compute Server instances can run on a Kubernetes node. In particular, we will focus on specific environmental configuration and infrastructure resources to get an idea of the number of pods that are possible, while acknowledging the significant caveat that the SAS Programming Runtime Environment is used for jobs with varying complexity and data volumes.   [ Part 1 | Part 2 ]   SAS Workload Management   As we've already discussed, SAS Workload Management brings additional powerful capabilities to the SAS Viya platform for scheduling dynamically-launched processing pods of the SAS Programming Runtime Environment as instantiated by SAS Compute, SAS Batch, and SAS Connect Servers as well as other pods running Python or R as started by the Batch service.   We'll refer to all of that collectively as "SAS Compute" here - or even more simply as "jobs" that the SAS Workload Orchestrator will manage (and to be clear, are not to be confused with Kubernetes "jobs" which are something else).   If you've worked with SAS Workload Management before (and its SAS 9 predecessor, SAS Grid Manager), then you're already familiar with many aspects of its operation and configuration. However, in the last few months, a couple of major changes have come down the pike that we should review first.   Timeline   Here’s a brief and abbreviated history of how workload management in SAS Viya has evolved recently.   ∵ Up to and including SAS Viya release 2023.07 – (notable items in bold)   SAS Workload Management was an optional offering that could be added to a SAS Viya deployment.   Without SAS Workload Management, then the SAS Launcher service relied on SAS_LAUNCHER_USER_PROCESS_LIMIT (and one for SUPER_USER) to restrict the number of SAS Compute processes that a single user could launch. This helps to prevent inadvertent and/or malicious denial-of-service where one user might saturate the system with too many jobs.   With SAS Workload Management active in the environment, then the Launcher’s user process limit is ignored because SAS Workload Orchestrator offers numerous fine-grained controls on job placement and execution.   With SAS Workload Management active in the environment, then SAS Workload Orchestrator was configured to allow a maximum number of jobs on a host that is equal to the number of CPU cores. This value can be changed at any time to suit the purposes of the environment to some multiple of CPU core count or any integer value.   ∵ Beginning with SAS Viya release 2023.08 – (notable items in bold)   The SAS Workload Management offering is included with SAS Viya and is activated by default.   By default, SAS Workload Orchestrator is configured to allow a maximum number of jobs on a host that is equal to the number of CPU cores. This value can be changed at any time to suit the purposes of the environment to some multiple of CPU core count or any integer value.   The Launcher’s user process limit is no longer evaluated unless SAS Workload Management is deactivated.   ∵ Beginning with SAS Viya release 2023.11 – (notable items in bold)   The SAS Workload Management offering is still included with SAS Viya and is still activated by default.   By default, SAS Workload Orchestrator is configured to allow a maximum of 250 jobs on a host. This value can be changed at any time to suit the purposes of the environment to some multiple of CPU core count or any integer value.   The Launcher’s user process limit is still not evaluated unless SAS Workload Management is deactivated.   Regardless of what’s going on with SAS Workload Management, there are other aspects of SAS Viya which play a role control the concurrency of jobs beyond the scope of this post to be considered.   From just one SAS Compute instance per CPU to 250 per node?   Does that mean that Viya should be able to run with 250 SAS Compute jobs on a host node? Probably not, but only because it’s unlikely that Kubernetes and the infrastructure hosting the environment has been set up with enough resource capacity to do so.   SAS Workload Management, along with the rest of Viya, is still backed by Kubernetes and subject to many of the same considerations and restrictions. And Kubernetes itself is still backed by the infrastructure it’s running on which is to say there are physical limitations of hardware that matter, too.   By setting 250 as the new default maximum for the number of jobs on a host, this effectively removes one of the throttling factors that SAS Workload Orchestrator uses to manage jobs. That is, this gets the default configuration of SAS Workload Management “out of the way” such that Kubernetes will be the primary throttling agent. This change was made so that if there’s a large number of SAS Compute jobs attempting to run in a new Viya deployment, then they’re more likely achieve utilization of the "compute" node pool that would be expected per typical Kubernetes workload planning.   Of course, the idea is that this throttling factor and other SAS Workload Orchestrator configurations can be changed at any time to suit your needs. Remember in the previous post we touched on the variability of work a SAS Compute process might be asked to perform. So, you might even choose to revert to the old "1 job per CPU" value (or a similar CPU-based multiple) if planning for large-effort jobs.   Infrastructure constraints   You’ll often hear that Kubernetes has a default maximum of 110 pods allowed to run per node. While true in some regards, it’s deceptive to assume because the value is configurable and also is subject to external influence, like physical attributes of the hosting infrastructure.   In a cloud provider like Amazon Web Services, the number of network interfaces and associated IP addresses is often a limiting factor that determines the maximum number of pods a node can host. This can be increased by choosing to run larger instance types as well as installing optional networking software, like a Container Networking Interface, to the Kubernetes cluster.   For example, we can query my test environment to see the maximum pods allowed and how it varies from node to node of my Kubernetes cluster:   $ kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.capacity.cpu,MEMORY:.status.capacity.memory,MAX_PODS:.status.capacity.pods,TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value NAME CPU MEMORY MAX_PODS TaintKey TaintValue ip-192-168-126-215.ec2.internal 8 32386544Ki 58 ip-192-168-38-70.ec2.internal 8 32042484Ki 58 ip-192-168-7-135.ec2.internal 8 32042484Ki 58 workload.sas.com/class cas ip-192-168-9-145.ec2.internal 8 32042484Ki 58 workload.sas.com/class cas ip-192-168-98-179.ec2.internal 8 32386544Ki 58 workload.sas.com/class cas ip-192-168-106-88.ec2.internal 8 32386544Ki 58 workload.sas.com/class cas ip-192-168-14-66.ec2.internal 32 130389628Ki 234 workload.sas.com/class compute ip-192-168-29-85.ec2.internal 32 130389628Ki 234 workload.sas.com/class connect ip-192-168-102-24.ec2.internal 16 65021468Ki 110 workload.sas.com/class stateful ip-192-168-72-173.ec2.internal 16 64333348Ki 110 workload.sas.com/class stateless   Notice the nodes that are tainted for “compute” and “connect” each have a MAX_PODS value of 234. That number is less than the 250 maximum jobs allowed per node value configured by default in SAS Workload Orchestrator and is a hard limit that cannot be exceeded.   To be clear, that MAX_PODS value can be edited as part of the kubelet settings (especially in your own local deployment of Kubernetes), but when it comes to cloud-provided Kubernetes, it’s set for you automatically to communicate the underlying infrastructure limits. Changing that alone won’t magically add more resources like CPU cores or addressable IP space, for example.   SAS Workload Orchestrator constraints   We began this post by mentioning the maximum jobs allowed per node in SAS Workload Orchestrator. But that’s just one parameter out of many that are important to consider. We can also set maximum jobs by queue, by user or group, as well as maximum jobs total for the cluster.   And we can go much farther than that with a wealth of scheduling options. We can close hosts (i.e., take them out of service temporarily, analogous to cordoning a node in Kubernetes). Thresholds can be set so that once a certain amount of host resource is being consumed, then new jobs are kept in the queue until enough of that resource is available. And that’s not all, we can combine the various factors and establish a comparison order to prioritize their evaluation for job placement.   One question you might ask is, “Why would I want to set an arbitrary limit on the number of compute processes per node that’s not based on CPU or RAM usage?”   A possible answer to that lies in fact that data-intensive processing has often been a bottleneck for SAS Compute. SAS can only process data as fast as it can be delivered to the CPU. And so, we often recommend starting conversations with infrastructure providers to ensure SAS compute has access to 125 MB/sec/core of i/o throughput. Kubernetes doesn’t monitor this particular metric in terms of managing workload and many of the lower-tier offerings from cloud-providers can struggle to meet this goal. Therefore, one way to ensure that SAS Compute is running at maximum efficiency might be to set a max jobs per host limit that restricts the number of SAS Compute so that those running can fully utilize the CPU as far as the i/o throughput rate allows.   SAS Compute processes can vary widely in terms of the tasks they perform and hence the resources they consume. Be careful with arbitrary, one-size-fits-all constraints so that you don’t overfit SAS Workload Orchestrator for some jobs at the expense of others.   How many SAS Compute jobs can run on a node?   As shown in this post, to answer this question requires evaluating the configuration and associated resources of your environment. Simply put, whatever factor yields the smallest number is likely the answer. This might require some investigation, inference, and experimentation.   We've talked about several concepts so far…   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Host closed? Or node cordoned? Is the host closed in SAS Workload Orchestrator (or cordoned in Kubernetes)? Then zero SAS Compute jobs can run there.    Pod resources requested initially   Next look at the combined CPU and RAM requests (if defined) for the containers inside the SAS Compute Server pod. Divide the total available resource by its associated composite request value in the pod and round the result down to the nearest integer to get an idea of how many can run on the host. For example, if the containers in a SAS Compute pod request a composite maximum of 750 millicores of CPU, then a Kubernetes node labeled for "compute" workload with 16 CPU could possibly run up to 21 (= ⌊16 cores ÷ 0.75⌋) SAS Compute jobs (not including CPU overhead consumed by OS, Kubernetes control plane, or other processes). Repeat this calculation for the compute pod's RAM request (and limit) compared to what's available on the node as well. Keep in mind that container requests are simply initial guesses. Actual resource usage could be less or much, much more (up to the optionally defined limit).   Job maximums Besides the maximum number of jobs defined per host from all queues, check out the other maximum job values that can be defined in SAS Workload Orchestrator as attributes of the queues themselves: Maximum jobs for this queue Maximum jobs per user in this queue Maximum jobs per host in this queue By default, they're not defined. However, if values are provided, then those numbers are somewhat arbitrary, informed only by the Viya administrator's own understanding of workload attributes. So, they might be a limiting factor, or not.   Other SAS attributes Consider other aspects of SAS Workload Management like schedule or suspend thresholds based on attributes like pgRate, swap, temp, runQueue15s, and so on. It's possible to configure SAS Workload Orchestrator to take these into consideration when it submits jobs to run in the cluster. Plus, general operation of SAS Viya as well as configuration of client components can affect the number of jobs that are allowed to run as well.   Max_Pods Investigate the MAX_PODS for the Kubernetes nodes assigned as part of the "compute" workload class and their associated infrastructure (i.e., instance type, CNI, etc.). This is a hard limit but still could be (and likely should be) higher than the effective limit contributed by other factors. Remember that the accepted Kubernetes default is 110 pods per node, however for cloud-provided Kubernetes cluster, this value is typically set based on physical resource capacity (like instance type size and the number of network connections it can handle).   While we’ve touched briefly on a wide range of topics and considerations that have notable bearing on the raw number of possible SAS Compute (and other dynamically-launched processing pods of the SAS Programming Runtime Environment) that can run in SAS Viya, we have not looked at an exhaustive list of all factors. Having an awareness of how Kubernetes and the SAS Viya platform operate along with familiarity with the specifics of the workload is very helpful in tuning the environment to suit the needs of your users.   Here's an attempt to summarize all of these concepts into one pithy statement:     In short, look across the various constraints defined for your environment to determine which has the final say in limiting the number of SAS analytic jobs that can run on a node. Even then, we're just making a best guess from an initial set of assumptions and configuration. The actual execution of jobs will likely vary from the theoretical limit depending on the resources consumed as work progresses. If a node succumbs to resource pressure, then Kubernetes will begin evicting pods based on their quality of service.   Determining the possible number of Compute jobs is a useful metric calculate to help ensure things are running smoothly and efficiently in your environment. However, it's hard to nail down a perfect number because Kubernetes is designed to be fluid in managing node resources. SAS Workload Management provides extra controls to best define the kind of jobs that SAS Compute will demonstrate, but it's still subject to Kubernetes and the infrastructure's capabilities to get the job done.   Coda   Wrapping up this series, we've seen that SAS Viya offers a variety of analytic engines which interact with Kubernetes and the underlying infrastructure in different ways.   We started off looking at CAS and learned that its default configuration will allow just one CAS pod (SMP or MPP, controller or worker, but not including the personal CAS server) per Kubernetes node because each pod is defined to request a majority of the node's CPU and RAM. For the high-performance, in-memory analytics work that CAS performs, this is usually the recommended approach to ensure efficient delivery of results.   For the pods that comprise the various runtime instantiations of the SAS Programming Runtime Environment, we found that SAS Workload Management has a huge part to play in providing flexibility to define how SAS Compute jobs are set loose to run in the environment. Out of the box, the default is to allow almost "unlimited" jobs (configured as Maximum Jobs Allowed per Node = 250) so that new deployments aren't artificially constrained and inadvertently underutilize the system.   There's a wide range in variability in terms of what a given SAS Compute job might do, from tiny queries or calculations completed in a milliseconds to large volume and/or computationally intense tasks that might run for hours. Understanding the expected workload in combination with the resource capacity the infrastructure provides is key to ensuring the system performs at optimum levels for efficiency and cost.   H/T   I'd like to tip my hat in gratitude with warmest regards to several people who helped explore this topic with me: David Stern, Scott McCauley, Edoardo Riva, Raphaël Poumarede, Joe Hatcher, Craig Rubendall, Seth Heno, and Doug Haigh. Any mistakes are my own, not theirs. 😉     Find more articles from SAS Global Enablement and Learning here. ... View more

Since August of 2023, the SAS Viya platform now includes the SAS Workload Management offering and activates it by default at initial deployment. If you're familiar with grid computing from SAS 9, then you're well aware of the powerful capabilities that this brings to SAS Viya. Now the concepts of jobs, queues, priority, and preemption can be offered at the user level for SAS Viya, and are no longer exclusively in the domain of the Kubernetes administrator.   This post is the second in a three-part series where we investigate how many SAS Compute (as a representative example of the SAS Programming Runtime Environment) can run in the Kubernetes cluster. Continuing from where we left off, we'll now look at the SAS Workload Orchestrator and how it functions to manage the SAS Compute workload.   In this post, we'll often look specifically at the SAS Compute Server pod. The items discussed here apply equivalently to the other related SAS Programming Runtime Environment servers, like SAS Batch and SAS/CONNECT.   [ Part 1 | Part 3 ]   Schedulers, briefly   Kubernetes includes a control-plane process known as kube-scheduler. It's a Kubernetes Controller responsible for assigning pods to nodes. It evaluates numerous attributes of the pods and nodes, ranks the best fit, and then binds the pod to the target node.    Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Effectively, there are two axes of considerations for scheduling pod placement:   Affinities: this covers everything that determines how pods relate to nodes and each other, including labels (and node selectors), taints (and tolerations), pod affinities (and anti-affinities), and so on. Quality of Service: Kubernetes has defined levels of QoS (Guaranteed, Burtstable, and BestEffort) but I’m also including other items under the umbrella that inform QoS such as pod requests and limits for CPU and RAM as well as the available host resources which Kubernetes monitors (like storage space, CPU, RAM, etc.).   With SAS Workload Management, we have additional scheduling capabilities to apply to Viya's dynamically-launched processing pods, like SAS Compute Server.     This provides a third axis:   SAS Workload Management: introduces the concepts of jobs, queues, priority, and pre-emption to the Kubernetes realm as applied to SAS Compute Servers (and other SAS Programming Runtime Environment servers like SAS Batch and SAS/CONNECT). It also provides a way to specify and track resource utilization and establish thresholds and other factors that impact how jobs are scheduled to various hosts.   One nifty aspect of Kubernetes is that extensibility is provided – and support for custom pod schedulers is included. SAS Workload Management provides the SAS Workload Orchestrator to act as the scheduler for dynamically-launched processing pods, like SAS Compute Server. In a Kubernetes cluster where SAS Workload Management is active, then SAS Workload Orchestrator is responsible for scheduling all of the SAS Compute Server pods independently of the kube-scheduler.     SAS Workload Orchestrator is constrained in two fundamental ways:   It only manages Viya's dynamically-launched processing pods of the SAS Programming Runtime Environment as instantiated by SAS Compute, SAS Batch, and SAS/CONNECT Servers as well as other pods running Python or R as started by the Batch service. That is, it doesn't schedule any other Viya pod workloads (as of stable-2023.11, subject to change).   It can only schedule pods to run on nodes labeled as "workload.sas.com/workload=compute"   For these reasons, SAS Workload Orchestrator implements its own logic as defined by jobs, queues, priority, etc. when scheduling SAS Compute Server pods to the "compute" nodes of the Kubernetes cluster. Unless otherwise directed, it effectively ignores the typical Kubernetes customization attributes such as node labels and taints, pod affinities (and anti-affinities), and so on.   It must, however, handle the pod and node attributes that correspond to physical resources. This means that SAS Workload Orchestrator will evaluate the total composite resource requests a dynamically-launched processing pod defines for CPU and RAM. It is also subject to infrastructure constraints such as the maximum pods allowed per node (configured for the kubelet as driven by the cloud-provider's managed Kubernetes service - in AWS, for example, this ranges from 4 to 737 pods depending on instance type).   Illustrating the SAS Compute Server pod   The items discussed here apply equivalently to the other related SAS Programming Runtime Environment servers, like SAS Batch and SAS/CONNECT.   Like other pods, the sas-compute-server pod includes both init containers and regular containers to perform its function.   Init containers in sas-compute-server pod:   sas-certframe: sets up encrypted communications for this pod with the rest of SAS Viya sas-config-init: prepares the environment for the main container   Regular containers in the sas-compute-server pod:   sas-programming-environment: this is the main container and why this pod exists - it provides the runtime engine for executing SAS program code sas-process-exporter: this is a sidecar container that helps out pod operations. It only exists when SAS Workload Management is active such that this container is injected into the pod by SAS Workload Orchestrator when it's created to track key metrics and other information.   Notice that each container shown here is defined with requests and limits for both CPU and RAM. For pods in general, that's not required (some pods might have no requests or limits defined at all), but is helpful to ensure that the sas-compute-server pod can be managed intelligently with the available environmental resources. The values shown above are the defaults, but can be modified to suit the needs of your environment.   Adjusting the SAS Compute Server's requests for CPU and RAM   SAS Environment Manager provides a path to edit the requests and limits for CPU and RAM for SAS Compute (and other SAS Programming Runtime Environment servers like SAS Batch and SAS/CONNECT) through the SAS Launcher configuration.   Open SAS Environment Manager > Configuration > View: All Services > select Launcher service > Edit sas.launcher.default Configuration:     Here you can see the defaults that will be used for requesting (and limiting) CPU and RAM usage by the sas-programming-environment container in the associated pod for SAS Compute, SAS Batch, or SAS/CONNECT. In this example, the container will request 50 millicores of CPU initially with a limit of 2 CPU cores and it will request 300 Megabytes of RAM initially with a limit set at 2 Gibibytes. Alternatively, advanced configuration of these values (for launcher contexts and associated compute contexts) can be made to the podTemplate.   SAS Compute's variable workload   The kind of work that dynamically-launched processing pods, like the SAS Compute process, can perform is widely varied. Depending on the task it's assigned, it might perform any kind of work from very tiny to very large. It might simply sit idle waiting for a new task or it might crunch through large data sets applying specialized analytics. Sometimes it might run auto-generated code as directed by a SAS Viya client, like SAS Model Studio, or it could run user-generated ad-hoc code submitted from SAS Studio or scheduled as part of a batch process. If using MP CONNECT technology, then one SAS Compute could programmatically spawn off a number of additional SAS Compute as child processes that run serially and/or in parallel.   This means setting the initial requests (and optionally, limits) for the SAS Compute pod can be challenging to fine-tune to a perfect ideal. Unless you're dealing with a well-understood repeatable procedure, then it's generally recommended to embrace this variability in a few ways:   Set requests that start small, but allow for growth as needed (with high limits or none at all). Setting requests for containers in the SAS Compute pod to achieve the Kubernetes Burstable quality of service is more flexible and more effectively ensures sufficient resource utilization.   Define compute contexts and partner with users on their appropriate application such that SAS Compute pod definitions are better tuned for the expected workload.    Employ SAS Workload Orchestrator's job, queue, and host attributes to schedule jobs to hosts best suited to the task(s) at hand.   One more thing when it comes to the sas-programming-environment pod. As the SAS runtime, it is internally configured by a number of SAS system options, and of particular interest here: MEMSIZE. MEMSIZE can be set in SAS Environment Manager as an invocation option that specifies the limit on the total amount of memory that the SAS session will try to use. The default value is 2 GB (hence the matching limit of 2 GB for the sas-programming-environment pod above). When planning for Kubernetes resource requests and limits, it’s recommended when changing these settings to keep them consistent to their purpose at all levels relative to each other (in the SAS runtime, in the SAS pod spec, in Kubernetes, and in infrastructure to host it).   Coming up   In the last post of this series, we'll figure out how many SAS Compute Servers can run on a given node and what the constraining factors are. H/T   I'd like to tip my hat in gratitude with warmest regards to several people who helped explore this topic with me: David Stern, Scott McCauley, Edoardo Riva, Raphaël Poumarede, Joe Hatcher, Craig Rubendall, Seth Heno, and Doug Haigh. Any mistakes are my own, not theirs. 😉     Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Viya is all about delivering powerful analytics capabilities while Kubernetes monitors and manages the environment so that Viya pods are scheduled to run on the optimal nodes and ensures that host machines don't become over-saturated with work.   The SAS Viya platform is designed to work in harmony with Kubernetes to deliver analytics in a robust and scalable way. And in particular, Viya employs a variety of analytic engines to accomplish this goal. Each analytic engine has a unique usage pattern for resources and we can customize the deployment to optimize for best results. For this post, we will focus on the SAS Cloud Analytic Services and then later look at the SAS Programming Runtime Environment, but note that similar concepts can also be explored for the SAS Micro Analytic Service, SAS Event Stream Processing, and so on.   Have you ever wondered how many pods of a runtime engine like CAS or Compute can run on a Kubernetes node? How is that determined? What changes can you make? What do we need to consider when trying to trying to match the workload to the available resources?   This is a three-part series where we take a swing at answering those questions. In this first post, we begin with looking at CAS and the unique considerations that a high-performance, in-memory analytics engine requires. The next post will expand on new capabilities introduced by SAS Workload Management and the impact that has on SAS Compute.  Then the last post will wrap up with evaluating the Kubernetes nodes to determine how many pods for runtime engines they can actually support running.   [ Part 2 | Part 3 ]   Containers, briefly   A container is intended to provide a lightweight, standalone, executable image of software. It starts with a base layer providing a basic operating system and then additional software is layered on as the container is built so that everything needed is there to run.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Containers offer a very flexible approach to running software with improved management and operation. However, the number of containers and their orchestration can become burdensome to maintain. That's where Kubernetes comes in - to provide a powerful framework for managing containerized workloads.   SAS Viya software is packaged into many container images which are downloaded to the Kubernetes cluster when needed for installation and execution.   Pods, briefly   The smallest unit of work in a Kubernetes cluster is a pod. A pod consists of one container or more that work together. The pod definition specifies which containers to include, their high-level role in the pod lifecycle, and optionally resource access and utilization.   From Kubernetes' perspective, there are two main classifications of containers in a pod: Init Container(s): zero or more, these containers are run first, in sequential order, when the pod starts up. Each must complete successfully before the next can run. Containers (a.k.a., app containers, regular containers): one or more, these containers perform the intended task of the pod. They start after all the init containers have completed and can run concurrently in perpetuity.   Requests and Limits, briefly   From a resources perspective, the pod definition might specify options for resource utilization like persistent volume claims for access to storage as well as the amount of RAM or CPU that's expected to be used. Specifically for RAM and CPU, Kubernetes allows for the possibility to specify requests and/or limits per container in the pod.   Requests are much like calling ahead to make a reservation for 4 people at a restaurant. The actual number of people that show up might be 4, or fewer, or more, and the restaurant will try to accommodate them. So a container's request for CPU or RAM is kind of a best guess as to what it might need, but it could end up using less or more at any given moment. Think of a SAS Compute Server that backs a user session in SAS Studio. Until the user submits program code, it's likely idle and not consuming anywhere near the requested resources. And then depending on the program tasks, it could soon consume a lot more resources than the initial request.   Limits put a cap on the amount of resources the container can use. When a CPU limit is reached, then the container's access to CPU is throttled to keep it in bounds while it continues to run. RAM limits, however, are treated very differently. When a RAM limit is exceeded by the container, then the OS can invoke the OOM (out of memory) Killer process. The OOM Killer will terminate the container which leads to the associated pod getting deleted from Kubernetes. Depending on factors, a new pod might be started to replace it. For a SAS Studio user, this means the unexpected termination of their program execution and loss of that session.   Most pod definitions for SAS Viya will specify some request and/or limit for at least one container.   Illustrating a CAS pod   Let’s take a quick look at the executable components of a SAS Cloud Analytic Services controller pod. Like other pods, the sas-cas-server-default-controller pod includes both init containers and regular containers to perform its function.   Init containers in the sas-cas-server-default-controller pod: sas-certframe: sets up encrypted communications for this pod with the rest of SAS Viya sas-config-init: prepares the environment for the main container   Regular containers in the sas-cas-server-default-controller pod: sas-cas-server: this is the main container and why this pod exists - it provides the in-memory analytics engine that makes this piece of MPP CAS work sas-backup-agent: this is a sidecar container to help with regularly scheduled backups. sas-consul-agent:  this is another sidecar container allowing CAS to work with the SAS Configuration Server Notice that each container shown here is defined with requests and limits for both CPU and RAM. For pods in general, that's not required (some pods might have no requests or limits defined at all), but for Viya, this is very helpful to ensure that the sas-cas-server-default-controller pod can be managed intelligently with the available environmental resources. The values shown above are the defaults, but can be modified to suit the needs of your environment.   We've dug in deep with the CAS controller pod here, but note that CAS worker pods follow the same approach.   Kubernetes looks at request values Kubernetes evaluates numerous factors to determine where to run a pod in the cluster. For this post, because we're ultimately working to figure out how many runtime pods can exist on a node, we'll focus on the request values for CPU and RAM that might be defined for the containers in a pod.   Kubernetes refers to the composite of request values for CPU or RAM (if they are defined) for the pod. Because init containers run sequentially to completion, Kubernetes only considers the highest request value for CPU or RAM across all init containers. For the regular containers, which can run concurrently for an arbitrary amount of time, it looks at the combined values requested for CPU or RAM. The highest composite request value across init and regular containers then is used as part of the determination for where to place the pod. This helps ensure that each pod has sufficient CPU and/or RAM resources.   Now let's take a closer look at the CAS controller pod to evaluate its containers' requests for CPU and RAM.   For the init containers in the CAS controller pod: $ kubectl get pod sas-cas-server-default-controller -o yaml | yq eval '.spec.initContainers[] | .resources.requests'   cpu: 500m memory: 500Mi cpu: 500m memory: 500Mi   The highest request for CPU of the init containers is 500 millicores. The highest request for RAM of the init containers is 500 mebibytes.   And for the regular containers in the CAS controller pod: $ kubectl get pod sas-cas-server-default-controller -o yaml | yq eval '.spec.containers[] | .resources.requests'   cpu: "5" memory: "22261674803" cpu: 100m memory: 50Mi cpu: 100m memory: 50Mi   The combined request for CPU across all regular containers is 5.2 cores ( =5 + 0.1 + 0.1 ). The combined request for RAM across all regular containers is 20.8 gibibytes ( =20.7 + 0.05 + 0.05 ).   So, when looking at the composite of all requests for resources, it will seem as if Kubernetes is treating the sas-cas-default-controller pod as requesting 5.2 cores of CPU and 20.8 gibibytes of RAM. It will only schedule the CAS Controller to a node with sufficient available resources (and in line with its various other considerations, like labels and selectors, taint and tolerations, etc).   Remember, these request values are just a starting point. The containers involved could actually use less or more of CPU and RAM (up to a defined limit, if provided) during their operation.   CAS default   You might be wondering why so much RAM and CPU for one of those containers in the CAS controller's pod? And that memory request is a very precise value, too - not a simple round number like " 20 GB ". That container is the actual CAS controller, of course. And it makes those resource requests because Viya defaults to an approach referred to as "auto-resourcing" such that it requests a majority of a node's CPU core count (half + 1 ) and RAM ( ~65% ) for each CAS instance (controller or worker).   In this example, the host for the CAS controller is an AWS EC2 m5.2xlarge with 8 CPU and 32 GiB of RAM. The idea is to ensure that each CAS instance (controller or worker) will request enough resources that Kubernetes won't try to place two of them together on a single node. CAS was designed with performance as its primary objective, so we expect to place each CAS pod on a dedicated node of the Kubernetes cluster. That is, the typical Viya deployment is configured to request a majority of CPU and RAM resources for the CAS pod so that just one will run per node for either SMP or MPP deployment of the CAS server (this concept does not apply to the personal CAS server).   You're not stuck with the default auto-resourcing approach. It is possible to specify your own requests and limits for CAS, if desired. The SAS documentation recommends setting the request equal to the limits for both CPU and RAM values so that Kubernetes will apply the Guaranteed quality of service to the CAS pod, helping to give it priority over other pods and preventing it from being evicted from the node in over-pressure situations. For more information about this, see:   SAS® Viya® Platform Operations Common Customizations > Adjust RAM and CPU Resources for CAS Servers   Coming up   In the next post of this series, we'll take a closer look at SAS Workload Management and its impact on the SAS Compute Server pods. In particular, we'll dive into the environmental aspects that affect the operation of SAS Workload Orchestrator as the scheduler for SAS Compute pods. ... View more

SAS Explore 2023 was held at the Aria Hotel and Resort in Las Vegas in early September. Even though the MGM properties were valiantly working through the aftermath of a hack while we were there, the SAS Explore conference seemed to go off without a hitch. For many of us in the SAS community, this was a first opportunity to see each other in person again. I'm thankful I could go and really enjoyed catching up with so many colleagues and friends.   I delivered a couple of messages for my part while I was there. The first was a presentation, The Intersection of Workload Placement Between Kubernetes and the SAS® Viya® Platform.     The objective was to help bridge the gap for folks with either SAS Viya or Kubernetes experience by focusing on how workload is handled by the various responsible services.   The presentation began with briefly looking at Kubernetes concepts like pods, services, nodes, node groups, autoscalers, labels and taints and then progressed into SAS Viya. In particular we looked at the four default workload classes for SAS Viya and how they're handled.     From there the presentation dove just a little deeper into the SAS Workload Management offering (included for free as part of the SAS Viya platform since August, 2023) as well as how to improve concurrency performance. We also briefly touched on the techniques that the SAS Cloud Analytic Services use to distribute workload across multiple worker hosts.   The next day I delivered a hands-on workshop, Delivering Powerful Analytics Capabilities with the SAS® Viya® Platform and Kubernetes. While the title was different than the presentation, the objective was essentially the same: to help introduce Kubernetes and SAS Viya concepts around workload placement.   This guided exercise provided each attendee with their own personal SAS Viya deployment running in Kubernetes. It began with a tour of the tools and capabilities that a SAS (and/or Kubernetes) administrator would ideally have access to for managing a SAS Viya deployment. For example, we demonstrated how each SAS Studio session requests its own backend SAS Compute Server.     From there, we talked about extending this concept for many users running at the same time and multiplied the concurrency yet again by showing SAS Model Studio running multiple pipelines at once. As covered in the presentation the day before, we modified the environment to better handle the increased concurrency so that the resources in the environment could be efficiently utilized.   The hands-on then switched gears to look at SAS Cloud Analytic Services (CAS). As you know, CAS is the high-performance, in-memory analytics engine that is designed to process the largest volumes of data as fast as possible. And the exercise was to convert CAS from running as a single-node server (SMP mode) to running as a single service distributed to run across multiple host machines (MPP mode).   The trick on that exercise is that the demo environment we were using wasn't intended to run MPP CAS. So we had to hack at the settings and essentially cram MPP CAS to run in an environment that was too small. The lesson there was to show how Kubernetes and SAS Viya configurations work together to keep things running efficiently and effectively. By forcing Kubernetes to allow MPP CAS to run, we could show exactly how those settings worked. The takeaway then is to understand why we configure SAS Viya to run in Kubernetes and how to plan for proper sizing in a real deployment.   I'm looking forward to the next conference coming up in 2024: SAS Innovate. In the meantime, if you're looking for more SAS Explore 2023 content, including the presentation materials, it's now available at:   SAS Communities > SAS Explore >   SAS Explore 2023 Presentations   https://communities.sas.com/t5/SAS-Explore-Presentations/tkb-p/SAS_Exploretkb-board     Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Viya relies on Kubernetes to provide a robust environment that is elastic, highly available and extensible. One way that Kubernetes achieves these goals is through sophisticated workload management controls. By default, SAS Viya takes advantage of these controls in specific ways to ensure a great software experience. But there’s no one-size-fits-all way to deploy SAS Viya. That's why understanding these controls can be useful to further fine-tune the environment to your site’s specific needs. Configuring Kubernetes for workload placement works well up to a certain point. But there are areas where Kubernetes can't “see” what’s happening. Instead of trying to manage the workload at the infrastructure level, configuration needs to occur higher at the application level. In SAS Viya, we have a multilevel approach with many levers and switches to configure workload placement, including multitenancy, CAS distribution of data across nodes, SAS Workload Management jobs controlled through the use of queues, priority and preemption, and additional SAS Viya platform controls for concurrency and resource utilization. ... View more

Carrying on with our larger series about scalability of SAS Viya, this post continues looking at scaling the SAS Cloud Analytic Services. In part 1, we looked specifically at using the viya4-deployment project to change the number of workers for MPP CAS. In this post, we'll consider a change that requires a planned outage by scaling the host machines for CAS to run on larger instance types with more CPU and RAM.   Review   We've already covered ground regarding a number of salient points to consider in the previous post. So please refer back there for more explanation about:   The IAC and DAC projects The difference between scaling out and scaling up Partnering with the SAS World-Wide Sizings Team for specific machine guidance How to use the DAC to scale out the number of MPP CAS workers   We'll continue to refer to this example deployment of SAS Viya: Select any image to see a larger version. Mobile users: If you do not see this image, scroll to the bottom of the page and select the "Full" version of this post.   We're still looking at CAS on the left of the illustration, but in this post we will scale up the CAS server by increasing the size of the instance types used for the CAS node pool to give them more CPU, RAM, and other resources as defined by the infrastructure provider.   IAC = Terraform (for this blog post)   The IAC project relies on a number of utilities, some of which are specific to the intended cloud provider used by your site. But in all cases, the IAC uses Terraform for provisioning the infrastructure components. So while this blog post highlights the Infrastructure as Code project, the most critical steps are performed using Terraform.   The point is, if your site uses Terraform (but not the IAC), then this post is still relevant to you.   Scaling up CAS with the IAC   Changing a virtual machine's instance type will necessitate shutting down the VM and restarting it on the new hardware. In this case, we have a node pool of four VMs for the CAS server. By definition, all of the hosts in a node pool rely on the same instance type. So by changing the instance type associated with the node pool, we will be required to restart all four VMs for CAS.   The CAS server is designed to be scalable and highly available - but it cannot tolerate this level of massive change at the lowest level without taking excessive care. It's far easier to plan an outage to take SAS Viya offline so that we can make this change quickly and efficiently.   That said, we can use the IAC project to change the instance type for the CAS node pool. This works whether we're running CAS in SMP mode or in MPP mode. And it's pretty easy...   What's changing?   The IAC can be configured to deploy AWS node groups (a.k.a., cluster node pools) for SAS Viya:   ## Cluster Node Pools config node_pools = { cas = { "vm_type" = "m5d.2xlarge" "cpu_type" = "AL2_x86_64" "os_disk_type" = "gp2" "os_disk_size" = 200 "os_disk_iops" = 0 "min_nodes" = 4 "max_nodes" = 5 "node_taints" = ["workload.sas.com/class=cas:NoSchedule"] "node_labels" = { "workload.sas.com/class" = "cas" } "custom_data" = "" "metadata_http_endpoint" = "enabled" "metadata_http_tokens" = "required" "metadata_http_put_response_hop_limit" = 1 },   In the example for this blog post, the vm_type specified at initial deployment of SAS Viya was m5d.2xlarge . Now we want to increase the number of CPU and amount of RAM on those CAS host machines.   Stop SAS Viya   Changing the instance type of a node pool will require shutting down the old nodes and bringing up new ones with the desired instance type. The CAS server doesn't tolerate this kind of change and should be taken offline first. Might as well take all of SAS Viya down with it to be safe.   Backup SAS Viya (just in case): See Retaining your SAS Viya Backup Take SAS Viya offline: Execute the sas-stop-all cron job:   NS={{ your_SAS_Viya_namespace }} kubectl create job sas-stop-all-`date +%s` --from cronjobs/sas-stop-all -n ${NS}   Change the instance type for CAS in the tfvars file   We saw an example of the CAS node pool definition in the configuration file for Terraform above.   Logon to the host machine where the initial deployment of SAS Viya was managed so we can use the IAC tools and configuration there. Edit the tfvars file and specify the new desired instance type for the CAS node pool. In this example, we'll request:   "vm_type" = "m5d.4xlarge"   Note: It's interesting that cloud providers differentiate cost based on CPU count within a family. If you consider that an m5d.4xlarge provides twice the CPU and RAM of a m5d.2xlarge and that it also costs twice as much, then there's a lot of incentive to consider scaling out instead. Another point to consider is that the networking doesn't keep pace with CPU - so running more smaller hosts rather than fewer larger hosts likely has a cumulative network throughput benefit. In addition to those points, the prior blog post demonstrated that scaling out MPP CAS can be done without service interruption of SAS Viya.   Make it happen   To put this configuration change into effect and specify a larger instance type for the CAS server, we first need to direct Terraform to plan the change to the instance type of the CAS node group.   $ terraform plan -input=false \ -var-file=/path/to/sasviya.tfvars \ -state /path/to/sasviya.tfstate \ -out /path/to/sasviya.tfplan   If that step completes successfully, then it should return a lot of output and near the end say something similar to:   Plan: 2 to add, 2 to change, 1 to destroy.   Next, we can direct Terraform to apply the updated plan to the cluster:   $ terraform apply -state /path/to/sasviya.tfstate \ "/path/to/sasviya.tfplan"   This takes several minutes to complete.   Monitor the newly instantiated nodes for the CAS node pool as they come online and reach Ready state in your environment.   Resume SAS Viya operations   After the node pool has been updated with hosts using the new instance type, then return SAS Viya to normal operations: Execute the sas-start-all cron job:   # NS={{ your_SAS_Viya_namespace }} kubectl create job sas-start-all-`date +%s` --from cronjobs/sas-start-all -n ${NS}   After the CAS server resumes its normal operation state, then reload any desired tables, formats, and other data as needed.   Alternatives   Using the IAC project is optional. Ultimately, the customer site is responsible for provisioning and maintaining the hardware environment for SAS Viya. Some sites will already have mature, standardized practices using their own tools that don't rely on Terraform.   As background for this post, I ran through two alternative approaches in the Amazon Web Services environment for scaling up MPP CAS:   Using the AWS Console to point-and-click in a web browser (doc) Using the AWS CLI utility for a programming-based devops approach (doc)   If those are of interest to you, drop me a line and I'm happy to share my notes.   Looking for more information?   If you're looking for more details about scaling out instead of (or also with) scaling up, then look back to Part 1 of this series where we show how the DAC project can change the number of host machines running the CAS server.       Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Cloud Analytic Services (CAS) is designed from the ground up to harness the power of many resources over multiple machines to scale and attack the largest analytic challenges. But scaling CAS is not (yet) something that can happen elastically (that is, automatically). The starting point is to pick a size and shape for your CAS server deployment and get things running. From there, monitor the environment and adjust the resources for CAS accordingly. In particular, we can choose to scale CAS up (by adding more CPU and RAM to the existing host machines) or out (by adding more machines that host the CAS server). Either way, there are follow-on choices and further implications to consider.   The IAC and DAC projects Typically we see SAS Viya running in a managed Kubernetes cluster hosted by an infrastructure provider. SAS offers optional tools to help achieve this objective. Two in particular are relevant for this post:   Infrastructure as Code The Infrastructure as Code (IAC) project is intended to guide our customers in provisioning hardware appropriate for SAS Viya. Its use is optional but can be very helpful for getting started quickly for demos, proof of concepts, initial test environments, etc.   The IAC project is available in four flavors on Github depending on a site's infrastructure preference: viya4-iac-aws: for Amazon Web Services GitHub.com > sassoftware > viya4-iac-aws viya4-iac-azure: for Microsoft Azure GitHub.com > sassoftware > viya4-iac-azure viya4-iac-gcp: for Google Cloud Provider GitHub.com > sassoftware > viya4-iac-gcp viya4-iac-k8s: for upstream open-source Kubernetes GitHub.com > sassoftware > viya4-iac-k8s   Deployment as Code The Deployment as Code (DAC) project is another offering from the IAC team. It is intended to guide customers with installing SAS Viya. Its use is also optional but it too is very helpful for getting started quickly.   The DAC project is available on Github, too: viya4-deployment: for any environment that meets requirements GitHub.com > sassoftware > viya4-deployment   If you have the IAC and DAC tools in your environment already, then they can streamline the configuration changes discussed here. Alternatively, you could use tools native to your site's infrastructure provider.   SAS Viya The configuration of SAS Viya is extremely flexible to support a variety of infrastructure requirements. And the IAC/DAC projects surface many of those configuration options. By default, an out of the box deployment of SAS Viya might look similar to:     For this post we're primarily interested in the far left side of the illustration where CAS is shown. This picture shows CAS deployed in MPP mode (one logical server running across multiple hosts). The CAS pods are labeled for "cas" workload and they correspond to a node pool that's been set aside as labeled (and tainted) for "cas" as well.   The colorization of the CAS nodes is meant to visualize the min, max, and desired values for the number of active nodes which are part of the auto scaling group definition and used by the cluster autoscaler to bring up (or take down) nodes on demand. In this case, four nodes are green (desired=4) to reflect that we configured the installation of MPP CAS to run with one controller and 3 workers.   As a reminder, the default approach is for each CAS node to run just one CAS pod. That's because CAS is a high-performance, in-memory analytics engine that consumes substantial CPU and RAM to return results as fast as possible.   And finally, this illustration only shows a portion of what your SAS Viya deployment might look like when hosted in a managed cloud environment. Many other components are not shown here. And chances are your site will vary from this simplified example in several ways.   Scaling Up vs. Scaling Out Scaling up means providing more CPU and RAM on the host - that is, to run on larger machines with a higher number of CPU cores and more RAM. To accomplish this in a managed cloud environment will mean changing the instance type for the CAS node pool. This will require terminating the current hosts and then replacing them with new ones which specify the desired instance type. In other words, scaling up by changing the instance type will necessitate an outage of SAS Viya.   Scaling out occurs when we add more host machines to process the workload. This can be accomplished with very little impact to end users as it typically means changing the desired number of nodes to a larger value (and the max as well, if that's the objective). While there's no service outage for SAS Viya, it can take several minutes for the new node to provision, containers to download, pods to spin up, and so on.   For an MPP deployment of the CAS server, both scaling up and scaling out are acceptable. We can add more workers to the CAS server and/or increase the size of each of the CAS server hosts.   Determining the exact instance type and number of nodes suitable for your site's workload are beyond the scope of this blog post. For official sizing guidance of SAS Viya deployments, we recommend working with the World-Wide Sizings Team. To do this well, the sizing effort is necessarily a collaboration between SAS and the customer. The Sizings team can help by asking and answering some critical questions as well as working with the customer to set expectations and devise a plan of attack to ensure scalability goals are met. For customers looking for help with getting the right hardware to run their SAS Viya workloads, your SAS account representative can begin an engagement with the World-Wide Sizings Team.   For the purpose of this post, we're starting with MPP CAS with 3 workers that's running on AWS instance type m5d.2xlarge (8 vCPU and 32 GB of RAM) in AWS.   Scaling out CAS with the DAC Adding more workers to the CAS server can be done without an interruption in service of SAS Viya. This means more analytic computation and data capacity can be rolled out for better overall performance in a live environment.   We can use the DAC project to change the number of CAS workers. It's really easy:   Edit the installation variables file used by the viya4-deployment project to specify the desired number of CAS workers: ## Update MPP worker count to 4 V4_CFG_CAS_WORKER_COUNT:'4' Direct the viya4-deployment project to run again and apply the change: $ viya4-deployment --tags "viya,install" # Note the 'baseline' tag is not needed for updating Viya config # Also, in my environment "viya4-deployment" is an alias referencing the ansible utility with necessary parameters to run as a container. If you follow along in OpenLens, then after the viya4-deployment is finished, you should see a new cas-worker pod started:   The alert triangle has a scary - but temporary - warning:     This happens because, even though the cluster autoscaler has fired up a new EC2 instance for this CAS worker pod, it's just not ready yet because it's still initializing:     Note: if a new node for the CAS worker is not initialized, confirm your auto scaling group hasn't reached its max value for node count. And also that the cluster autoscaler is running and behaving properly.     Be patient and eventually Kubernetes will notice the new CAS node is ready and then the CAS worker pod will finish its startup and join the CAS server.   Start up a CAS session in SAS Studio and it should report it's now using 4 workers: 80 cas; NOTE: The session CASAUTO connected successfully to Cloud Analytic Services sas-cas-server-default-client using port 5570. The UUID is e235fe0f-2b68-0548-bb8b-827242e2f484. NOTE: The SAS option SESSREF was updated with the value CASAUTO. NOTE: The SAS macro _SESSREF_ was updated with the value CASAUTO. NOTE: The session is using 4 workers.   Important to note:   Scaling out the number of CAS workers does not require a planned outage. However, as of August 2023, scaling in the number of CAS workers does.   After the new worker joins the CAS server, then existing data and sessions are unchanged. New CAS sessions created after scaling out will use all workers and new data loaded to CAS will distribute to all workers. It is possible to run code to redistribute existing data across all CAS workers.   Alternative: If your site is deployed or maintained without relying on the DAC, then SAS documentation explains how to manually change the number of workers for MPP CAS.   The upcoming Part 2 of this post will discuss more details about scaling up instead of (or also with) scaling out where we show how the IAC project can change the instance type of the CAS host machines. ... View more

Management of Kubernetes workload is often focused on how pods are scheduled to nodes of the cluster. Factors which affect those assignments include the use of labels (and node selectors), taints (and tolerations), pod (anti-)affinities, requests, limits, and more. At another level, SAS Viya Workload Management extends the ability to manage workload into the user domain with the concepts of jobs, queues, resources, and more.   Now let's take a step back and review how the infrastructure that was provisioned for SAS Viya has an impact on the distribution of workload in the Kubernetes cluster.   As discussed in earlier posts, Kubernetes is responsible for distributing jobs across the cluster and SAS Workload Management further extends this capability to accommodate the unique needs of the SAS Viya platform. As powerful as they are, they can only act by utilizing the resources that are provided by the infrastructure.     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The infrastructure itself can vary widely. The variability across the major cloud providers is notable, but sites also have the option of providing hardware through their own data center, subject to their own standards and practices. For this post, we'll focus on the idea of a managed cloud-based deployment of SAS Viya.   Instance Type > Node   Provisioning host machines in a cloud environment is typically done by specifying the desired instance (or machine) type. That is, a classification of machine with the desired CPU, RAM, disk, I/O throughput and other capabilities or features. Once the host machine is provisioned as part of the Kubernetes cluster, it's referred to as a node.   Instance types come in a range of sizes and capabilities. Some are optimized for general use, whereas others prioritize database, CPU, memory, disk, and/or network I/O. This allows the opportunity to provision machines best suited to your workload.   From a Kubernetes perspective, the amount of work a given node will be asked to handle is managed in several ways. For example, Kubernetes defaults to a maximum of 110 pods per node. In a cloud-managed environment, that limit is often much lower, based on the size of the instance type as well as optional enhancements that might be in play, like which container network interface (CNI) is installed.   Another example for SAS Viya is if automatic resources are enabled for CAS. If so, then each CAS pod (either worker or controller) will request the super majority of CPU and RAM resources on a node. This ensures that each node is dedicated for use by a single CAS pod with the goal of achieving best possible in-memory analytics performance.   Selecting the appropriate size instance type to host your Kubernetes cluster involves numerous considerations and should be undertaken with care. For customers looking for help with getting the right hardware to run their SAS Viya workloads, their SAS account representative can begin an engagement with the World-Wide Sizings Team.   Nodes > Node Pools   A node pool is essentially a group of nodes in the cluster with the same resources and configuration (that is, the same instance type). One aspect of their design is to provide a definition of scalability. Defining a node pool allows for multiple machine instances to run in support of a specific need. Node pools can be defined to have a minimum, maximum, and desired number of nodes running. These values set the bounds for scaling the number of machines running to have enough for any given demand.   Scaling up a single machine instance to have more CPU, RAM, etc. typically requires an outage. Alternatively, if your software supports running multiple copies across different host machines, then you can scale up host resources by starting additional nodes (via the node pool definition). This can be a very effective way to scale up without service interruption and also helps to address high availability requirements as well.   There are two major classification of node pools: System node pool - one or more node pools for running the Kubernetes control plane services User node pool - zero or more node pools for running applications   A user node pool is essentially where anything that's not directly involved in managing the Kubernetes cluster is run. Any node pools defined for SAS Viya specifically are considered user node pools.   Node Pools > SAS Workload Classes   From a SAS Viya perspective, workload classes correspond to node pools provisioned for the environment.   SAS defines four basic classes of workload for the SAS Viya platform: CAS = high-performance, in-memory analytics engine Compute = traditional SAS Programming Runtime Environment engine Stateful = critical infrastructure services to support SAS Viya operations Stateless = resilient components committed to specific functionality And don't forget the fifth workload class: The Kubernetes control plane Workload classes are realized in the Kubernetes cluster through the use of labels and taints on the nodes (and corresponding node selectors and tolerations on the pods, respectively) as well as pod affinities. Essentially this means there's a lot of variability that Kubernetes considers in how jobs get scheduled to run on nodes. Taking those factors into consideration, it's possible to tune the SAS Viya platform to right-size the environment for effective use.   Minimum Scale   It is technically feasible to run a Kubernetes "cluster" on a single host machine. If that machine has sufficient CPU, RAM, and other resources for the given workload, it could provide the functionality desired. However, that's counter to SAS Viya's expected use of Kubernetes to manage workloads across multiple host machines not only for workload optimization but also to provide robust availability of service.   With that in mind, defining node pools should be weighed from a cost perspective in that each node pool must be represented by at least one host machine for it to handle the associated workload class(es). That is, with four SAS workload classes (and a fifth to run the Kubernetes control plane, if desired), then at least 4 (or 5) machines must be up in order for those jobs to have a place to run. If a node pool is scaled down such that zero nodes are up, then some (or maybe even all) of that associated workload has no place to execute.   Starting at Standard Scale   By default, the SAS Viya platform is configured with four workload classes and they're expected to correspond to four node pools. This helps separate the major service types from each other and allows them to scale independently.   Furthermore, availability operations for some processes might require two (or more) replica pods to ensure continuity of service. And some processes will require at least three functionally-identical pods to run the service - with at least two online to reach quorum and sustain operational service. While it's often technically feasible to run two or more replica pods on a single node, that doesn't achieve the goal of high availability, so this is another aspect to consider when tallying the minimum number of host machines running (and hence, cloud costs) to support your SAS Viya platform.     In the illustration above, we show: N nodes for CAS (3 for MPP CAS with basic HA, 4 for full HA, more as needed) X nodes for SAS Compute (2 or more as needed for HA and workload) 3 nodes for stateful services (some require 3 instances for HA) 2 nodes for stateless services (where 2 instances are sufficient for HA) 3 nodes for the Kubernetes control plane (not visible if using cloud-managed Kubernetes) These values are not requirements, but should be considered as a starting point in planning for any highly available deployment of the SAS Viya platform. The actual values per node pool will vary based on a number of factors.   Also, bear in mind that some offerings, like SAS Viya with SingleStore, define additional workload classes per their specific architecture requirements.    Starting at Larger Scale   The minimum and maximum number of nodes in a given node pool will depend on many considerations. Let's look at a few from the SAS Viya platform's perspective where we might need to ensure more are running than what's outlined for a typical starting point as shown above.   Starting with CAS, it achieves scalability and availability goals by running across multiple hosts (referred to as MPP mode). If running an MPP CAS Server with full availability features and sizable data and/or concurrency requirements, then two controller nodes and several worker nodes will be necessary. At its smallest then, expect at least four nodes to run (2 controller + 2 workers) with additional workers to be considered as workload demands.   For some sites, there might be a need to run multiple CAS servers (either SMP or MPP). For example, some analytic tasks might be better served on nodes that have GPUs. Those machines usually cost more to run and so a dedicated workload class (and associated node pool) might be established to manage it.   Similarly for workloads in support of the SAS Programming and Runtime Environment (that is, the compute workload class) which include SAS Compute, SAS Batch, and SAS Connect servers, different types of host machines with access to specialized resources and/or data might be useful. Combine this with the SAS Workload Management offering so that jobs can be easily configured to run on the best node for their goals (for required resources, priority execution, lower cost, etc.)   In managing and extending the standard workload classes, it might be worthwhile to establish additional workload classes for other analytic workloads if needed to accommodate specialization required at the site. For example, the SAS Micro Analytic Service is a memory-resident, high-performance program execution service intended to support real-time operations (more info). By default, its pods are configured with a node selector to run in the stateless node pool (although it is certainly not a stateless service). If its use is significant, then defining a workload class for MAS will help to ensure it gets appropriate resources for the expected response time.   Starting at Smaller Scale   Not all sites require high redundancy and maximum performance with critical workload separation. It's possible to run the SAS Viya platform using a more restrained approach with a smaller footprint to reduce overhead and maintenance costs.   In that case, the standard set of workload classes might be consolidated to run on fewer node pools with the goal of running on fewer (but possibly larger) host machines. This reduction of complexity won't magically lessen SAS Viya's resource requirements, but for sites which have established that resource consumption will be modest, then optimizing to a smaller footprint is often desirable.   We can combine the infrastructure components of the environment to run together. This means creating a single node pool to host the SAS Viya stateless and stateful workload classes. It could also run the Kubernetes control plane if you're responsible for that as well.   That leaves the CAS and compute workload classes. The SAS analytic engines can consume significant resources if they're worked hard and so it's worth considering keeping their workload classes assigned to dedicated node pools for efficient and effective performance. However, if your site can tolerate less performance, then combining the CAS and compute workload classes to a single node pool might lead to some cost savings..    The objective of combining workload classes to run on fewer node pools is to allow for running fewer host machines in minimal circumstances to save money. Be careful, if you're still running numerous hosts (that is, the number of running nodes is greater than the number of node pools defined), then you have less control over the workload and likely are not running efficiently or cost effectively.   Coda   This post continues the series discussing what it means to scale the SAS Viya Platform along a continuous spectrum. Understanding how the infrastructure is sized, provisioned, managed, and utilized is part of the foundation to ensuring SAS Viya runs efficiently and effectively. Along the way, this post also linked to several posts from my GEL team colleagues and are provided here as well:   Scott McCauley, Performance Tuning for the SAS Viya Platform Gilles Chrzaszcz, SAS Viya – CAS Server Node Affinity And Toleration Rob Collum, The SAS Workload Management Approach to Autoscaling Mike Goddard: What happened? That doesn’t look like the SAS Viya deployment I wanted! Creating custom Viya topologies – realizing the workload placement plan Creating custom SAS Viya topologies – Part 2 (using custom node pools for the compute pods)   Find more articles from SAS Global Enablement and Learning here. ... View more

Recently I was given access to a Kubernetes cluster. The Kubernetes administrator who provided that access wasn't online and I was curious: what level of access and permissions did I have in the cluster? In attempting to query Kubernetes for the answer, I learned some interesting aspects about client certificate authentication, what that means for ascertaining your Kubernetes identity, and some crucial considerations for production environments.   Accessing a Kubernetes Cluster   Access to a Kubernetes cluster is typically given by providing a kubeconfig file. The kubeconfig doesn't "configure Kubernetes" itself. Instead it contains the access protocol that can be used by a Kubernetes client like the kubectl CLI utility or something else such as the OpenLens IDE. In fact, one kubeconfig file can contain definitions for connecting multiple "contexts" across different clusters. That is, it's a handy one-stop shop to drop all of your Kubernetes connection info.   Besides identifying the physical location of the cluster, another critical purpose of the context in a kubeconfig file is that it identifies who you are to the cluster. Direct kubectl to show the configuration:   $ kubectl config view   And it returns an abbreviated view with sensitive parts redacted:   apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://server.demo.sas.com:6443 name: k8s-viya contexts: - context: cluster: k8s-viya user: admin@k8s-viya name: k8s-viya current-context: k8s-viya kind: Config preferences: {} users: - name: admin@k8s-viya user: client-certificate-data: DATA+OMITTED client-key-data: DATA+OMITTED   Now let's break this down a little and have kubectl identify the contexts available and which is in use:   $ kubectl config get-contexts   With results:   CURRENT NAME CLUSTER AUTHINFO NAMESPACE * k8s-viya k8s-viya admin@k8s-viya   In my kubeconfig, there's just one context. Notice there's an asterisk "*" under the Current column for the context that kubectl is currently using to connect to Kubernetes. And we can see under the AuthInfo column that the user is identified as admin@k8s-viya .   [ cue record-scratching sound ]   Hold up - there exists no such "user" as admin@k8s-viya in my cluster. When using client certificate authentication (as shown above), that "user" only exists in the definition of the client certificate. It's basically just a label that helps clarify the purpose of the context. But - to be clear - there's nothing defined in Kubernetes that associates admin@k8s-viya with any authentication or authorization structure.   "Okay," the voice I sometimes hear in my head says, "but there's a third-party plugin for kubectl that can tell me the effective user."   And indeed, there is. Once you install the krew plugin manager and have that install the kubectl-whoami plugin, then it's simple to use:   kubectl whoami And it says:   system:admin   "Ah-ha!" that voice in my head exclaims, "I knew it! Kubernetes says the user is system:admin ".   [ cue louder record-scratching sound ]   Yeahhhhh, no. As it turns out, there's nothing defined in my Kubernetes cluster that refers to system:admin either - as we'll see, it comes from the client certificate in the context definition. Whatever that so-called "user" term is, it's intended to be useful for auditing purposes, but as we saw earlier, it might not have direct bearing on who you are to Kubernetes or what you can do.   What Can I Do in Kubernetes?   As it turns out, Kubernetes will let you ask it what capabilities you can perform. Are you a fan of the Twenty Questions game? You can play it with Kubernetes by asking it what "resource" you can "verb":   $ kubectl auth can-i --help   Check whether an action is allowed. VERB is a logical Kubernetes API verb like 'get', 'list', 'watch', 'delete', etc. TYPE is a Kubernetes resource. Shortcuts and groups will be resolved. NONRESOURCEURL is a partial URL that starts with "/". NAME is the name of a particular Kubernetes resource. This command pairs nicely with impersonation. See --as global flag. Examples: # Check to see if I can create pods in any namespace kubectl auth can-i create pods --all-namespaces # Check to see if I can list deployments in my current namespace kubectl auth can-i list deployments.apps # Check to see if service account "foo" of namespace "dev" can list pods # in the namespace "prod". # You must be allowed to use impersonation for the global option "--as". kubectl auth can-i list pods --as=system:serviceaccount:dev:foo -n prod # Check to see if I can do everything in my current namespace ("*" means all) kubectl auth can-i '*' '*' Of the examples shown, the most useful one to get a definitive answer is the last one which basically asks if you can "do anything" to "any resource". If the response back is "yes", then you've probably won the game because it's a good bet that you're the cluster-admin (you'd need to specify the ‑‑all‑namespaces parameter to be sure).   But if you're not the cluster-admin, you could go on and on and on asking Kubernetes questions trying to ascertain the limits of your access in the environment. Wouldn't you rather have a more straightforward approach?   Kubernetes RBAC   Kubernetes employs a security scheme known as Rules-Based Access Control (RBAC). There's a lot to it, but let's focus on just two things: Roles and RoleBindings (and their Cluster equivalents).   A Role is basically just a set of permissions - the things you can do (i.e., "resources" that you can "verb"). A Role is limited to given namespace whereas a ClusterRole applies to the entire Kubernetes cluster.   A RoleBinding associates a Role with a list of subjects (i.e., users, groups, and/or service accounts). And again, a ClusterRoleBinding performs a similar function, but applied to the entire Kubernetes cluster.   What we want to do is figure out how the identity referenced in the kubeconfig file corresponds to the RBAC controls of the cluster. The (Cluster)RoleBinding is the linchpin that holds it all together. We just need to connect the dots. As shown above, however, the so-called "users" we found referenced by kubectl and in the kubeconfig haven't been helpful so far.   A Closer Look at the Client Certificate   When we asked kubectl to give us a view of the current context configuration earlier, it hid some critical information with DATA-OMITTED . Let's ask to see that now:   $ kubectl config view --raw -o json   And just looking at the interesting part with the "user" definition here:   "users": [ { "name": "admin@k8s-viya", "user": { "client-certificate-data": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRlZ0F3SUJBZ0lJQUpPNXBkbmtmSFV3Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOamcxTlRVek9UY3hNQjRYRFRJek1EVXpNVEUzTWpZeE1Wb1hEVEkwTURVegpNREUzTWpZeE1Wb3dNREVYTUJVR0ExVUVDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhGVEFUQmdOVkJBTVRESE41CmMzUmxiVHBoWkcxcGJqQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1xxxDlBd0VIQTBJQUJCaldZMGlpaVZyR0NVVkwKZEIvNnlvbm5GZVdkQVBsaVdZbXlCVEpDakFhMHp4cmhIS0tobGxXSk1TaWpZVVdPalJ6dFJ2SHBlcHE0aDJPTwpEZlJ5YzdpalNEQkdNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBakFmCkJnTlZIU01FR0RBV2dCUzdhdlZiUDVqZnNWOVZMSFFRUlZrbzNLMnNFekFLQmdncWhrak9QUVFEQWdOSUFEQkYKQWlFQXJLNlA4L1VGSDR3SUZMWmxoU21VakxibGVmODZpWXd1RGdaem0rdHBBaTBDSUd0RlBaeW02OUhmREFiQQpLWVVJOFpacEZibkR1R2h4WlFqRkxmbDdrbE5zCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdFkyeHAKWlc1MExXTmhRREUyT0RVMU5UTTVOekV3SGhjTk1qTXdOVE14TVRjeU5qRXhXaGNOTXpNd05USTRNVGN5TmpFeApXakFqTVNFd0h3WURWUVFEREJock0zTXRZMnhwWlc1MExXTmhRREUyT0RVMU5UTTVOekV3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFSa1FhUysxYmZ2N2EwTWJrbVA5SStiUG4xa3cwL2Z0dU9sQTJYa3ROS3oKWU95VjkwMUFodmxyV3JQTXI4akRpZXNYemlpNnAwd2g1amthbnZwZ2NOMVFvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVXUycjFXeitZMzdGZlZTeDBFRVZaCktOeXRyQk13Q2dZSUtvWkl6ajBFQXdJRFNRQXdSZ0loQUlLMFRyOG1rNHF5UDJiaENpMG5xRmtXUzB1c2hDM0YKckhUdk5rU2V0N3VsQWlFQTFhaVd2Y3p1WnR1NEJVcmJpMXdONzB1VWU3cmF2cWtFSjRYL3htR0R2ME09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K", "client-key-data": "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSU12RVNxVTU2K0NGS1g3a0hUN01rb1BLQ0tvc052MXFneXltRTNyaTh0N09vQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFR05aalNLS0pXc1lKUlV0MEgvcktpZWNWNVowQStXSlppYklGTWtLTUJyVFBHdUVjb3FHVwpxxxt4S0tOaFJZNk5ITzFHOGVsNm1yaUhZNDROOUhKenVBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=" } } ],   In particular, the client-certificate-data is most useful. As shown above, it's base64 encoded (not encrypted; a useful technique to ensure clean copying across platforms).   If we get the decoded content (using " base64 -d ") of the client-certificate-data and then give that to the openssl utility to explain it:   $ kubectl config view --raw -o json | grep client-certificate-data | awk -F"\"" ' { print $4 } ' | base64 -d | openssl x509 -in /dev/stdin -text # kubectl = return the full context content # grep = only show the line with "client-certificate-data" # awk = return the 4th delimited element (the base64 encoded string) # base64 = decode the string # openssl = decode the certificate   We get back:   Certificate: Data: Version: 3 (0x2) Serial Number: 41580943552838773 (0x93b9a5d9e47c75) Signature Algorithm: ecdsa-with-SHA256 Issuer: CN = k3s-client-ca@1685553971 Validity Not Before: May 31 17:26:11 2023 GMT Not After : May 30 17:26:11 2024 GMT Subject: O = system:masters, CN = system:admin Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:18:d6:63:48:a2:89:5a:c6:09:45:4b:74:1f:fa: ca:89:e7:15:e5:9d:00:d9:62:59:89:b2:05:32:42: 8c:06:b4:cf:1a:e1:1c:d2:a1:96:55:89:31:28:a3: 61:45:8e:8d:1c:ed:46:d1:e9:7a:9a:b8:87:63:8e: 0d:f4:72:73:b8 ASN1 OID: prime256v1 NIST CURVE: P-256 X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Client Authentication X509v3 Authority Key Identifier: keyid:BB:6A:F5:5B:3F:98:DF:B1:5F:55:2C:74:10:45:59:28:DC:AD:AC:13 Signature Algorithm: ecdsa-with-SHA256 30:45:02:21:00:ac:ae:8f:f3:f5:05:1f:8c:08:14:b6:65:85: 29:94:8c:b6:e5:79:ff:3a:89:8c:2e:0e:06:73:9b:eb:69:02: 2d:02:20:6b:45:3d:9c:a6:eb:d1:df:0c:06:c0:29:85:08:f1: 96:69:15:b9:c3:b8:68:71:65:08:c5:2d:f9:7b:92:53:6c   Look at the Subject line:   Subject: O = system:masters, CN = system:admin   Now we're getting somewhere. First of all, we can see that the kubectl-whoami plugin found the identity it reported back from the CN (common name) attribute. Now in some clusters, it's possible that could be defined to refer to a user defined in a RoleBinding. But as established earlier, it's not in the cluster I'm using.   Notice that there's also an O (organization, a.k.a., group) defined: system:masters . Let's try looking for that in the cluster:   $ kubectl get clusterrolebindings -o wide | grep system:masters   With the answer:   NAME ROLE AGE USERS GROUPS cluster-admin ClusterRole/cluster-admin 28d system:masters   We found the ClusterRoleBinding named cluster-admin and it's associated with the ClusterRole with similar name. According to the documentation describing the cluster-admin ClusterRole, this user has full, unlimited ` cluster-admin ` privileges in the cluster. The Kubernetes administrator must really like me… I can literally do anything I want in this cluster.   Let's double-check to confirm:   $ kubectl auth can-i "*" "*" --all-namespaces   yes   And so here at the end, we finally found the missing piece that Kubernetes is using to identify who it really thinks I am. The context in the kubeconfig given to me by the Kubernetes administrator grants me full, unlimited permissions via the cluster-admin ClusterRole on all resources by associating with the system:masters group.   Why Does This Matter?   First of all, I personally was somewhat surprised at the one-way direction of creating access control using client certificate authentication as well as the level of isolation. Essentially, it's almost as if the Kubernetes administrator waved a magic wand (to create the RBAC controls and associated client certificate), placed them in a puzzle box (the kubeconfig file with multi-encoded attributes), and gave it to me. So yeah, the Kubernetes administrator knows what's what - but it takes some investigation for me to figure it out in reverse.   Secondly, the arbitrary nature of the "users" defined can be perplexing. They might not exist anywhere in the cluster's RBAC scheme, however they will be logged for auditing purposes as appropriate. Still, since the user names can be so casually defined, there's little to guarantee their effectiveness from an auditing perspective.   Third, I learned that the " system:masters " is a special group whose behavior is hard-coded into Kubernetes, bypassing all RBAC rights checks. There's no changing it. So a kubeconfig file that references the system:masters group provides nearly irrevocable cluster-admin privileges. If a kubeconfig file was misplaced, the only way to invalidate it would be to renew the certificates used by the cluster (or wait for the certificate's defined expiration date).   My key takeaways are that 1) client certificate authentication is difficult to properly secure, 2) take care to define an alternative to the cluster-admin RoleBinding (i.e., avoid associating users with system:masters ), and 3) consider using an alternative authentication scheme (like OpenID Connect or Active Directory) for production environments.   H/T   Special thanks to my GEL team colleagues, especially @StuartRogers (identifying weaknesses in the client certificate authentication approach) and @DavidStern (reverse-engineering the X.509 token content). Any mistakes in this post are certainly mine, not theirs. 😉    ... View more

The SAS Viya Platform provides a powerful analytics environment with a rich set of features and seemingly infinite combination of deployment options. Given a detailed understanding of the workload at hand, we can right-size the SAS Viya Platform so that it is scaled to handle almost any challenge for any number of users acting on any amount of data.   While we can stick the landing to right-size SAS Viya for any given problem, there are some considerations to weigh when attempting to slide along the spectrum of possible scalability options. We cannot simply twist a knob to scale a SAS Viya deployment from zero to infinity. Along the way, crossing the Rubicon becomes a necessity. That is, we will need implement some major changes to the architecture and infrastructure at key points. If the change is significant enough, it may even require a fresh deployment of SAS Viya with a migration from the old environment to the new.   This blog post introduces a new series where we discuss the potential challenges of moving the SAS Viya Platform along the spectrum of scalability.   World-Wide Sizings For customers looking for help with getting the right hardware to run their SAS Viya workloads, the SAS account representative can begin an engagement with the World-Wide Sizings Team. To do this well, the sizing effort is necessarily a collaboration between SAS and the customer. The Sizings team gets the ball rolling by asking (and answering) some critical questions as part of a collaboration to set expectations and devise a plan of attack which ensure scalability goals are met.   Having an understanding of where the customer wants to go with SAS Viya coupled with detailed knowledge of how the relevant technologies scale is an important step along the path to success.   The Radio Analogy To help explain the idea of scaling along a continuous spectrum, let's consider the AM/FM radio. Tuning the radio  can be done by scanning along the adjacent frequencies - like when turning the knob and listening for breaks in the static. Or if you have stations preset, you can jump directly to the frequency of your desired station.   CZmarlin, CC BY-SA 3.0, via Wikimedia Commons   With SAS Viya, we can definitely jump to a desired point of scalability (like the radio's preset). But when it comes to elastic deployments that can automatically scale up or down, that's more like twisting the knob on an old radio to move along each of the frequencies one after the other.   FM can transmit 15× more information than AM. And it's important to note that no matter how far you turn the knob on the AM band, you'll never get to FM. For SAS Viya deployments in the cloud, there are situations where we can slide along the elastic spectrum to a point and then from there, we need to take a more drastic action to get to the next level of performance (like switching from AM to FM).   To take this analogy to its extreme, consider replacing a 1980-era car's stereo with a modern bluetooth streaming unit that also includes satellite radio and a DVD player. All of these technologies help deliver music to your ears, but getting the song you want to hear could range anywhere from a twist of the dial to switching bands to changing source input… all the way to tearing out the old unit and putting in something completely new.   Let's look at the areas of scalability spectrum and the jump points between them that we should consider when planning the elasticity of a SAS Viya deployment.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   An example of interrupted spectrum of scalability There's no official explanation of a continuous spectrum of scalability, but it can be useful to identify where scaling the system might encounter challenges that need additional efforts.   These days, we primarily run SAS software on virtual machines. This isn't required - but yields numerous benefits in terms of repeatability, resilience, and re-use. VM images can be saved and then deployed many times over. When running in the cloud, we can place a VM image on a machine instance with the CPU, RAM, and other resource attributes we desire. If that instance size is too small, it's relatively easy to scale up to run on a new, larger instance with more CPU, RAM, and other resource attributes instead.   While this is simple in concept, there's an outage that must considered. The underlying instance isn't resized on the fly with more CPU, RAM, and all. No, we must start up a brand new instance and have it run the VM image with our software. In this way it's scalable, but not continuous - there's an interruption to service.   An example of continuous spectrum of scalability The SAS Compute Server is offered as part of the SAS Viya Platform. It provides a way for users to each have their own dedicated runtime for processing SAS code. Users (and SAS developers) can even elect to spin up additional SAS Compute Servers on demand for additional concurrent processing power. The SAS Viya Platform (and SAS 9 before that) supports the ability to run SAS Compute Servers across multiple host machines.   So let's say we've defined a node pool in Kubernetes which is labeled exclusively for use by SAS Compute Servers (not required, but often desirable). That node pool is currently running on 3 machine instances in the cloud. We can provision additional nodes to that pool and when they're ready, then SAS Viya will work with Kubernetes to schedule new SAS Compute Servers to run there, too. If the system quiets down, we can even scale down and decommission under-utilized nodes.   This kind of scalability takes some configuration to employ, but it can be fully automated with a Kubernetes cluster autoscaler and with new features introduced with the SAS Workload Manager in the stable-2023.05 release. Regardless of the automation - even if we manually scale the node pool up/down - the point is that the scalability from the users' point-of-view is continuous. There's no interruption in service when we change the resources available for the SAS Compute Server by changing the number of nodes it can run on.   When continuous runs into interrupted The SAS Viya Platform is designed to maximize scalability. Even so, it still must work within the constraints of Kubernetes, the cloud provider, and third-party software on which it relies. There are several areas where we hit the logical limit of continuous scalability and must deal with an interruption to achieve the scalability goal (like reaching the end of the AM dial and switching to the FM band).   Examples of this include: The deployment of the SAS Viya Platform defaults to working with an "internal" instance of Crunchy Postgres. For improved scalability, efficiency, availability, and/or standardization, a customer site may prefer to use an "external" Postgres server, like the managed solutions offered by the cloud providers. While SAS Viya can run with either type, changing from one Postgres to another will require a planned outage of the entire SAS Viya cluster.   SAS Cloud Analytic Services can be deployed to run either SMP (as a single host) or MPP (distributed to run across multiple hosts). After that deployment, an SMP CAS server can be converted to operate as MPP, however that requires a restart of your CAS server and results in the termination of all active connections and sessions as well as the loss of any in-memory data.   A node pool is a group of Kubernetes nodes that share the same instance type, i.e., physical configuration (CPU, RAM, storage, networking, and OS) as well as Kubernetes configuration (like maximum number of pods). While the node pool itself can be modified on the fly to run with more or less nodes, to effectively change the instance type (to get a different machine) requires defining a new node pool, draining the old node pool, and confirming the workload migrates successfully. Some SAS Viya components, like CAS, cannot make this move "live", requiring an outage to complete.   The impact of these changes can range from a short-term outage of a select set of services to the full redeployment of the SAS Viya Platform. Especially where external services are involved, changes to the architecture can take time and effort to implement. Also, the additional resource costs can be significant as well.   Where is this going? This blog post introduces a new series to take a closer look at some of the scalability waypoints where an interruption to service might be needed to gain additional processing power. From a customer perspective, we want to ensure that if their goal is to start small and grow over time, then the ideal scenario won't start too small nor too large. Instead we want to start at a size where scaling up over time can be as continuous as possible - avoiding major deployment modifications which interrupt operations. When we consider scaling a SAS Viya Platform deployment from a small to larger size, we need to be aware of where these possible challenges lie so we can avoid them with sufficient planning, or if that's not possible, then to handle them in a timely and efficient manner.   Find more articles from SAS Global Enablement and Learning here. ... View more

The steps to uninstall SAS Viya are clearly documented in the SAS Viya Administration > Deployment Guide. That should definitely be your first stop any time you want to remove SAS Viya from a Kubernetes cluster. The steps are easy to follow and get the job done with minimal fuss. But if you're like me, chances are you either missed a step or perhaps inserted an unnecessary one. If that's happened to you, then you might find that Kubernetes can get hung up and unable to complete deleting the resource(s) you've set it to. So let's take a look at what can go wrong as well as how to prevent and/or correct those problems.   We'll begin with explaining a little bit about how Kubernetes deletes objects and then highlight areas where this intersects with uninstalling SAS Viya.   Kubernetes Finalizers   When we direct Kubernetes to delete a simple object, then we expect to see Kubernetes delete the object as intended, followed by updating its internal etcd database to reflect the change.   But that's not always the case.   There are some circumstances where the software requires an opportunity to perform some pre-delete actions before the resource is actually deleted. That's where Kubernetes Finalizers come into play. When a finalizer is defined for an object, then that signals Kubernetes that a controller or Kubernetes Operator will first get a chance to delete any dependent objects first. For example, if deleting a PVC, then the Kubernetes controller will first ensure the underlying storage device is released before actually removing the notation of the PVC's existence from etcd.   So when we attempt to delete a resource with a finalizer defined, then Kubernetes adds the ` deletionTimestamp ` to the object. This does a couple of things: 1) it makes the resource read-only (only removing the finalizer attribute is allowed then) and 2) allows the object's controller to notice the change so it can take the pre-delete action.   Once the object's controller has finished its pre-delete activity, then it updates the resource to remove the finalizer references. This then allows Kubernetes to proceed with actually deleting the object.   Why might finalizers fail?   There are many reasons why finalizers might fail. But to keep this post somewhat pithy, let’s look at two simple scenarios.   The first scenario is that the finalizer is unable to complete its prescribed task. For example, perhaps a cloud's load balancer isn't deleting cleanly which leaves the finalizer stuck waiting. The second scenario is that no controller associated with the finalizer is present to actually pre-delete anything and subsequently remove the finalizer reference from the object. There are a couple of reasons for this. One is that the controller process might not be running when it's expected to be there. Another is that maybe there's no controller to act on the finalizer as defined (a variation on the first thing). The latter can be used when you want to manually manage pre-deletion activities yourself.   A finalizer tag on a resource is really just an arbitrary string - Kubernetes doesn't usually understand the third-party values stored there. When Kubernetes sees a finalizer reference on an object to be deleted, all it does is mark the object as ready for deletion and then it awaits removal of the finalizer reference. It won't delete the object until that finalizer reference is removed, waiting forever in some cases.   It's a very simple process that can sometimes lead to complicated outcomes.   Finalizers used by SAS Viya software   Primarily there's one piece of software included with SAS Viya where finalizers play an important role: internal Crunchy PostgreSQL version 5 (updated from version 4 with SAS Viya stable 2022.10).   In the YAML definition below, we can see the finalizer associated with the ` postgres-operator `:   apiVersion: postgres-operator.crunchydata.com/v1beta1 kind: PostgresCluster metadata: annotations: { snipped } creationTimestamp: '2022-12-15T13:47:17Z' finalizers: - postgres-operator.crunchydata.com/finalizer generation: 1 labels: sas.com/admin: namespace sas.com/deployment: sas-viya In order to successfully uninstall SAS Viya, we need to allow the ` postgres-operator ` the opportunity to properly perform its pre-delete activities. If we don't, then we will find it difficult to complete the removal of SAS Viya as intended.   Part of the challenge with Crunchy in particular is that it works using cluster-wide resources like Custom Resource Definitions (CRD) to instantiate namespace resources. Due to how the scoping works, attempting to uninstall SAS Viya by improperly executing the documented steps out of order can lead to a stalled deletion. Typically, this is the result of a race condition where the ` postgres-operator ` acting as a finalizer isn't running (or might not even exist anymore) and so the objects it's supposed to clean up aren't updated such that Kubernetes gets stuck in a wait state.   There are other software components associated with SAS Viya that also define finalizers, but most of those rely on Kubernetes built-in functionality to get the job done and so don't present much problem.   The happy path to uninstall SAS Viya   The first rule of uninstalling SAS Viya is to follow the steps documented in the SAS Viya Administration > Deployment Guide.   The second rule of uninstalling SAS Viya is to follow the steps documented in the SAS Viya Administration > Deployment Guide.   For brevity, I'll hit on the key areas to make the point.   # your namespace here NS=my-sas-viya # 1. delete the Postgres cluster from the namespace kubectl -n ${NS} delete postgresclusters --selector="sas.com/deployment=sas-viya" # 2. delete the namespace (and most SAS Viya assets) kubectl delete ns ${NS} # --- (optional) cluster-wide assets --- # 3. show the CRD provided by SAS Viya kubectl get crd --selector "sas.com/admin=cluster-wide" kubectl get crd --selector "sas.com/admin=cluster-api" # 4. delete the CRD provided by SAS Viya kubectl delete crd --selector "sas.com/admin=cluster-wide" kubectl delete crd --selector "sas.com/admin=cluster-api" # 5. show the clusterroles provided by SAS Viya kubectl get clusterrole --selector "sas.com/admin=cluster-wide" # 6. delete the clusterroles provided by SAS Viya kubectl delete clusterrole --selector "sas.com/admin=cluster-wide"   Steps № 1 and 2 will remove the specified instance of SAS Viya software from the Kubernetes cluster (i.e., postgrescluster and namespace).   Steps № 3 - 6 are optional. They reference cluster-wide elements that might still be in-use by other SAS Viya deployments in your cluster. So only remove them if they're really no longer needed.   Next we'll look at the (mis-)steps to avoid…   Do not stop the SAS Viya services first   To be clear, the documentation doesn't tell you to stop (i.e., scale down) the SAS Viya services prior to uninstalling. This is one of those steps that an over-zealous SAS administrator (like this author) might add to the process - but it's a bad thing.   Remember the ` postgres-operator ` needs to be running so it can perform its finalizer duties. If it's not, then attempting to uninstall SAS Viya will ultimately lead to a stalled execution of the ` kubectl delete ` command for the affected resources.   If you have already scaled down the SAS Viya services and haven't yet begun the uninstall process, then scale those services back up so they're running as expected.   Do not start uninstalling SAS Viya by deleting its namespace   Up through stable-2022.09, you were supposed to begin by deleting the SAS Viya namespace. But not anymore.   Now we must start by deleting the ` postgresclusters ` associated with SAS Viya (step № 1 of the Happy Path above). This will ensure that the finalizer process is properly executed to remove custom resources defined and used by postgres in the namespace.   If we skip that step and jump ahead to deleting the namespace first, then Kubernetes will delete the ` postgres-operator ` before it gets its chance to complete the pre-delete activities it's supposed to perform. This will leave the SAS Viya namespace stuck in a Terminating state forever - or until we manually remove the finalizer reference from the namespace's definition (one of the things the ` postgres-operator ` is expected to do for us).   Furthermore, once you've initiated and stalled the namespace deletion, then you cannot fix it by attempting to retroactively delete ` postgresclusters `. As we say in the South, "That ship's done sailed." The Crunchy operator that would've handled those protected resources is already gone. So, now it's your responsibility to ensure those pre-delete activities are done and the appropriate finalizer state shown on the affected resources.   You broke the uninstall, now fix it   So, if like me you've inadvertently gotten hung up with finalizer resources preventing you from uninstalling SAS Viya. Chances are, this might even prevent you from re-installing SAS Viya. Whatever the case, you need to get things working again.   The first step is to ensure that any finalizer-bound resources are properly cleaned up. This will prove tricky if you don't know what those steps are. You might try cracking open the code of the operator responsible to suss out those actions and then execute them.   Sidebar: You might think that you don't care because maybe this environment was only used for testing/demo purposes. Or maybe this deployment is part of an automated pipeline to temporarily spin up an environment, get the desired results, and then blow it away. But keep in mind that deleting something from Kubernetes' awareness doesn't guarantee the physical items are really gone. Kubernetes might not be aware of the pod any more, but the process could still be running in a VM on an instance in the cloud - continuing to charge money to your organization. There are similar considerations for device storage, load balancers, and so on.   The next step after properly cleaning up those resources is to patch out the finalizer reference on the those objects so that Kubernetes can proceed with its part of the deletion process.   Sidebar: Removing the finalizer references first is an effective way to let Kubernetes complete deletion of the affected objects. This is sometimes referred to as the "nuclear option" because it arbitrarily removes the finalizer reference without performing the due diligence it implies.   Force deletion of the SAS Viya namespace   Fortunately, the fix for dealing with failed execution of the ` postgres-operator.crunchydata.com/finalizer ` is to remove the finalizer reference from the namespace. The code below essentially implements the "nuclear option":   # your namespace here NS=my-sas-viya # get the namespace definition kubectl get namespace ${NS} -o json >/tmp/${NS}.json # remove the "kubernetes" line from the finalizers spec sed -i '/kubernetes/d' /tmp/${NS}.json # put the namespace definition without finalizer into effect kubectl replace --raw "/api/v1/namespaces/${NS}/finalize"-f/tmp/${NS}.json # clean up rm /tmp/${NS}.json   Simply put, this script finds all instances of the word "kubernetes" in the namespace definition's JSON. Specifically, it'll find several entries similar to:   "apiVersion": "v1", "kind": "Namespace", "metadata": { "creationTimestamp": "2022-12-13T21:55:08Z", "labels": { "kubernetes.io/metadata.name": "my-sas-viya" }, "name": "lab", "resourceVersion": "3944", "uid": "4d15d2f4-95df-4d41-b12b-ce951d3ff7a2" }, "spec": { "finalizers": [ "kubernetes" ] }, "status": { "phase": "Active" } ... ... ...   The script removes the "kubernetes" value from all of the finalizer specs it finds. With no finalizer defined, then Kubernetes interprets the change to mean it can now delete the resource. In this case this allows Kubernetes to successfully terminate the SAS Viya namespace.   If you're running a ` kubectl delete ns ` command and it was hung up, then you should see it complete after running the script above (probably in a different terminal window).   Bonus finalizer!   But wait, there's more! Remember earlier where I mentioned that Crunchy defines cluster-wide resources? In particular, there's the ` postgresclusters.postgres-operator.crunchydata.com `. And in normal operation, it doesn't have a finalizer defined for it.   In a normal running SAS Viya environment, we ask for CRD and their associated finalizers:   $ kubectl get crd -o custom-columns=Kind:.kind,Name:.metadata.name,Finalizers:.metadata.finalizers   And see:   Kind Name Finalizers CustomResourceDefinition postgresclusters.postgres-operator.crunchydata.com < none > {additional results not shown}   But now consider this from the perspective of a failed uninstall that we're trying to recover from. We've essentially directed Kubernetes to remove the custom resources (as part of the namespace delete) which are described by this CRD. So Kubernetes applies its own finalizer to the CRD to ensure it's only removed after its dependent resources have been deleted first.   Now it's got a finalizer that wasn't there before:   Kind Name Finalizers CustomResourceDefinition postgresclusters.postgres-operator.crunchydata.com [customresourcecleanup.apiextensions.k8s.io] {additional results not shown} While an interesting result, now we're stuck again. Deleting the CRD cannot finish because there's a finalizer associated with it - and that finalizer isn't able to do its job.   Since we already know the custom resources in the SAS Viya namespace have been deleted at this point, then we know it's okay to remove the finalizer reference from the CRD:   $ kubectl patch crd/postgresclusters.postgres-operator.crunchydata.com -p '{"metadata":{"finalizers":[]}}' --type=merge   With that removed, then Kubernetes can delete the CRD, too. Similar to the namespace deletion above, if you're running a ` kubectl delete crd ` command and it was hung up, then you should see it complete after running the patch above (probably in a different terminal window).   One more thing   If you don't get that CRD deleted because it remains stuck in a Terminating state, and then you attempt to re-install SAS Viya to the Kubernetes cluster, you might hit this error message:   Error from server (MethodNotAllowed): error when creating "/work/deploy/manifest.yaml": create not allowed while custom resource definition is terminating Now that you've read this post, hopefully the error message makes sense. Trying to reinstall Viya while some of the old components are stuck in a Terminating state won't be successful. When you see that, removing the associated finalizer reference - ideally after ensuring the pre-delete tasks that it implies have been completed - should allow the resource to delete from Kubernetes. Then re-installing SAS Viya will proceed as planned.   Wrapping up   The use of Kubernetes Finalizers is intended to help protect an environment from the accidental deletion of resources to ensure smooth operation and clean up. As we've seen, the implementation of finalizers involves trade-offs in terms of flexibility and simplicity. Understanding where these considerations come into play when uninstalling SAS Viya should help if deletion of resources cannot complete normally.   I'd also like to thank my colleague Christian Provenzano for his technical insights on this subject.   Find more articles from SAS Global Enablement and Learning here. ... View more