SAS Workload Management
Kubernetes inherently comes with workload management controls. These are great for an IT administrator to maintain stability and harmony in their environment. These controls can extend to SAS. You may ask yourself, "since we have native workload management in our infrastructure, do I need more?" The answer is, "Yes." In this article I'd like to introduce a new solution introduced by SAS, for SAS cloud deployments: SAS Workload Management.
Like my colleague @EdoardoRiva explains in his community article, Kubernetes provides workload management. In addition, he points out the latest release of SAS Viya is deployed on Kubernetes. Edoardo provides a high level overview of the advantages on SAS Workload Management. And in a very recent article Edoardo explains how the architecture is impacted when SAS WLM is added. Having a curious mind, I was interested in discovering what SAS Workload Management brings in addition to the existing Kubernetes workload capabilities.
I wanted to see it live and experience myself how it all works, so let me explain how I configured a demo covering the essentials of SAS Workload Management.
By reading the SAS Viya Workload Management documentation I started to learn about pre-emption, queues, compute contexts, host-type, etc. Now let's put it all together.
Spin up Kubernetes Cluster with multiple compute nodes
I started with creating a SAS Viya Kubernetes environment in Azure by using the instructions in the viya4-iac-azure GitHub repository. In contrast with what I normally do (single node), I created an AKS cluster with four compute nodes in the SAS Compute Node Pool. I wanted to play with spreading workload over the different compute hosts and reserving a host for specific tasks. Figure 1 displays the organization of the cluster members.
Figure 1 - Kubernetes cluster organization for SAS Viya deployment
Remember, if you’re using the SAS IaC scripts to create your AKS cluster it’s very easy to configure a AKS cluster with four compute nodes. Specify the min_nodes and max_nodes parameters in the Terraform.tfvars file as seen in Figure 2 below.
Figure 2 - min_ and max_nodes parameter settings in the Terraform.tfvars file
SAS Environment Manager & Workload Orchestrator
After deploying SAS Viya I navigate to the SAS Environment Manager, where I open the Workload Orchestrator. Consider the Dashboard tab in Figure 3 below.
Figure 3 - Workload Orchestrator Dashboard in SAS Viya Environment Manager
Figure 3 shows me I have three Queues available. I also get confirmation I have four compute servers available, which I can use to spread the SAS workload.
Checking the current SAS workload details on those queues is done via the Queue tab (Figure 4 below). Similar there’s a Hosts tab to see the current workload on all the hosts (Figure 5 below).
Figure 4 - Workload Orchestrator Queues
Figure 5 - Workload Orchestrator Hosts
Configuring Queues
Before launching any workload, let’s check the Configuration tab (Figure 6 below). Here we have a lot of options to define where the jobs submitted to a specific queue will run.
Figure 6 - Workload Orchestrator Configuration tab
Consider the Queue Configuration in Figure 7. First of all, you can assign a priority to the queue. Jobs submitted to queues with a higher priority will run before jobs submitted to queues with a lower priority. Look back to Figure 4 and it's clear I have configured three queues in my demo environment: default, priority and interactive, respectively having priority 10, 20 and 30.
There’s also possibility to specify a list of other queues the queue can preempt.
Note that I selected for the interactive queue as Host Type SAS Studio interactive. This means for the interactive queue I would like to run the jobs that are submitted to this queue on hosts reserved purely for this type of work. More about configuring Host Types in the next section.
As my goal is to do a basic SAS Workload Management demo, so for now, that’s enough configuration for me. However, it should be clear there are many more to configurable queue parameters. For example, how many jobs can run per user and per host, who can submit jobs to the queue, or who can administer this queue. Also, worth mentioning is the possibility to specify a completely new group of settings based on the time of the day.
Please the SAS Workload documentation for further details.
Configuring Host Types
As mentioned earlier, I configured the interactive queue to run jobs on hosts that are of type SAS Studio Interactive. These Host Types are configured by an administrator when selecting in the same configuration tab the Host Types (see Figure 8).
For the SAS Studio Interactive Host Type, I assigned one specific compute node. Here, I selected it by name. But in the field Host Properties, you can also make use of Kubernetes Node Labels. I did that as well. It should be in sync with a label on my compute nodes.
By running the following kubectl command I found my compute nodes and validated a "wlm" label exists.
kubectl get nodes -o=custom-columns=NODENAME:.metadata.name,LABELS:.metadata.labels | grep compute | grep wlm
Figure 9 - compute nodes
Lastly, I want to focus on another configuration difference between my default host type and the SAS Studio interactive host type on my demo environment (see Figures 10 and 11).
I specified a different number for maximum jobs allowed. For the default hosts it’s set to four, whereas for the SAS Studio interactive I allow five jobs. During the demo it will help me to quickly discover which node is recognized as compute node to run SAS Studio jobs.
Configuring contexts
Let's now explore how to make sure a SAS Studio job is ending on the interactive queue and the associated compute node reserved for that queue. Please refer to another article from Edoardo to learn all the details about the SAS runtime environment with the latest SAS Viya release on Kubernetes. For now it’s important to know when requesting a SAS programming environment a Kubernetes SAS programming pod launches. Depending how or from where the request is coming, a different context will be used:
- Compute context (with SAS Studio, SAS Model Studio, ..)
- Connect context (with SAS/Connect)
- Batch context (when submitting program via sas-viya-client)
Simply said, the program launches with a specific context, as seen in Figure 12.
Figure 12 - List of contexts
And with each compute, batch, or connect context a launcher context is associated, as seen in Figure 13.
Figure 13 - Launcher contexts
SAS Studio compute context
Let’s take the example of SAS Studio: in Figure 14, a SAS Studio compute context will go together with SAS Studio Launcher context.
Figure 14 - SAS Studio compute and launcher contexts
And as seen in Figure 15, in the SAS Studio Launcher context, one specifies the SAS Workload Orchestrator queue. I assigned the interactive queue. Remember jobs submitted to this queue will launch on computes nodes of host type SAS Studio interactive.
Figure 15 - SAS Studio launcher context
SAS viya-cli batch-context
As I will also use sas-viya-cli to launch programs from batch in the demo let me share how to configure the possibility to run priority batch jobs.
First, I go to the batch contexts. Next, to the predefined default context. I added an extra one here, a new, customized context named priority (see Figure 16).
Figure 16 - Batch context
Checking the details, notice I specify in this context the SAS Workload Orchestrator queue. I assigned the priority queue as it has a higher priority than the default queue. I could have specified the Workload Orchestrator queue on the associated SAS Launcher context (that would be the SAS Batch Service Launcher context), but I preferred to make use of this Batch Context Workload Orchestrator field.
The bottom line is, I now have the choice to launch batch jobs with a default or priority context. As the default context runs on the default queue, jobs submitted on that queue have to wait for an available slot on the compute nodes or might be even preempted by jobs with a higher priority.
Demo script
Time to see this in action. The following script launches SAS batch jobs with the SAS Viya CLI. I’m using three different profiles, each authenticating with a different user.
#!/bin/bash
export SSL_CERT_FILE=/data/config/vdm/security/cacerts/**
INGRESS_URL=https://myhost.sas.com
/opt/sas/viya/home/sas-viya --profile default profile set-endpoint "${INGRESS_URL}"
/opt/sas/viya/home/sas-viya --profile default profile toggle-color off
/opt/sas/viya/home/sas-viya --profile default profile set-output text
/opt/sas/viya/home/sas-viya --profile default auth login -u frederik -p password
/opt/sas/viya/home/sas-viya --profile prod profile set-endpoint "${INGRESS_URL}"
/opt/sas/viya/home/sas-viya --profile prod profile toggle-color off
/opt/sas/viya/home/sas-viya --profile prod profile set-output text
/opt/sas/viya/home/sas-viya --profile prod auth login -u edoardo -p password
/opt/sas/viya/home/sas-viya --profile test profile set-endpoint "${INGRESS_URL}"
/opt/sas/viya/home/sas-viya --profile test profile toggle-color off
/opt/sas/viya/home/sas-viya --profile test profile set-output text
/opt/sas/viya/home/sas-viya --profile test auth login -u sundaresh -p password123
After the authentication script, I loop over a set of commands, each launching SAS Programs. Each loop launches two jobs with the default profile, one job with the test profile, and a series of jobs launch SAS programs with the production profile. If you look carefully, you will see when using the prod profile I add "-c priority". This means the request launches with the priority context of the batch context and consequently submits it to the priority queue.
for i in {1..50}
do
echo "Welcome $i times"
/opt/sas/viya/home/sas-viya --profile default batch jobs submit-pgm -c default --pgm prog1_hmeq.sas --restart
/opt/sas/viya/home/sas-viya --profile test batch jobs submit-pgm -c default --pgm prog2_baseball.sas --restart
/opt/sas/viya/home/sas-viya --profile default batch jobs submit-pgm -c default --pgm prog2_baseball.sas --restart
/opt/sas/viya/home/sas-viya --profile prod batch jobs submit-pgm -c priority --pgm prog2_baseball.sas --restart
/opt/sas/viya/home/sas-viya --profile prod batch jobs submit-pgm -c priority --pgm prog3_baseball.sas --restart
/opt/sas/viya/home/sas-viya --profile prod batch jobs submit-pgm -c priority --pgm prog4_baseball.sas --restart
/opt/sas/viya/home/sas-viya --profile prod batch jobs submit-pgm -c priority --pgm prog5_baseball.sas --restart
done
Queue and Host statuses
Soon after the initial loop runs, the priority queue will run the maximum of 12 jobs. A the same time the default queue has all jobs in pending state (see Figure 17).
This is exactly what we expected to be happen because we defined max number of four running jobs and we also requested three out of four compute nodes for normal workload use. So, once you have 12 jobs you have reached the maximum and jobs submitted to the priority queue will take precedence over the default queue.
Also, note that one host has nothing to do because we reserved it for SAS Studio sessions. That’s why the first three hosts have status OPEN-FULL and the last one OPEN-OK (see Figure 18).
Figure 18 - Host Status
If a SAS Studio session is launched, a running program will be show on the interactive queue (see Figures 19 and 20).
Figure 19 - SAS Studio session
Figure 20 - Updated interactive queue
Recorded demo
If you want to see a demo of the SAS Viya Workload Management, refer to the video below by my colleague Mark Schneider (@Mark_sas) SAS Advisory Product Manager. Mark talks about SAS WLM and introduces the demo described in this blog. Have a look and let us know your thoughts.
SAS INNOVATE 2024
Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.
If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website.
