BookmarkSubscribeRSS Feed

SAS Workload Management

Started ‎01-12-2022 by
Modified ‎01-13-2022 by
Views 3,936

Kubernetes inherently comes with workload management controls. These are great for an IT administrator to maintain stability and harmony in their environment. These controls can extend to SAS. You may ask yourself, "since we have native workload management in our infrastructure, do I need more?" The answer is, "Yes."  In this article I'd like to introduce a new solution introduced by SAS, for SAS cloud deployments: SAS Workload Management.


Like my colleague @EdoardoRiva explains in his community article, Kubernetes provides workload management. In addition, he points out the latest release of SAS Viya is deployed on Kubernetes. Edoardo provides a high level overview of the advantages on SAS Workload Management. And in a very recent article Edoardo explains how the architecture is impacted when SAS WLM is added.  Having a curious mind, I was interested in discovering what SAS Workload Management brings in addition to the existing Kubernetes workload capabilities.


I wanted to see it live and experience myself how it all works, so let me explain how I configured a demo covering the essentials of SAS Workload Management.


By reading the SAS Viya Workload Management documentation I started to learn about pre-emption, queues, compute contexts, host-type, etc. Now let's put it all together.


Spin up Kubernetes Cluster with multiple compute nodes


I started with creating a SAS Viya Kubernetes environment in Azure by using the instructions in the viya4-iac-azure GitHub repository. In contrast with what I normally do (single node), I created an AKS cluster with four compute nodes in the SAS Compute Node Pool. I wanted to play with spreading workload over the different compute hosts and reserving a host for specific tasks. Figure 1 displays the organization of the cluster members.


Figure 1 - Kubernetes cluster organization for SAS Viya deploymentFigure 1 - Kubernetes cluster organization for SAS Viya deployment


Remember, if you’re using the SAS IaC scripts to create your AKS cluster it’s very easy to configure a AKS cluster with four compute nodes.  Specify the min_nodes and max_nodes parameters in the Terraform.tfvars file as seen in Figure 2 below.


Figure 2 - min_ and max_nodes parameter settings in the Terraform.tfvars fileFigure 2 - min_ and max_nodes parameter settings in the Terraform.tfvars file


SAS Environment Manager & Workload Orchestrator


After deploying SAS Viya I navigate to the SAS Environment Manager, where I open the Workload Orchestrator. Consider the Dashboard tab in Figure 3 below.


Figure 3 - Workload Orchestrator Dashboard in SAS Viya Environment ManagerFigure 3 - Workload Orchestrator Dashboard in SAS Viya Environment Manager


Figure 3 shows me I have three Queues available. I also get confirmation I have four compute servers available, which I can use to spread the SAS workload.


Checking the current SAS workload details on those queues is done via the Queue tab (Figure 4 below). Similar there’s a Hosts tab to see the current workload on all the hosts (Figure 5 below).


Figure 4 - Workload Orchestrator QueuesFigure 4 - Workload Orchestrator Queues


Figure 5 - Workload Orchestrator HostsFigure 5 - Workload Orchestrator Hosts


Configuring Queues


Before launching any workload, let’s check the Configuration tab (Figure 6 below). Here we have a lot of options to define where the jobs submitted to a specific queue will run.


Figure 6 - Workload Orchestrator Configuration tabFigure 6 - Workload Orchestrator Configuration tab



Consider the Queue Configuration in Figure 7. First of all, you can assign a priority to the queue. Jobs submitted to queues with a higher priority will run before jobs submitted to queues with a lower priority.  Look back to Figure 4 and it's clear I have configured three queues in my demo environment: default, priority and interactive, respectively having priority 10, 20 and 30.


Figure 7 - Workload Orchestrator Queue ConfigurationFigure 7 - Workload Orchestrator Queue Configuration


There’s also possibility to specify a list of other queues the queue can preempt.   


Note that I selected for the interactive queue as Host Type SAS Studio interactive. This means for the interactive queue I would like to run the jobs that are submitted to this queue on hosts reserved purely for this type of work. More about configuring Host Types in the next section.  


As my goal is to do a basic SAS Workload Management demo, so for now, that’s enough configuration for me. However, it should be clear there are many more to configurable queue parameters. For example, how many jobs can run per user and per host, who can submit jobs to the queue, or who can administer this queue.  Also, worth mentioning is the possibility to specify a completely new group of settings based on the time of the day. 


Please  the SAS Workload documentation for further details.


Configuring Host Types


As mentioned earlier, I configured the interactive queue to run jobs on hosts that are of type SAS Studio Interactive.  These Host Types are configured by an administrator when selecting in the same configuration tab the Host Types (see Figure 8).


Figure 8 - Workload Orchestrator Hosts ConfigurationFigure 8 - Workload Orchestrator Hosts Configuration


For the SAS Studio Interactive Host Type, I assigned one specific compute node.  Here, I selected it by name. But in the field Host Properties, you can also make use of Kubernetes Node Labels.  I did that as well.  It should be in sync with a label on my compute nodes.


By running the following kubectl command I found my compute nodes and validated a "wlm" label exists. 


kubectl get nodes,LABELS:.metadata.labels | grep compute | grep wlm
As you can see in the results in Figure 9, three out four nodes have "wlm=default" but one is labelled with "wlm=interactive".  That’s the node I’m targeting for Host Type SAS Studio Interactive and where I would like to run jobs submitted to the interactive queue. I will discuss shortly how to configure so only SAS Studio jobs land & run on this interactive queue and associated compute host labelled "wlm=interactive".
Figure 9 - compute nodesFigure 9 - compute nodes


Lastly, I want to focus on another configuration difference between my default host type and the SAS Studio interactive host type on my demo environment (see Figures 10 and 11).


Figure 10 - default Host TypeFigure 10 - default Host Type      Figure 11 - SAS Studio Interactive Host TypeFigure 11 - SAS Studio Interactive Host Type


I specified a different number for maximum jobs allowed. For the default hosts it’s set to four, whereas for the SAS Studio interactive I allow five jobs. During the demo it will help me to quickly discover which node is recognized as compute node to run SAS Studio jobs.


Configuring contexts


Let's now explore how to make sure a SAS Studio job is ending on the interactive queue and the associated compute node reserved for that queue.  Please refer to another article from Edoardo to learn all the details about the SAS runtime environment with the latest SAS Viya release on Kubernetes.  For now it’s important to know when requesting a SAS programming environment a Kubernetes SAS programming pod launches.  Depending how or from where the request is coming, a different context will be used:

  • Compute context (with SAS Studio, SAS Model Studio, ..)
  • Connect context  (with SAS/Connect)
  • Batch context (when submitting program via sas-viya-client)

Simply said, the program launches with a specific context, as seen in Figure 12.


Figure 12 - List of contextsFigure 12 - List of contexts


And with each compute, batch, or connect context a launcher context is associated, as seen in Figure 13.


Figure 13 - Launcher contextsFigure 13 - Launcher contexts

SAS Studio compute context


Let’s take the example of SAS Studio: in Figure 14, a SAS Studio compute context will go together with SAS Studio Launcher context.


Figure 14 - SAS Studio compute and launcher contextsFigure 14 - SAS Studio compute and launcher contexts


And as seen in Figure 15, in the SAS Studio Launcher context, one specifies the SAS Workload Orchestrator queue. I assigned the interactive queue.  Remember jobs submitted to this queue will launch on computes nodes of host type SAS Studio interactive.


Figure 15 - SAS Studio launcher contextFigure 15 - SAS Studio launcher context


SAS viya-cli batch-context


As I will also use sas-viya-cli to launch programs from batch in the demo let me share how to configure the possibility to run priority batch jobs. 


First, I go to the batch contexts. Next, to the predefined default context. I added an extra one here, a new, customized context named priority (see Figure 16).

Figure 16 - Batch contextFigure 16 - Batch context


Checking the details, notice I specify in this context the SAS Workload Orchestrator queue. I assigned the priority queue as it has a higher priority than the default queue. I could have specified the Workload Orchestrator queue on the associated SAS Launcher context (that would be the SAS Batch Service Launcher context), but I preferred to make use of this Batch Context Workload Orchestrator field. 


The bottom line is, I now have the choice to launch batch jobs with a default or priority context. As the default context  runs on the default queue, jobs submitted on that queue have to wait for an available slot on the compute nodes or might be even preempted by jobs with a higher priority.


Demo script


Time to see this in action. The following script launches SAS batch jobs with the  SAS Viya CLI.  I’m using three different profiles, each authenticating with a different user.


export SSL_CERT_FILE=/data/config/vdm/security/cacerts/**
/opt/sas/viya/home/sas-viya --profile default profile set-endpoint "${INGRESS_URL}"
/opt/sas/viya/home/sas-viya --profile default profile toggle-color off
/opt/sas/viya/home/sas-viya --profile default profile set-output text
/opt/sas/viya/home/sas-viya --profile default auth login -u frederik -p password

/opt/sas/viya/home/sas-viya --profile prod profile set-endpoint "${INGRESS_URL}"
/opt/sas/viya/home/sas-viya --profile prod profile toggle-color off
/opt/sas/viya/home/sas-viya --profile prod profile set-output text
/opt/sas/viya/home/sas-viya --profile prod auth login -u edoardo -p password

/opt/sas/viya/home/sas-viya --profile test profile set-endpoint "${INGRESS_URL}"
/opt/sas/viya/home/sas-viya --profile test profile toggle-color off
/opt/sas/viya/home/sas-viya --profile test profile set-output text
/opt/sas/viya/home/sas-viya --profile test auth login -u sundaresh -p password123


After the authentication script, I loop over a set of commands, each launching SAS Programs.  Each loop launches two jobs with the default profile, one job with the test profile, and a series of jobs launch SAS programs with the production profile.  If you look carefully, you will see when using the prod profile I add "-c priority".  This means the request launches with the priority context of the batch context and consequently submits it to the priority queue.  

for i in {1..50}
        echo "Welcome $i times"
        /opt/sas/viya/home/sas-viya --profile default batch jobs submit-pgm -c default --pgm --restart
        /opt/sas/viya/home/sas-viya --profile test batch jobs submit-pgm -c default --pgm --restart
        /opt/sas/viya/home/sas-viya --profile default batch jobs submit-pgm -c default --pgm --restart
        /opt/sas/viya/home/sas-viya --profile prod batch jobs submit-pgm -c priority --pgm --restart
        /opt/sas/viya/home/sas-viya --profile prod batch jobs submit-pgm -c priority --pgm --restart
        /opt/sas/viya/home/sas-viya --profile prod batch jobs submit-pgm -c priority --pgm --restart
        /opt/sas/viya/home/sas-viya --profile prod batch jobs submit-pgm -c priority --pgm --restart


Queue and Host statuses


Soon after the initial loop runs, the priority queue will run the maximum of 12 jobs. A the same time the default queue has all jobs in pending state (see Figure 17).


Figure 17 - Queue StatusFigure 17 - Queue Status


This is exactly what we expected to be happen because we defined max number of four running jobs and we also requested three out of four compute nodes for normal workload use.  So, once you have 12 jobs you have reached the maximum and jobs submitted to the priority queue will take precedence over the default queue.


Also, note that one host has nothing to do because we reserved it for SAS Studio sessions.  That’s why the first three hosts have status OPEN-FULL and the last one OPEN-OK (see Figure 18).


Figure 18 - Host StatusFigure 18 - Host Status

 If a SAS Studio session is launched, a running program will be show on the interactive queue (see Figures 19 and 20).


Figure 19 - SAS Studio sessionFigure 19 - SAS Studio session


Figure 20 - Updated interactive queueFigure 20 - Updated interactive queue

Recorded demo


If you want to see a demo of the SAS Viya Workload Management, refer to the video below by my colleague Mark Schneider (@Mark_sas)  SAS Advisory Product Manager.  Mark talks about SAS WLM and introduces the demo described in this blog.  Have a look and let us know your thoughts.




SAS Studio Interactive Host Type was well explained

Version history
Last update:
‎01-13-2022 08:23 AM
Updated by:



Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.

If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website. 

Register now!

Free course: Data Literacy Essentials

Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning  and boost your career prospects.

Get Started

Article Tags