BookmarkSubscribeRSS Feed

4 Tips for a Successful Start with SAS Workload Management

Started ‎03-18-2022 by
Modified ‎03-18-2022 by
Views 2,277

Since SAS Workload Management became available with SAS Viya, I've been telling everyone how easy it is to install it: that's a significant improvement when compared to SAS Grid Manager on SAS 9. So easy, that I initially forgot to check if you have to do anything at all. Just include SAS Workload Management in your license, install SAS Viya following the same instructions as if it was not there, and it's done! Well, at least, so it seemed.


SAS' goal to make the modernization process as simple as possible results in SAS Workload Orchestrator administrative interface being so similar between SAS 9 and SAS Viya, that even the most experienced SAS administrator may forget that there are differences between the two versions. These include a few steps that you should follow during the initial deployment, to properly configure SAS Workload Management.


Let's see four points that I initially overlooked so that you can start your SAS Workload Management experience on the right foot!  


1. Define a topology for the SAS compute nodes


What does this mean? SAS Workload Management documentation explains this requirement quite clearly:


The only nodes that can run SAS Workload Orchestrator jobs (pod requests) are ones on which the SAS Workload Orchestrator daemonset is present. The daemonset can be deployed on only nodes that contain the label If no nodes contain this label, then there are no nodes that are available to process the SAS Workload Orchestrator jobs.


For most SAS Viya solutions, labeling of compute nodes is strongly recommended, but not required, so you may opt to skip it. If you then add SAS Workload Management to the environment but forget to perform this pre-requisite, then no node will be available to execute SAS jobs.


Administrators may notice this issue from the Workload Orchestrator page in Environment Manager: the Dashboard page, as well as the Hosts page, will show no items available:



Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.



Users will be unable to execute any SAS code; for example, in SAS Studio they will get a message stating that no SAS Compute Server is available:







Finally, Kubernetes administrators will notice that the SAS Workload Orchestrator daemonset does not have any pod running:

kubectl -n ${NS} get daemonset sas-workload-orchestrator

sas-workload-orchestrator   0         0         0       0            0           <none>          14h


To fix this, let’s satisfy the requirement and add the right label to a few nodes:


kubectl label nodes <sasnode1>


kubectl label nodes <sasnodeN>

As soon as this change is applied, the sas-workload-orchestrator deamonset pods start on the desired nodes:

kubectl -n ${NS} get daemonset sas-workload-orchestrator

sas-workload-orchestrator   2         2         2       2            2           <none>          14h

You can see that now the Workload Orchestrator page in Environment Manager correctly reports the labeled nodes:




and, more importantly, users can successfully enjoy SAS!  


2. Define the SAS Workload Management ClusterRoles and ClusterRoleBindings


SAS Workload Orchestrator collects information about resources on the nodes that can be used to run jobs. To obtain accurate resource information, it requires additional privileges not available to default service accounts.


If you do not read the SAS Workload Management specific documentation, you may skip this passage and simply install SAS Viya as usual. SAS Workload Management will be installed and jobs will run, everything may seem fine. Yet, you will notice in the Workload Orchestrator page in Environment Manager that some information is missing from the Hosts tab:





  • only a limited set of statistics related to Host resource utilization can be reported (i.e., the number of CPU cores will be there, but the number of CPU cores allocated by Kubernetes to running jobs will be missing)
  • the properties field will be empty. This is where SAS Workload Orchestrator should report all custom labels that you may have assigned to Kubernetes nodes. Normally, you would use these properties to define host types and assign them to queues; in this case, you would lose this capability.



The solution is easy: you can find the definition of the ClusterRole with the required permissions, and the ClusterRoleBinding that assigns that role to the sas-workload-orchestrator service account, in the overlay available at $deploy/overlays/sas-workload-orchestrator. Add the overlay to the resources block of the base kustomization.yaml file, then deploy - or redeploy - SAS Viya as usual. Here is an example:


- sas-bases/overlays/sas-workload-orchestrator

Note that you may have to restart the SAS Workload Orchestrator deamonset pods for them to acquire the new permissions; do it without worrying, because this will not impact running jobs.  


3. Be aware of default workload management behavior


When SAS Workload Management is included in your environment, SAS Viya enforces limits to the number of running compute sessions differently. Even without any explicit configuration, by default, the following will happen:


  1. The default capability to prevent users from running unlimited SAS compute servers is disabled. SAS Workload Orchestrator provides more advanced capabilities to obtain the same protection using limits on queues or hosts. This will not leave you unprotected without any limit by default, as you can read in the next point.
  2. Every installation includes a default Host Type definition that initially applies to all nodes; it includes a setting for “Maximum jobs allowed” equal to -1, which means “as many jobs as CPU cores available”. For example, a grid with 4 compute nodes with 8 cores each, by default, cannot run more than 32 jobs. This limit includes every kind of SAS compute session, be it a batch job, an interactive session used by SAS Studio, or a pool of available servers. If you try to start more, the sessions in excess will be kept pending by SAS Workload Orchestrator, and the client application that tried to launch them will probably timeout before being able to get a compute server. Time to tune your grid!


  3. The only queue available out of the box is the default one, which does not include any limit nor privilege.

    This is different than SAS Grid Manager for Platform on SAS 9, which always includes a priority queue. With SAS Viya, if you want a priority queue, you can explicitly define it. Additionally, no queues are initially specified in any compute, batch, connect, or launcher context: this means that every job will end up in this default queue.



If you are not aware of these default settings, you may be surprised to find either unexpected limits, or, on the contrary, more sessions than anticipated!

4. Be aware of additional monitoring capabilities that are included


The first time I tested SAS Workload Orchestrator, I immediately recognized the web administration interface, so similar to SAS Grid Manager on SAS 9, so I did not bother to verify if there was anything else available. Well, there is. If you choose to install the SAS Viya monitoring components, then the "SAS Launched Jobs - Node Activity" and "SAS Launched Jobs - User Activity" Grafana dashboards are provided to enable you to view information about SAS jobs. It’s a valuable capability that can help you monitor and administer your grid: get familiar with them and add them to the list of tools that SAS Administrators should use!





Before closing, I'll leave you with a bonus tip: familiarize yourself with the available material about SAS Workload Management - the list grows every week!


SAS Viya documentation:

SAS Workload Orchestrator Overview
Workload Orchestrator Page
Use the SAS Launched Jobs Dashboards


Communities articles:

SAS Grid and SAS Viya: together to provide advanced workload management
SAS Workload Management on SAS Viya: Deployment Architecture
SAS Grid Manager Modernization: Do You Have a Plan?
The Evolution of SAS Workload Management
Workload Management from the Command Line


Ask the expert webinar:

How Can SAS® Workload Management Optimize Cloud Resources ...? Q&A, Slides, and Recording  

Find more articles from SAS Global Enablement and Learning here.

Version history
Last update:
‎03-18-2022 02:48 PM
Updated by:



Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.

If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website. 

Register now!

Free course: Data Literacy Essentials

Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning  and boost your career prospects.

Get Started

Article Tags