4 Tips for a Successful Start with SAS Workload Management
Since SAS Workload Management became available with SAS Viya, I've been telling everyone how easy it is to install it: that's a significant improvement when compared to SAS Grid Manager on SAS 9. So easy, that I initially forgot to check if you have to do anything at all. Just include SAS Workload Management in your license, install SAS Viya following the same instructions as if it was not there, and it's done! Well, at least, so it seemed.
SAS' goal to make the modernization process as simple as possible results in SAS Workload Orchestrator administrative interface being so similar between SAS 9 and SAS Viya, that even the most experienced SAS administrator may forget that there are differences between the two versions. These include a few steps that you should follow during the initial deployment, to properly configure SAS Workload Management.
Let's see four points that I initially overlooked so that you can start your SAS Workload Management experience on the right foot!
1. Define a topology for the SAS compute nodes
What does this mean? SAS Workload Management documentation explains this requirement quite clearly:
The only nodes that can run SAS Workload Orchestrator jobs (pod requests) are ones on which the SAS Workload Orchestrator daemonset is present. The daemonset can be deployed on only nodes that contain the label workload.sas.com/class=compute. If no nodes contain this label, then there are no nodes that are available to process the SAS Workload Orchestrator jobs.
For most SAS Viya solutions, labeling of compute nodes is strongly recommended, but not required, so you may opt to skip it. If you then add SAS Workload Management to the environment but forget to perform this pre-requisite, then no node will be available to execute SAS jobs.
Administrators may notice this issue from the Workload Orchestrator page in Environment Manager: the Dashboard page, as well as the Hosts page, will show no items available:
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Users will be unable to execute any SAS code; for example, in SAS Studio they will get a message stating that no SAS Compute Server is available:
Finally, Kubernetes administrators will notice that the SAS Workload Orchestrator daemonset does not have any pod running:
kubectl -n ${NS} get daemonset sas-workload-orchestrator
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
sas-workload-orchestrator 0 0 0 0 0 <none> 14h
To fix this, let’s satisfy the requirement and add the right label to a few nodes:
kubectl label nodes <sasnode1> workload.sas.com/class=compute
…
kubectl label nodes <sasnodeN> workload.sas.com/class=computeAs soon as this change is applied, the sas-workload-orchestrator deamonset pods start on the desired nodes:
kubectl -n ${NS} get daemonset sas-workload-orchestrator
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
sas-workload-orchestrator 2 2 2 2 2 <none> 14hYou can see that now the Workload Orchestrator page in Environment Manager correctly reports the labeled nodes:
and, more importantly, users can successfully enjoy SAS!
2. Define the SAS Workload Management ClusterRoles and ClusterRoleBindings
SAS Workload Orchestrator collects information about resources on the nodes that can be used to run jobs. To obtain accurate resource information, it requires additional privileges not available to default service accounts.
If you do not read the SAS Workload Management specific documentation, you may skip this passage and simply install SAS Viya as usual. SAS Workload Management will be installed and jobs will run, everything may seem fine. Yet, you will notice in the Workload Orchestrator page in Environment Manager that some information is missing from the Hosts tab:
- only a limited set of statistics related to Host resource utilization can be reported (i.e., the number of CPU cores will be there, but the number of CPU cores allocated by Kubernetes to running jobs will be missing)
- the properties field will be empty. This is where SAS Workload Orchestrator should report all custom labels that you may have assigned to Kubernetes nodes. Normally, you would use these properties to define host types and assign them to queues; in this case, you would lose this capability.
The solution is easy: you can find the definition of the ClusterRole with the required permissions, and the ClusterRoleBinding that assigns that role to the sas-workload-orchestrator service account, in the overlay available at $deploy/overlays/sas-workload-orchestrator. Add the overlay to the resources block of the base kustomization.yaml file, then deploy - or redeploy - SAS Viya as usual. Here is an example:
resources:
...
- sas-bases/overlays/sas-workload-orchestratorNote that you may have to restart the SAS Workload Orchestrator deamonset pods for them to acquire the new permissions; do it without worrying, because this will not impact running jobs.
3. Be aware of default workload management behavior
When SAS Workload Management is included in your environment, SAS Viya enforces limits to the number of running compute sessions differently. Even without any explicit configuration, by default, the following will happen:
- The default capability to prevent users from running unlimited SAS compute servers is disabled. SAS Workload Orchestrator provides more advanced capabilities to obtain the same protection using limits on queues or hosts. This will not leave you unprotected without any limit by default, as you can read in the next point.
- Every installation includes a default Host Type definition that initially applies to all nodes; it includes a setting for “Maximum jobs allowed” equal to -1, which means “as many jobs as CPU cores available”. For example, a grid with 4 compute nodes with 8 cores each, by default, cannot run more than 32 jobs. This limit includes every kind of SAS compute session, be it a batch job, an interactive session used by SAS Studio, or a pool of available servers. If you try to start more, the sessions in excess will be kept pending by SAS Workload Orchestrator, and the client application that tried to launch them will probably timeout before being able to get a compute server. Time to tune your grid!
- The only queue available out of the box is the default one, which does not include any limit nor privilege.
This is different than SAS Grid Manager for Platform on SAS 9, which always includes a priority queue. With SAS Viya, if you want a priority queue, you can explicitly define it. Additionally, no queues are initially specified in any compute, batch, connect, or launcher context: this means that every job will end up in this default queue.
If you are not aware of these default settings, you may be surprised to find either unexpected limits, or, on the contrary, more sessions than anticipated!
4. Be aware of additional monitoring capabilities that are included
The first time I tested SAS Workload Orchestrator, I immediately recognized the web administration interface, so similar to SAS Grid Manager on SAS 9, so I did not bother to verify if there was anything else available. Well, there is. If you choose to install the SAS Viya monitoring components, then the "SAS Launched Jobs - Node Activity" and "SAS Launched Jobs - User Activity" Grafana dashboards are provided to enable you to view information about SAS jobs. It’s a valuable capability that can help you monitor and administer your grid: get familiar with them and add them to the list of tools that SAS Administrators should use!
Coda
Before closing, I'll leave you with a bonus tip: familiarize yourself with the available material about SAS Workload Management - the list grows every week!
SAS Viya documentation:
SAS Workload Orchestrator Overview
Workload Orchestrator Page
Use the SAS Launched Jobs Dashboards
Communities articles:
SAS Grid and SAS Viya: together to provide advanced workload management
SAS Workload Management on SAS Viya: Deployment Architecture
SAS Grid Manager Modernization: Do You Have a Plan?
The Evolution of SAS Workload Management
Workload Management from the Command Line
Ask the expert webinar:
How Can SAS® Workload Management Optimize Cloud Resources ...? Q&A, Slides, and Recording
Find more articles from SAS Global Enablement and Learning here.
