SAS® Workload Orchestrator- Associate Kubernetes Cluster Nodes With Queues

3 Likes

As of the stable 2023.08 release, the SAS® Workload Orchestrator offering is now included, and enabled by default, as part of the SAS® Viya platform deployment.

SAS® Workload Orchestrator is the component that provides the SAS Viya administrator with the ability to manage the SAS Programming Runtime (SPRE) workload (SAS Compute Server, SAS Connect Server, and SAS Batch Server). When enabled, workload is managed using queues that offer multiple settings to fine-tune how the workload will be scheduled.

As a SAS Viya administrator, I may want to redirect workload to specific Kubernetes cluster nodes to ensure service and performance to certain groups of users. Using queues configuration, the SAS Viya administrator can now associate Kubernetes cluster nodes to a specific queue which will force any workload sent to that queue to wind up on its associated node. In this blog, I will show you an example of how to configure nodes and queues to accomplish this.

Workload managed by queues

As soon as the SAS® Workload Orchestrator solution is deployed and enabled in your SAS Viya deployment each SPRE job that is submitted for processing, whatever its SAS Viya server context, is placed in a queue. The job remains in the queue until resources are available for the job to be executed. All jobs submitted to the same queue share the same scheduling, control policy, and resources. When SAS® Workload Orchestrator is enabled, the Queues become the primary method to manage the SAS Viya deployment workload.

By default, there is a single queue that is defined for the SAS® Workload Orchestrator: the default queue. If no other queues are defined, all workload is automatically sent against this default queue.

For better workload control, the SAS Viya administrator can define additional queues for different priorities and types of workloads using the Workload Orchestrator application in SAS Environment Manager.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

Note: For more details about queue parameters, please look at this documentation: queues configuration

Depending on which queue the jobs are submitted, SAS® Workload Orchestrator chooses which job to process based on the queue priority and the availability of required resources.

SAS Viya server contexts can be configured to use specific queues. If a queue is not specified, the default queue is used.

However, this is not the only way to manage the workload. Queues and server contexts are used to prioritize the workload, but additional configuration can be used to ensure that the workload will be directed to specific resources in the Kubernetes cluster.

Direct workload to specific nodes

Assigning different hosts or host types to different queues ensures that all jobs submitted to these queues will be executed on specific hosts. Scenarios where this approach may apply are:

Direct jobs to specific nodes.

That provide specific resources (GPU, CPU, memory, disk).
Fully dedicated to a business unit (ensure that each business unit has access to its required resources).

Better manage the workload in your SAS Viya deployment (i.e. low-priority jobs can be executed on specific hosts without having to be impacted by high-priority jobs that are executed on other hosts).

This strategy requires extra settings for both the Kubernetes cluster and the SAS® Workload Orchestrator solution.

The scenario to illustrate these settings

I have a SAS Viya deployment with two nodes dedicated to computing (intnode04 and intnode05).
I need to ensure that two business units (Sales and HR) will have access to a dedicated computing host.

Business Unit

Queue

Host Type

Kubernetes Cluster Node

HR

HR Host

intnode04

Sales

Sales Host

intnode05

Other jobs

default

intnode04 and intnode05 depending on availability and priority (lower)

Kubernetes settings

A few Kubernetes configuration tasks have to be done first.

The sas-workload-orchestrator ClusterRoles and ClusterRoleBindings have to be applied in your SAS Viya deployment.

These settings allow the SAS® Workload Orchestrator to be able to:

Match host-to-host type based on host properties (access the nodes' labels).
Have an accurate picture of the resources on the nodes (all Kubernetes components, not only for pods started by SAS® Workload Orchestrator).
Be able to correctly scale nodes.

Define nodes' labels to control where SAS® Workload Orchestrator jobs run

Jobs managed by SAS® Workload Orchestrator can run only on nodes where the sas-workload-orchestrator daemonSet is deployed. This daemonSet can be deployed on only nodes that contain the label workload.sas.com/class=compute.

In my scenario, both intnode04 and intnode05 are labeled workload.sas.com/class=comput

A label has to be assigned to any nodes we want to direct workload to (must be repeated when new nodes(pools) are added).

In my scenario, intnode04 is labeled nodeJobPriority=hr, and intnode05 is labeled nodeJobPriority=sales.

Workload Orchestrator configuration

Now we need to define Host-Types and Queues in SAS® Workload Orchestrator to complete our configuration tasks.

Define a host type with a Host properties parameter that matches the label we assigned to targeted nodes. This ensures that only specific nodes will be used. host type is a configuration that refers to a group of one or more similar nodes.

In my scenario, I must create two new Host Types (HR Host and Sales Host). Each Host Types must be set with a specific Host properties value to ensure that it points to a specific labeled node (nodeJobPriority=hr for HR Host, and nodeJobPriority=sales for Sales Host)
In a queue definition, specify a Host type into the Host types parameter to ensure that this queue will only use specific nodes for its jobs execution.

In my scenario, I must create two new Queues (HR and Sales). Each Queues must be set with a specific Host types value to ensure its jobs will be submitted against a specific labeled node (HR Host for HR, and Sales Host for Sales

As a result, we can observe that...

Any workload sent to the HR queue is directed to the intnode04 node that is labeled nodeJobPriority=hr.

Any workload sent to the Sales queue is directed to the intnode05 node that is labeled nodeJobPriority=sales.

The default workload is submitted to both intnode04 and intnode05 nodes depending on priority and resources availability.

Since the SAS® Workload Orchestrator is provided with SAS Viya, the administrator can use it to better manage the workload using several strategies.

I hope this article has been helpful to you.

Special thanks to SAS Technical writers, @RobCollum, @AjmalFarzam, and @EdoardoRiva

References:

SAS documentation:

SAS® Viya® Platform Administration

SAS Communities:

Scaling to new heights. Exploring the auto-scaling capabilities of Viya Workload Management (@EoinByrne)

touwen_k

@GillesChrzaszcz

As we do not have experience yet with SAs Workload Orchestrator, we would like to know what the best practice is. We are not going to use preemption yet as you have to find out how to program jobs to restart. So if we schedule priority jobs and the other less important jobs fail because they are too long in a pending state, what is best practice if you want these less important jobs to run as well? Is extending the default time-out of compute session the best practice of are there another solutions? Pls advise as we are eager to use these feature.