When deploying SAS Viya, the default behavior schedules both stateful and stateless workloads across any available node pools. Generally, this setup is ideal, as Kubernetes is tasked with efficiently allocating pods to the appropriate nodes. However, there may be some edge cases where stricter scheduling of workloads is required. For example, it may be desirable to segregate stateful and stateless workloads to better utilise resources across different node pools, such as in situations where solutions like SAS Visual Investigator (with pods that demand more storage or memory) are deployed across node pools of varying sizes. In this post, we'll talk about confining stateful workloads to stateful node pools, and stateless workloads exclusively to stateless node pools.
Ensuring Workload Separation
The primary challenge lies in controlling where different types of pods are scheduled. By default, both StatefulSet (stateful) and Deployment (stateless) pods tolerate both stateful and stateless node classes. This flexibility can sometimes result in stateful pods being scheduled on stateless nodes, which might not have appropriate resource allocations, and could potentially lead to throttling or pod eviction during high activity from stateless pods.
Two approaches can be taken to address this challenge.
Setting up required node affinity
Node affinity allows us to define rules that specify where pods should be scheduled, based on the labels applied to nodes. This approach offers more granularity and flexibility, allowing for strict scheduling of workloads on particular nodes. In this case, we can switch the affinity rules from 'preferred' to 'required', as covered by my colleague Raphaël Poumarede in his post.
Begin by labeling the nodes to distinguish between stateful and stateless node pools.
kubectl label nodes workload.sas.com/class=stateful
kubectl label nodes workload.sas.com/class=stateless
Add the following node affinity rules to StatefulSets to ensure these pods are scheduled only on stateful nodes.
spec:
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: workload.sas.com/class
operator: In
values:
- stateful
Similarly, ensure that stateless pods (Deployments) are scheduled only on stateless nodes by modifying the YAML templates as follows:
spec:
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: workload.sas.com/class
operator: In
values:
- stateless
Rebuild and redeploy to apply these changes.
Modifying tolerations
Tolerations work with taints applied to nodes, determining which pods are allowed to run on nodes with specific taints. By default, stateful and stateless pods tolerate both node classes. Modifying tolerations provides an alternative, simpler method of enforcing workload placement.
We can modify (remove) the tolerations in the YAML files that define the pods to achieve segregation of stateless and stateful workloads.
Here’s a sample patch to remove the stateful toleration from all Deployment resources to confine stateless pods to nodes optimised for stateless workloads.
---
apiVersion: builtin
kind: PatchTransformer
metadata:
name: remove-stateful-toleration
patch: |-
- op: remove
path: /spec/template/spec/tolerations
value:
- effect: NoSchedule
key: workload.sas.com/class
operator: Equal
value: stateful
target:
group: apps
kind: Deployment
version: v1
An update to kustomization.yaml is also required:
...
transformers:
- remove-stateful-toleration.yaml # Path to your patch file
Once applied, this patch will ensure that deployment resources are no longer scheduled on stateful nodes, improving the separation of workloads and enforce the changes across the environment.
Choosing the right approach
Both approaches outlined here offer effective methods for ensuring that stateful and stateless workloads are scheduled on the appropriate node pools in SAS Viya deployments. Other types workloads can also be segregated as desired. Note, however, that the default behaviour of scheduling across any available node pool is usually perfectly appropriate.
Required affinity provides more flexibility and control to enforce scheduling rules based on node labels, which can be easily adjusted if new node pools are added or removed. There is, however, an added administration overhead of maintaining node labels.
Modifying tolerations offers a simpler method for restricting pod placement by removing tolerations for nodes to repel unsuitable pods, but there is somewhat less flexibility, such as in handling changes to node pools.
Either method (or a combination of both methods) can be used to ensure stricter workload distribution in your SAS Viya environment with pods more efficiently distributed across node pools.
My thanks to Raphaël Poumarede, Harshit Soni and Henrique Lima for their contributions.
Find more articles from SAS Global Enablement and Learning here.
