Deploying SAS Viya in a Shared Kubernetes Cluster Using Tolerations and Node Affinity

3 Likes

Kubernetes normally decides where pods should run based on available resources and the scheduling constraints defined for each workload.

For a standard SAS Viya deployment, SAS strongly recommends labeling and tainting the Kubernetes nodes according to the SAS Viya workload-placement plan, especially nodes that host Compute and CAS workloads. This enables the different SAS Viya workload classes to be placed on appropriate nodes.

For more information, see Plan the Workload Placement in the SAS Viya Platform Deployment Guide.

However, there are situations where a more customized workload-placement strategy can be useful.

One example is a shared on-premises or bare-metal Kubernetes cluster that has spare capacity available on one or more physical nodes. Rather than provisioning a completely separate Kubernetes cluster for a small DEV or UAT environment, it might be desirable to deploy the non-production SAS Viya environment into the existing cluster while restricting its workloads to specifically designated nodes.

In this article, I will demonstrate how the following Kubernetes scheduling mechanisms:

node labels
nodeSelector
node affinity
taints
tolerations

can be combined with Kustomize PatchTransformer resources to place a SAS Viya UAT deployment onto a designated Kubernetes node. The example uses the following custom node label:

sas=uat

and the following taint:

sas=uat:NoSchedule

The designated node in this example is:

k8s-node-uat

The same approach can be extended to multiple nodes by applying the same label and taint to each node.

Important: The scenario described here has a different objective: deliberately isolating a smaller DEV or UAT deployment onto spare capacity in a shared cluster. It should therefore be considered a targeted non-production topology rather than a general SAS Viya bare-metal sizing recommendation.

The patch transformers demonstrated in this article were developed and tested against Viya LTS 2026.03 cadence release.

The Kubernetes resources, custom resources, and JSON Patch paths used by Viya can change between cadence releases. Therefore, these transformers should be treated as examples for LTS 2026.03 rather than as generic patches that can automatically be applied to every Viya release.

Taint the UAT node

The following taint is applied to the node:

sas=uat:NoSchedule

A NoSchedule taint prevents Kubernetes from scheduling new pods onto the node unless those pods contain a matching toleration. Existing pods are not evicted.

Therefore, the taint helps prevent normal application workloads from consuming the capacity that has been reserved for UAT.

Add a toleration to the UAT pods

The UAT workloads contain:

tolerations:
  - key: sas
    operator: Equal
    value: uat
    effect: NoSchedule

This means that the UAT pods are permitted to run on the tainted node.

However, a toleration does not instruct Kubernetes to place the pod on that node. It only removes the scheduling restriction created by the taint. Kubernetes documentation specifically notes that tolerations allow scheduling but do not guarantee scheduling. This is why another scheduling constraint is required.

Add a nodeSelector

For standard Kubernetes resources in this example, the pod specification contains:

nodeSelector:
  sas: uat

Kubernetes only schedules that pod onto nodes containing the matching label. nodeSelector is the simplest Kubernetes mechanism for constraining a pod to nodes with particular labels.

Kubernetes Scheduling Mechanism Matrix

Achieving strict workload isolation in a shared cluster requires combining several Kubernetes scheduling features. The matrix below outlines how these mechanisms work in tandem for this UAT topology:

Label and Taint the UAT Node

First apply the node label:

kubectl label node k8s-node-uat sas=uat --overwrite

Then apply the taint:

kubectl taint node k8s-node-uat sas=uat:NoSchedule --overwrite

Confirm the node label:

kubectl get node k8s-node-uat --show-labels

You should see:

sas=uat

The taint can be checked with:

kubectl describe node k8s-node-uat

Look for:

Taints: sas=uat:NoSchedule

At this point, pods without an appropriate toleration should generally no longer be scheduled onto the node.

Disable SAS Workload Orchestrator

For this UAT topology, SAS Workload Orchestrator is not required. Disabling it also avoids the requirement to provide a Kubernetes node with the SAS Workload Orchestrator compute-class label. Follow the procedure in the SAS documentation to disable SAS Workload Orchestrator: Disable or Enable SAS Workload Orchestrator

With SAS Workload Orchestrator disabled, the UAT node does not need to be labeled with:

workload.sas.com/class=compute

SAS Viya Requires More Than One Transformer

Depending on the deployed products and release, workloads can originate from:

Standard Deployment, StatefulSet and DaemonSet Workloads

The first transformer covers the standard Kubernetes workloads.

# ---------------------------------------------------------------------------
# Dedicates SAS Viya workloads to nodes labeled/tainted sas=uat
#
# Node prerequisite:
#   kubectl label node k8s-node-uat sas=uat --overwrite
#   kubectl taint node k8s-node-uat sas=uat:NoSchedule --overwrite
#
# Covers Deployment, StatefulSet, and DaemonSet workloads.
# ---------------------------------------------------------------------------

# --- Tolerations ---

apiVersion: builtin
kind: PatchTransformer
metadata:
  name: add-tolerations-to-deployments
patch: |-
  - op: add
    path: /spec/template/spec/tolerations
    value:
      - key: "sas"
        operator: "Equal"
        value: "uat"
        effect: "NoSchedule"
target:
  kind: Deployment
---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: add-tolerations-to-statefulsets
patch: |-
  - op: add
    path: /spec/template/spec/tolerations
    value:
      - key: "sas"
        operator: "Equal"
        value: "uat"
        effect: "NoSchedule"
target:
  kind: StatefulSet
---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: add-tolerations-to-daemonsets
patch: |-
  - op: add
    path: /spec/template/spec/tolerations
    value:
      - key: "sas"
        operator: "Equal"
        value: "uat"
        effect: "NoSchedule"
target:
  kind: DaemonSet

# --- nodeSelector ---

---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: add-nodeselector-to-deployments
patch: |-
  - op: add
    path: /spec/template/spec/nodeSelector
    value:
      sas: "uat"
target:
  group: apps
  kind: Deployment
---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: add-nodeselector-to-statefulsets
patch: |-
  - op: add
    path: /spec/template/spec/nodeSelector
    value:
      sas: "uat"
target:
  group: apps
  kind: StatefulSet
---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: add-nodeselector-to-daemonsets
patch: |-
  - op: add
    path: /spec/template/spec/nodeSelector
    value:
      sas: "uat"
target:
  group: apps
  kind: DaemonSet

Jobs and CronJobs

SAS Viya also contains Jobs and CronJobs.

# ---------------------------------------------------------------------------
# Forces Job and CronJob workloads onto the node labelled sas=uat.
# ---------------------------------------------------------------------------


apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-uat-jobs-existing-affinity
patch: |-
  - op: add
    path: /spec/template/spec/tolerations/-
    value:
      key: sas
      operator: Equal
      value: uat
      effect: NoSchedule
  - op: add
    path: /spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/-
    value:
      key: sas
      operator: In
      values:
        - uat
target:
  group: batch
  version: v1
  kind: Job
  name: "^sas-(commonfiles|crunchy-.*)$"


---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-uat-openssl-job
patch: |-
  - op: add
    path: /spec/template/spec/tolerations
    value:
      - key: sas
        operator: Equal
        value: uat
        effect: NoSchedule
  - op: add
    path: /spec/template/spec/affinity
    value:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: sas
                  operator: In
                  values:
                    - uat
target:
  group: batch
  version: v1
  kind: Job
  name: "^sas-create-openssl-ingress-certificate$"


---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-uat-cronjobs
patch: |-
  - op: add
    path: /spec/jobTemplate/spec/template/spec/tolerations/-
    value:
      key: sas
      operator: Equal
      value: uat
      effect: NoSchedule
  - op: add
    path: /spec/jobTemplate/spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/-
    value:
      key: sas
      operator: In
      values:
        - uat
target:
  group: batch
  version: v1
  kind: CronJob

Notice that some of these resources already have affinity structures.

Instead of replacing them, the transformer adds another matchExpression:

- key: sas
  operator: In
  values:
    - uat

This uses:

requiredDuringSchedulingIgnoredDuringExecution

which is a hard node-affinity requirement. Kubernetes will not schedule the pod unless the node satisfies the expression. This performs the same hard placement role as our nodeSelector, but node affinity supports a more expressive syntax.

Launcher PodTemplates

One of the easiest workload types to overlook is the Kubernetes PodTemplate.

SAS Launcher uses pod templates when dynamically creating several types of SAS Viya workloads.

For example, this can include Compute Server and Connect sessions.

The following transformer handles the PodTemplates used in this example:

# ---------------------------------------------------------------------------
# Forces SAS Launcher-spawned Jobs onto the node labelled sas=uat, by
# patching the PodTemplate resource
#
# Prerequisite:
#   kubectl label node k8s-node-uat sas=uat --overwrite
#   kubectl taint node k8s-node-uat sas=uat:NoSchedule --overwrite
#
# ---------------------------------------------------------------------------

apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-uat-podtemplates
patch: |-
  - op: add
    path: /template/spec/tolerations/-
    value:
      key: sas
      operator: Equal
      value: uat
      effect: NoSchedule
  - op: add
    path: /template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/-
    value:
      key: sas
      operator: In
      values:
        - uat
target:
  kind: PodTemplate
  name: "^(sas-admin-content-loader|sas-batch-cmd-pod-template|sas-batch-pod-template|sas-cas-pod-template|sas-catalog-job|sas-compute-job-config|sas-connect-pod-template|sas-data-agent-server-colocated|sas-job-flow-scheduling-flow-orchestrator-pod-template|sas-launcher-job-config|sas-model-builder-job|sas-process-exporter-template|sas-pv-backup|sas-pv-restore|sas-qkb-bootstrap)$"

---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-uat-podtemplate-prepull
patch: |-
  - op: add
    path: /template/spec/tolerations/-
    value:
      key: sas
      operator: Equal
      value: uat
      effect: NoSchedule
  - op: add
    path: /template/spec/affinity
    value:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: sas
                  operator: In
                  values:
                    - uat
target:
  kind: PodTemplate
  name: "^sas-prepull$"

CASDeployment

CAS is managed through the SAS CASDeployment custom resource.

For the deployment used in this example, the following patch adds the UAT toleration and required node affinity to the CAS controller template:

# ---------------------------------------------------------------------------
#
# Prerequisite:
#   kubectl label node k8s-node-uat sas=uat --overwrite
#   kubectl taint node k8s-node-uat sas=uat:NoSchedule --overwrite
#
# ---------------------------------------------------------------------------

apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-uat-casdeployment-controller
patch: |-
  - op: add
    path: /spec/controllerTemplate/spec/tolerations/-
    value:
      key: sas
      operator: Equal
      value: uat
      effect: NoSchedule
  - op: add
    path: /spec/controllerTemplate/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/-
    value:
      key: sas
      operator: In
      values:
        - uat
target:
  group: viya.sas.com
  version: v1alpha1
  kind: CASDeployment

OpenDistroCluster

The deployment used for this example also contains an OpenDistroCluster custom resource.

The transformer is:

# ---------------------------------------------------------------------------
#
# Prerequisite:
#   kubectl label node k8s-node-uat sas=uat --overwrite
#   kubectl taint node k8s-node-uat sas=uat:NoSchedule --overwrite
#
# ---------------------------------------------------------------------------

apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-uat-opendistrocluster
patch: |-
  - op: add
    path: /spec/template/spec/tolerations/-
    value:
      key: sas
      operator: Equal
      value: uat
      effect: NoSchedule
  - op: add
    path: /spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/-
    value:
      key: sas
      operator: In
      values:
        - uat
target:
  group: opendistro.sas.com
  version: v1alpha1
  kind: OpenDistroCluster

Important: The exact search-related custom resources used by SAS Viya can vary by release.

This is another reason that workload placement transformers should be reviewed whenever the deployment assets are updated.

Crunchy PostgreSQL

The following transformer modifies the PostgresCluster custom resource. Three workload categories are covered:

PostgreSQL instances
pgBackRest repository
pgBackRest jobs

# ---------------------------------------------------------------------------
# Forces Crunchy PostgresCluster workloads onto the node labelled sas=uat.
#
# Prerequisite:
#   kubectl label node k8s-node-uat sas=uat --overwrite
#   kubectl taint node k8s-node-uat sas=uat:NoSchedule --overwrite
# ---------------------------------------------------------------------------

apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-uat-postgrescluster-instances
patch: |-
  - op: add
    path: /spec/instances/0/tolerations/-
    value:
      key: sas
      operator: Equal
      value: uat
      effect: NoSchedule
  - op: add
    path: /spec/instances/0/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/-
    value:
      key: sas
      operator: In
      values:
        - uat
target:
  group: postgres-operator.crunchydata.com
  version: v1beta1
  kind: PostgresCluster
---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-uat-postgrescluster-repohost
patch: |-
  - op: add
    path: /spec/backups/pgbackrest/repoHost/tolerations/-
    value:
      key: sas
      operator: Equal
      value: uat
      effect: NoSchedule
  - op: add
    path: /spec/backups/pgbackrest/repoHost/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/-
    value:
      key: sas
      operator: In
      values:
        - uat
target:
  group: postgres-operator.crunchydata.com
  version: v1beta1
  kind: PostgresCluster
---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: sas-uat-postgrescluster-jobs
patch: |-
  - op: add
    path: /spec/backups/pgbackrest/jobs/tolerations/-
    value:
      key: sas
      operator: Equal
      value: uat
      effect: NoSchedule
  - op: add
    path: /spec/backups/pgbackrest/jobs/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/-
    value:
      key: sas
      operator: In
      values:
        - uat
target:
  group: postgres-operator.crunchydata.com
  version: v1beta1
  kind: PostgresCluster

Add the Transformers to kustomization.yaml

The transformer files can be stored under a site configuration directory such as:

site-config/
└── sas-uat/
    ├── sas-uat-node-workload-transformers.yaml
    ├── job-and-cronjob-sas-uat-transformer.yaml
    ├── podtemplate-sas-uat-transformer.yaml
    ├── casdeployment-sas-uat-transformer.yaml
    ├── opendistrocluster-sas-uat-transformer.yaml
    └── postgrescluster-sas-uat-transformer.yaml

They can then be referenced from the deployment’s kustomization.yaml:

transformers:

transformers:
  - site-config/sas-uat/sas-uat-node-workload-transformers.yaml
  - site-config/sas-uat/job-and-cronjob-sas-uat-transformer.yaml
  - site-config/sas-uat/podtemplate-sas-uat-transformer.yaml
  - site-config/sas-uat/casdeployment-sas-uat-transformer.yaml
  - site-config/sas-uat/opendistrocluster-sas-uat-transformer.yaml
  - site-config/sas-uat/postgrescluster-sas-uat-transformer.yaml

Conclusion

Kubernetes gives administrators several complementary mechanisms for controlling workload placement.

For the UAT scenario demonstrated here:

Node label:

sas=uat

identifies the permitted nodes.

Node taint:

sas=uat:NoSchedule

discourages unrelated workloads from consuming those nodes.

tolerations:
  - key: sas
    operator: Equal
    value: uat
    effect: NoSchedule

allows the UAT workloads to use the nodes.

Finally:

nodeSelector:
  sas: uat

or the equivalent required node affinity ensures that the UAT workloads remain on the designated nodes.

For organizations operating shared on-premises Kubernetes infrastructure, this can provide a useful way to take advantage of spare physical capacity for a smaller DEV or UAT SAS Viya deployment without allowing that environment to freely consume nodes intended for other workloads.