The cas controller template include a toleration for workload.sas.com/class=cascontroller, and the worker template one for workload.sas.com/class=casworker, so you could use node taints to control the placement of one versus the other.
Outside of the taints and tolerations, workload placement is done using affinity. The CAS controller for example has a node affinity for the workload.sas.com/class=cas label and against nodes with the compute/stateful/stateless class labels. There is also an anti-affinity for other CAS pods, to prevent multiple CAS pods from running on the same node.
In your case you've stated:
- You have 5 nodes, 3 of which are master nodes.
- You would like to run CAS workers and stateful services on the three master nodes.
- You would like to run compute on one of the worker nodes and the controller on the other worker node.
Because the label workload.sas.com/class can only have one value, you would only be able to use that for the compute worker node.
The label workload.sas.com/class=cas would apply to both the workers and the controller, so either a worker or the controller would end up on the other worker node.
You could modify the nodeAffinity of the controllerTemplate and workerTemplate in the CASDeployment object using a transformer to have these target casworker and cascontroller labels, then use those to label the master nodes and worker node.
You could then similarly patch the statefulsets to have an affinity for the casworker label.
These are affinities without taints on the node and accompanying tolerations on the pods, so these workloads could still end up on other nodes.
--
Greg Wootton | Principal Systems Technical Support Engineer