What Kubernetes permissions do I need to deploy and maintain SAS Viya ?

1 Like

A few weeks ago, I was contacted by my colleague Frederik (from the Global Technology Practice) as his large UK customer was challenging him on our requirements, in terms of permissions in Kubernetes for the deployment and administration of SAS Viya.

It made me realize that when we deploy SAS Viya internally at SAS or in our Deployment or Administration lab environments, we often do it with the Kubernetes administrator full access. With this kind of access, we can create but also delete any cluster-wide resource, such as someone else’s namespace, someone else persistent volume (where critical information might be stored), etc.

While it is very handy and make our life easier each time we need to deploy/redeploy or change anything in SAS Viya, it is quite unlikely the Kubernetes administrator, working for your customer, accepts to give you such kind of access (especially if the Kubernetes cluster is shared with other applications).

Instead, they will probably tell you: "Okay SAS admin, now could you tell me exactly what kind of RBAC* authorizations you need in my Kubernetes cluster to deploy your application?"

If you have experienced this situation or think you may face it sooner or later, you might find this blog useful. Its goal is to propose an approach where the Kubernetes administrator can delegate the administration of the SAS Viya environment to someone with limited privileges in the cluster.

(*) RBAC stands for Role Based Access Control, you can learn more about it there

Review of the existing resources

Viya ARK

The Viya ARK pre-installation playbook checks Creation/Deletion of: CRD, Role, RoleBinding, Deploy, Service, Ingress/Gateway objects.

For example, here is how the "Viya ARK Pre-Install check" report looks like when using a restricted service account (who is a namespace admin and has some additional cluster-wide read-only permissions on namespaces and PVs).

Select any image to see a larger version.

Mobile users: To view the images, select the "Full" version at the bottom of the page.

The report is consistent, since to perform the whole deployment, Cluster admin permissions are required. However, it does not provide answers to the customer questions discussed in the introduction of this post.

Official Documentation

In February 2022, a new section “Cluster Resources and Roles That Require Elevated Permissions” was added in the SAS Operations documentation to provide additional guidance on this topic.

While the doc section remains at a high level (and could be confusing to someone who is new to the Kubernetes RBAC model), it explains that the Kubernetes cluster administrator could create additional roles for delegated administrators who could then deploy and administrate the software.

One of the recommendations is to provide “a delegated, namespace-level administrator with the "get", "list," and "watch" permissions on all resources cluster-wide.”

With this level of permissions, the delegated administrator (let’s call him the SAS Viya administrator) can see all resources (including secrets) in every namespace of the cluster. It is not something that will always be acceptable for customers…and as we’ll see with practical examples, it is not really required.

Challenge and suggested approach

When the SAS consultant cannot have the cluster administration privileges to perform the SAS Viya deployment and administration, the following approach could be followed to address the customer’s requirement.

Use the manual deployment method.
Create a "SAS Viya admin" service account that will be used by the SAS consultant.
Give permissions to the "SAS Viya admin" service account in the SAS Viya namespace.
Split the initial manual deployment commands (depending on their "scope") between the Kubernetes Administrator and the SAS consultant.
Only if needed, add specific permissions as needed for additional administration tasks.

What if I want to use an orchestrated deployment method ?

Important : Splitting the initial deployment tasks between the K8s admin and the SAS Consultant (for example with a namespace scoped privileges) is only possible with the manual deployment method.

If the chosen deployment method is the sas-orchestration, then the "cluster level" privileges are required each time the software has to be redeployed…which is something that happens each time you need to update your kustomization.yaml file and force SAS Viya to pick up configuration changes.

If the chosen deployment method is the SAS Viya Platform Deployment Operator, then the re-deployment is always managed through a reconciliation process that is handled by the deployment operator and run under the Deployment Operator's Cluster-role.

Scopes and sas.com/admin labels

The SAS Viya platform is made up of various Kubernetes objects.

Some objects have a “cluster-wide” scope (they are global for the whole cluster and not limited to a specific namespace), others have a "namespace" scope.

In the SAS Viya manifest (site.yml), each object is in a category (or scope) determined by a "sas.com/admin" label.

There are 4 categories (listed in the table below that comes from the SAS official documentation). However, these categories don’t necessarily match what Kubernetes considers as "namespaced" object (Configmaps, Pods, PodTemplates, Services, etc..) and "non-namespaced" objects (Nodes, PersistentVolumes, ClusterRoles, clusterRolebinding, etc…). As an example, apart from one exception, the "cluster-local" category is associated to "namespaced" objects.

The selectors "sas.com/admin=cluster-wide" and "sas.com/admin=cluster-api" are associated to Cluster level objects.

It means that the kubectl apply command, that target these kinds of objects, can only be executed by someone with the default cluster-admin ClusterRole bound with a ClusterRoleBinding (which gives privileges at the cluster level – i.e. your almighty Kubernetes Administrator).

The selector "sas.com/admin=namespace" is associated with "namespace level" objects. So, someone with a namespace administrator role for the specific namespace, could run the kubectl apply command that targets these objects.

Then we have the "sas.com/admin=cluster-local" level, which is a mixed bag of various kinds of objects that also exist in the other categories. Since there is generally at least one object with this label that is at the cluster-wide level, a namespace administrator role would NOT be able to run the kubectl apply command that target the "cluster-local" scope with full success…

The issue with the "cluster-local" label

This selector is a bit more problematic since it contains 99% of namespace level object ("configmaps", "secrets", "persistentvolumeclaims", "rolebindings", "podtemplates") but also potentially cluster-wide objects (ClusterRoleBinding for the SAS Workload Orchestrator).

Actually, in our test environment (with the "SAS® Viya®" offering), the only object that prevented us successfully applying the "sas.com/admin=cluster-local" level as a namespace administrator is the "sas-workload-orchestrator" ClusterRoleBinding object.

The issue with this "cluster-local" label is that : if for a given Viya configuration/customization, we need to modify an object that is at this level, then it will be necessary to run the kubectl apply command at the "cluster-local" level…So our namespace level Viya administrator will have to call the Kubernetes admin just to run the kubectl apply at this level (even if the modified object is at the namespace level ! ) ☹️

Removing the "cluster-local" label and assigning the associated resources either to "cluster-wide" or "namespace" categories, would greatly simplify things as it would allow Viya administrators with namespace privileges to perform almost any kind of Viya configuration/customization.

Required role and additional permissions

A recommendation of the official SAS Documentation is to provide "a delegated, namespace-level administrator with the "get", "list," and "watch" permissions on all resources cluster-wide."

Ideally, the namespace administration privileges would be enough (without the need to give permission on all resources cluster-wide). What we can do is to use one of the Kubernetes existing "built-in" roles.

As noted in the official Kubernetes RBAC documentation, the "cluster-admin" built-in ClusterRole "when used in a RoleBinding, gives full control over every resource in the role binding's namespace, including the namespace itself."

Here is an example on how to give this role to a "viya-admin-lab" service account.

# create the manifest
    tee /tmp/create-binding-for-viya-admin-lab.yaml > /dev/null << EOF
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: sas-rolebinding-namespace
      namespace: lab
    subjects:
    - kind: ServiceAccount
      name: viya-admin-lab
      namespace: lab
    roleRef:
      kind: ClusterRole
      name: cluster-admin
      apiGroup: rbac.authorization.k8s.io
    EOF

# apply the manifest
    kubectl apply -f /tmp/create-binding-for-viya-admin-lab.yaml

We simply create a "RoleBinding" object to attach the built-in "cluster-admin" ClusterRole to our "lab" namespace. So, while it is a "ClusterRole", it can be limited to the scope of a particular namespace.

With the above privilege definition, someone using our "viya-admin-lab" service account could change everything inside our Viya namespace but could not do anything else outside of the namespace.

Note: Here we assume that we have already created the "viya-admin-lab" service account and generated the associate KUBECONFIG file.

Break down the responsibilities

The table below shows the breakdown of tasks between the Kubernetes administrator and our SAS Viya administrator using the service account described above (with his privileges limited to the SAS Viya namespace).

Task	Responsibility	Notes
Deploy SAS Viya	Shared	The Kubernetes administrator runs the "kubectl apply" commands for "cluster-api", "cluster-wide" and "cluster-local" levels. The SAS Viya administrator runs "kubectl apply" command for the namespace level (and potentially for the "cluster-local" level).
Update SAS Viya	Shared
Stop/Start the SAS Viya platform	SAS Viya administrator (namespace admin)	This task is handled by Cronjobs defined at the namespace level.
Backup and restore the SAS Viya environment	SAS Viya administrator (namespace admin)
Perform common customizations (change CAS or SAS programming runtime settings, change CAS topology, etc…)	SAS Viya administrator (namespace admin)	While the SAS Viya administrator cannot change the Custom Resource Definitions (CASDeployment definition), he can modify Custom Resource instances (CASDeployment instances), which is enough to change the configuration of a deployed CAS Server.
Allocate additional permissions to the SAS Viya administrators when/if needed for specific configuration.	Kubernetes Administrator	We haven’t tested all possible configuration/customization. Some may exceptionally require additional privileges. The additional permissions would be added as required.

Note: The described tasks have all been tested in the GEL Deploying SAS Viya workshop Hands-on.

Conclusion

Hopefully this post helped you to understand which kind of authorization is required in Kubernetes for a SAS Consultant or a SAS Administrator to accomplish his tasks related to the SAS Viya deployments.

The content presented here comes from practical experimentation in a Viya LTS 2023.10 environment deployed on an Upstream Opensource Kubernetes cluster (in version 1.27).

If you want to see these concepts by yourself in practice, note that the SAS Global Enablement and Learning team is developing specific SAS® Viya 4 deployment hands-on content within workshop style content. Contact your local SAS Education team for more details if this is of interest to you.,.

Bonus: Some useful commands to check permissions and scope

Finally, as a little bonus, you can find below some useful commands to explore the Kubernetes RBAC implementation.

"kubectl-whoami" and "kubectl auth can-i" commands.

# the whoami plugin can be installed with krew ()
kubectl-krew install whoami
kubectl whoami
# can I create pods ?
kubectl auth can-i create pod -n lab
# can I create the resources corresponding to the "sas.com/admin=namespace" label in the SAS Viya manifest ?
for res in service deployment replicaset replicationcontroller job cronJob statefulset daemonset podtemplate
do
        # test the create privilege for each resource
        echo "testing creation right on $res"
        kubectl auth can-i create $res -n lab
done

See Rob Collum’s post for more about Kubernetes identity and KUBECONFIG files.

List "namespaced" objects.

kubectl api-resources -o wide
kubectl api-resources --namespaced=false
kubectl api-resources --namespaced=true

List objects in the "cluster-local" scope:

kubectl api-resources --verbs=list -o name   | xargs -n 1 kubectl get --show-kind --ignore-not-found -l "sas.com/admin=cluster-local" -n lab
# in all resources that support the "list" request verbs

Find more articles from SAS Global Enablement and Learning here.