The potential impact of a Kubernetes policy engine on SAS Viya

[#1]

Introduction

A Kubernetes policy engine can be a powerful tool to govern and secure a Kubernetes cluster. Whilst they are a powerful tool, having a policy engine in place can complicate a SAS Viya deployment in an instance. Take it from someone who has been around the block and has encountered these policy engines. If you want to learn more about how a policy engine can impact a SAS Viya deployment and learn about what a policy engine is and the current field of policies engines, keep on reading!

In this blog I will explain what a policy engine is, how it works, how these engines compare against each other, the types of policies that an engine can enforce, how policies can impact a SAS Viya deployment and how policies are added to the policy engine.

What is a Kubernetes policy engine?

A Kubernetes policy engine is a tool for keeping your cluster safe and secure. This is done through the definition of policies which are enforced by the engine. A policy determines what the end user is allowed to do on the cluster. Whenever a Kubernetes object is created by the end user a policy evaluates if the request is allowed, and the engine will act based on what has been defined within the policy.

An example of a policy that could be enforced by a Kubernetes Policy engine, is a policy that only allows container images to be pulled from a trusted container registry.

The added value of having a policy engine in place on a Kubernetes cluster is that it allows you to control how you deploy to Kubernetes and ensure that things are done up to your organizational standards through policies.

There are several open-source policy engines that can be deployed to a Kubernetes cluster. Currently the most popular ones are:

• Kyverno
• Open Policy Agent (OPA)
• jsPolicy

All three of these engines are good at what they do. Personally, I have encountered OPA and Kyverno while deploying SAS Viya.

Have you encountered other policy engines like jsPolicy? And what were your experiences?

Let me know in the comments!

How does a policy engine work?

To understand on a high-level how a Kubernetes Policy engine works, let me try to break it down into smaller steps. Let’s assume that we have a YAML file that contains the definition of a Pod. And we want to deploy this Pod to a Kubernetes cluster. Within the Kubernetes cluster a policy engine is running which will enforce a set of policies that are defined.

1. The user executes a kubectl command to apply the YAML file containing the definition of a Pod.
2. Then the request is forwarded to the policy engine running on the Kubernetes cluster.
3. The policy engine will process the request, and it will apply the available policies to the definition of the Pod.
4. If the Pod definition requires changes due to a policy in place, the policy engine will mutate the Pod definition.
5. If the Pod definition does not meet a validation policy in place, the policy engine will block the deployment of the Pod to the Kubernetes cluster.
6. If the Pod definition passes all policies, the Pod will be deployed to the Kubernetes cluster.

The types of policies.

The above description already reveals that there are two types of policies that can be applied by a Kubernetes policy engine.

Type of policy	Description
Mutating policies	A policy that can make an actual change to a Kubernetes Object definition
Validating policies	A policy that flags and blocks any Kubernetes Object that doesn’t comply with a specific policy

Let’s demonstrate the differences between these two by providing an actual example.

Use case

We work within an organization that uses Kubernetes, and it is only allowed to deploy container images from a trusted container registry to the cluster. We want to deploy a pod to the cluster that uses a container image from docker hub.

Given the above use case, let’s have a look at the results if we apply the two types of policies.

Policy	Result
Mutate the pod definition if it doesn’t use a trusted container registry	The pod will be deployed onto the cluster and the container image registry will be automatically changed to our trusted container registry
Validate if the pod is using a trusted container registry and block it if that is not the case	The pod will NOT be deployed to the cluster because it is using a registry that is not trusted by our organization.

It’s important to understand the nuance between these two different types. A validating policy will throw an error message, and it will clearly state why the policy has been triggered. A mutating policy can make changes without you knowing about it. The latter category can therefore be tricky to debug when it makes a change under the hood that prevents your SAS Viya deployment from running properly.

How do I add a policy to a policy engine?

This is very straight forward. All policies from the various policy engines have their own Custom Resource Definition (CRD). Creating a policy simply means creating an object of that CRD on the Kubernetes cluster through a YAML file.

Adding a policy to a policy engine is done through executing a command like

kubectl apply -f policy.yaml

In the previous paragraph I’ve used the example of a policy that allows only to pull images from a trusted registry. I’ve provided an example of such a policy for each policy engine here.

• Kyverno – policy.yaml
• OPA – policy.yaml
• jsPolicy – no example available

Mind your step! How policies can impact a SAS Viya deployment.

Example policy	Impact
Pods are only allowed to pull images from a trusted container registry	You will need to mirror all container images to the trusted container registry and apply the patch to have a deployment pull from that registry.
Each container in a pod needs to describe their resource usage by specifying request and limits	Almost all the pods in a SAS Viya deployment describe their resource usage. Two containers I’m aware of are part of the internal Postgres database
Pods are only allowed to mount volumes of specific types	hostPath is often on the list of volume types that isn’t allowed. If you plan on using hostPath for SAS Work or CAS Disk cache you will need to switch to a different alternative.
Only a specific set of users is allowed to run privileged pods	Most pods in a SAS Viya deployment don’t require privilege escalation. One I’m aware of is kaniko that is used to build container images when publishing from Model Manager.
Usage of emptyDir is only allowed by pods that specify a sizeLimit for the emptyDir	By default both CAS Disk Cache and SAS Work use emptyDir. If you plan on using emptyDir your deployment will be impacted. Consider switching to another method.
Certificates need to be renewed every 100 days	This impacts a SAS Viya deployment in two places. One the internal CA certificate that is used to encrypt all backend communications and the other one is the ingress certificate. There is no way around this one. But be aware renewing the internal CA certificate requires downtime.
Pods need to specify runasUser. If not specified a default user is injected into the pod definition	By default, all pods in a SAS Viya deployment use 1001 (sas) as the user to run all processes. However, this is not specified in the pod definition. If a different user is automatically injected into the pod definition the pods will not start. As a workaround you can write a patch that adds the runasuser attribute to the pod definition with the correct user 1001.
Only specific users are allowed to exec into a container	This will not impact your deployment but can tricky when you for example want to debug specific issue that you run into. In some cases, it can help the debug process to exec into the container.

The above list is not an exhaustive list of examples. As you can imagine there are numerous policies that can impact a SAS Viya deployment. Too many to list here. These are the ones that I have encountered in the field. Have you encountered different policies that impacted your SAS Viya deployment?

Let me know in the comments!

The important thing to note here is to make sure you understand which policies are in place through a policy engine before embarking on a deployment of SAS Viya.

Can I see which policies are in place?

That’s the million-dollar question! I’m sorry to say the answer is that it depends. Often in situations where there are policies in place, your permissions on the actual cluster will be limited. That means that you will not be able to check on the cluster what is in place.

The best course of action is to check with the Kubernetes Administrator who deployed the policy engine. But if you do have the permissions, you can try retrieving a list of the policies that are in place by running a kubectl command.

Policy engine	Command
Kyverno	kubectl get clusterpolicy
jsPolicy	kubectl get jspolicy
OPA	kubectl get constrainttemplate

While the above gives you a list of policies, the downside is that you need to list the content of each individual policy to understand what the policy does. And to make matters worse you need to check whether the policy is validating or mutating. My recommendation is to check with the Kubernetes administrator upfront.

Conclusion

Can a Kubernetes policy engine have an impact on a SAS Viya deployment. Most definitely! But it depends on the policies that are in place on the Kubernetes cluster. The important takeaway from this blog is to make sure you understand which policies are running on a cluster by reaching out to the Kubernetes cluster administrator!

Then you can assess how they will impact the deployment and create a plan to mitigate them. There are too many policies to list here but the list provided in this blog will hopefully give you some guidance on what to look out for.

Be aware of those mutating policies. They can make life hard for you. Respect the authority of the policy engine!