The Interdependence of SAS Viya and Kubernetes Administrators

[#1]

Like it or not, SAS Viya 4 is going to have a tremendous impact on every Viya administrator.  No longer will a Viya admin be able to lord over a domain of dedicated hardware without regard for other applications or administrators.   Viya 4 will only run in a Kubernetes cluster which will likely host other application and be managed by a Kubernetes admin.  Many of the tasks usually performed by a Viya admin such as monitoring services, examining logs, and starting/stopping/restarting services are going to rely on existing cluster management tools such as Grafana, Kibana, and kubectl commands.  The tasks still need to be done but getting them done will require cooperation and collaboration with the Kubernetes admin who will need to enable the Viya admin with sufficient access and authorization to the tools.  Ultimately, the line blurs between the responsibilities a Viya admin is handles and those a Kubernetes admin handles.  The fact that these two areas of responsibility overlap in many cases creates an interdependence between the two roles.

 

Personas

Consider the Kubernetes admin for a moment. With a focus and responsibility for the entire cluster, they

• Are responsible for the heath, safety, and performance of the entire Kubernetes cluster.  They will not be happy with applications that consume excessive resources that affect other applications.
• Deal with all applications running within the cluster, not just SAS Viya.  They will not want to have to use additional tools that are application specific.
• Have an expectation that applications running within the cluster behave in a certain way.  They are used to using applications like Grafana, Prometheus, and Kibana to manage the cluster and will expect all applications to fit seamlessly into their existing infrastructure.  They do not have time to keep track of special case situations for each and every application.
• May, but probably will not, have any experience with SAS Viya.  They probably will not know how CAS pods differ from Go service pods.

  Consider the SAS Viya admin. With a primary focus on SAS Viya, they

• Are responsible for the health, safety, and performance of the SAS Viya application.  They do not really care so much about other applications as long as their SAS Viya users are happy.
• Know a lot about Viya and how to deal with the SAS Viya application and its resources.  They are familiar with the workload patterns of CAS, the Compute service, and how SAS Studio usage differs from Visual Analytics usage.
• May not be overly familiar with with tools like Grafana, Prometheus, and Kibana that they will now need to interact with.  They are more used to using SAS Environment Manager, the sas-admin command line, or custom scripting.  With Viya 4, the Viya admin will probably need help either accessing these supporting tools or, if not given access, will depend on the Kubernetes admin for tasks they used to handle.
• May, but probably will not, have much experience with Kubernetes or containers.  They will struggle when told they cannot just exec into a pod and look at something on the file system.

[I am going to refer to the Viya admin and the Kubernetes admin in singular form throughout this post, fully aware that each persona may involve multiple people.]  

To oversimplify it, the Viya admin is responsible for a portion of the cluster while the Kubernetes admin is responsible for the entire cluster, including Viya.

However, in order for both admins to be successful, they will need to help each other and combine their domain knowledge to solve problems.  The Viya admin may need to help the Kubernetes admin understand workload patterns in Viya in order to properly allocate certain Viya pods to appropriate nodes.  Think CAS...without knowing something about how it works, the Kubernetes admin may not understand that they should separate worker pods or try to bind them to nodes with sufficient resources.  Likewise, the Kubernetes admin may need to help the Viya admin by performing certain actions on behalf of the Viya admin that require elevated cluster privileges.  Or it may be necessary for the Kubernetes admin to grant limited privileges to the Viya admin to allow them to self-manage a Viya namespace.  

There are a few areas where one can envision situations where the Viya admin and Kubernetes admin will need to collaborate including:

• Monitoring
• Logging
• Starting, stopping, or restarting services.

Monitoring

So who will be responsible for monitoring Viya 4?  I believe the Viya admin and the Kubernetes admin will both participate in monitoring activities.  Because Viya is but one  part of the cluster, the Kubernetes admin monitoring the cluster will be looking at the metrics for  Viya as a part of the bigger picture.  Similarly, the Viya admin will want to see how things are going in the Viya namespace but is less likely to care as much about non-Viya activity and resources.  They will both be using something like Grafana, but the Viya admin will probably be given access to see only the resources and namespaces for Viya.

I think we can expect cases where the Kubernetes admin notices 'unusual' resource usage patterns emanating from the Viya namespace and subsequent conversations in which the Viya admin explains what CAS_DISK_CACHE  does or how Viya users can easily cause CPU or memory spikes when running high-end analytics.

The Viya admin may also see unexpected situations after which they may need to work with the Kubernetes admin to better understand the overall workload on the cluster, not just the VIya workload.

Together, the two admins help each other understand the nuances of Viya or how Viya behaves as a citizen of the larger cluster community.  

 

Logging

Logging in Viya 4 will likely continue to be primarily the domain of the Viya admin but there will still be a need to work with the Kubernetes admin to view log messages. Log messages from Viya pods are streamed to stdout where they are collected, aggregated, and persisted by Elastic Search and eventually viewed with an application like Kibana.

Viya admins who are used to combing physical Viya logs will now either have to use Kibana or go to the Kubernetes admin every time logs need examining.  I don't believe it will take long before the Kubernetes admin simply grants access to Kibana to the Viya admin with access to see log messages emitted from the Viya namespace.  

 

Starting, Stopping, or Restarting Services

This is an interesting one.  Before Viya 4, Viya admins could issue scripts or commands to start, stop, or restart Viya services.  In the Kubernetes world, starting, stopping, or restarting Viya services is primarily done via the kubectl command to manipulate pods and which requires elevated privileges.

For example, if a change was made requiring a restart of the identities service to surface, a command similar to this would delete the current identities service pod but cause Kubernetes to restart a new one to satisfy the manifest requirements that there be at least one identities service pod:

kubectl --namespace viyadev delete pod identities-5d66fcf9fb-tbv6v

Typically, the Kubernetes admin would be responsible for running such a command due to its power.  At first, one can imagine the Viya admin having to ask the Kubernetes admin to restart identities for them.  Maybe it happens twice, or three times...a day.  At some point, the Kubernetes admin will decide that they do not want go through this every day.  In that case, the Kubernetes admin will probably delegate certain abilities to the Viya admin but limiting their scope to just one or more Viya namespaces.  At this point, the Viya admin can be more self-sufficient but also be prevented from accidentally running commands outside of the Viya world.  

I could go on with other examples but I think you get the point.  There is not going to be a set-in-stone list of tasks that a Viya admin always does and another list for the Kubernetes admin to do.  The admins will likely work together and need to help each other to keep users happy.  

 

Who should do what?

SAS is certainly aware of the need to help Kubernetes admins learn about Viya and to help Viya admins learn about Kubernetes.  Viya 4 documentation splits these two worlds into our best guess as to who will do what. The SAS Viya Operations documentation is for the areas Kubernetes admins will likely oversee such as deployment and maintenance.

The more traditional Viya admin tasks are covered in SAS Viya Administration documentation which helps with authorization, data, promotion, and other areas.

Who does what will largely depend on the skills of the two admins.  It may take a Viya admin some time to prove themselves worthy of the trust of the Kubernetes admin before being granted more access to tools outside of Viya itself which means the Kubernetes admin will bear a larger administrative burden, at least at first.  Over time, as the Viya admin gains Kubernetes skills and as the Kubernetes admin builds a better understanding of Viya and a trust of the Viya admin, they will begin to build independence.  Until then, keeping users happy will require a certain amount of interdependence between the two admins.