SAS administration command-line interface in a container: Part 2 Kubernetes and Argo Workflow
In my previous post, we built a container image that included the sas-admin cli and used it to perform administration tasks. In this post we will take the container built in the previous post and leverage the Kubernetes cluster to automate some more complex batch processing. Hopefully you will see that this is where the power of this approach really shows up.
There are a few ways we can run the container in Kubernetes. Using native functionality we can use either a job or a cronjob. A Kubernetes job is a little different than the other objects in a cluster. Usually Kubernetes attempts to keep things like deployments and statefulsets running. Kubernetes jobs execute one or more pods and when the pods are completed the job terminates. Kuberenetes cronjobs are very similar to jobs except they have a scheduling component so that they can run repeatedly on a schedule.
In the first example we will create a Kuberenetes cronjob that will run nightly. The job will use the containerized sas-admin cli to export a Viya folder, its sub-folders and any content to a Viya package (json file). The job is scheduled nightly and can act as a partial backup of the Viya environment or it could be used to synch content with another Viya environment (dev-test-prod).
Below is the Kubernetes manifest that will create the cronjob.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
The short video below shows the process of building the manifests and running the kubernetes cronjob. The process will build the manifest for the cronjob using Kustomize, the kustomization.yaml file reads in the cronjob definition and the namespace defintions as resources, it also generates two configmaps for the the sas-admin cli profile and authentication token, these configmaps are mounted into the pod the is created by the cronjob. Take a look at the video, just FYI it has captions but no sound.
The Kuberenetes cronjob is great for a simple task like this, however it is not very easy to do more complex processing where you may have dependencies or tasks running in parallel (possible but not easy). That is where tools like Argo Workflow can help. To quote "Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes."
Argo Workflows are implemented natively in Kubernetes. Among other things they:
- Define workflows where each step in the workflow is a container.
- Model multi-step workflows as a sequence of tasks or capture the dependencies between tasks using a directed acyclic graph (DAG).
- Easily run compute intensive jobs for machine learning or data processing.
Looks like a great fit for Viya 4.
In this next video (again captions but no sound) we will take advantage of Argo Workflow and our sas-admin cli container to create and run a flow that will automate the initial administration steps to prepare a new Viya 4 deployment for users. Assume you have spun up a new Viya 4 deployment and you want to create a standardized environment for users. Our flow will leverage Argo Workflow and the sas-admin container image to:
- Setup identities : 01-setup-identities.sh
- Create a preliminary folder structure : 02-create-folders.sh
- Apply an authorization schema to the folder structure: 03-setup-authorization
- Create caslibs for data access: 04-setup-caslibs.sh
- Load Data: 05-setup-loaddata.sh
- Apply CAS authorization: 06-setup-casauth.sh
- Load some test content: 07-load-content.sh
The yaml for the Argo workflow is similar to that of the Kubernetes cronjob. The additional functionality provided allows you to set dependencies between jobs and/or run jobs in parallel. Take a look at this video to see how easy it is to create an Argo workflow to initialize a Viya environment for users.
I am sure you can think of many use cases for this functionality. I could envisage automated migrations from SAS Viya 3.5 or from SAS Viya 4 to another SAS Viya 4, complex ETL or model processing and many more. I hope you found these two articles useful. Here is some additional reading.
