Automating model delivery with SAS Viya on the Red Hat OpenShift Container Platform
Introduction
The Red Hat OpenShift Container Platform (OCP) is a popular choice for many SAS customers running SAS Viya in a private cloud infrastructure. OpenShift not only offers a hardened, enterprise-ready Kubernetes distribution but also comes with a rich ecosystem of 3rd-party applications and extensions to the core platform.
In this blog I’d like to discuss how tools taken from this ecosystem can be used for automating the delivery process for analytical models developed with SAS Viya. Unlike other blogs, which have focused on model training and model governance aspects, this blog discusses the steps needed to make analytical models available to applications (or users) for consumption. Typically, “model serving” topics rather belong to the domain of DevOps engineers, not data scientists.
Why at all should you bother about automating the lifecycle management of your models? It has become clear over the past years that decisions, driven by analytical models, have become more and more relevant to everyday business processes. We’ve not only seen a heavy increase in the sheer number of analytical models, but also in the frequency in which these models need to be revised and updated. Automating these processes is key to keep the pace without losing stability and robustness.
This blog describes an example setup for model automation on the OCP platform, using the capabilities of SAS Viya in conjunction with tools taken from the OCP ecosystem: OpenShift Pipelines (Tekton) and OpenShift GitOps (ArgoCD). When used in combination, these apps form the CI/CD (“Continuous Integration / Continuous Delivery”) toolchain which typically powers all DevOps systems.
This diagram shows the major steps which will be discussed in this blog:
The blog focuses on the steps shown in the center and on the right-hand side of the picture – the preparation and deployment of analytical models trained on the SAS Viya platform. The process steps shown above are actually “distribution-agnostic” - however this blog is “distribution-aware”. You could replace a tool in the CI/CD toolchain with another (e.g. Tekton with Jenkins) without changing the process flow, but there are distribution-specific specialties to be considered when choosing OpenShift as your model delivery platform.
Infrastructure overview
Figure 2 should help to get a better understanding of the infrastructure. Like most “*opsy” architectures, the infrastructure we must set up for ModelOps automation might look confusing at the beginning, but it will soon start making sense hopefully.
The typical automation flow is shown on the horizontal axis from training (left) to serving models (right), while on a vertical axis the picture shows the key “systems” starting from users and apps (i.e. producers and consumers), followed by the Kubernetes infrastructure and the external repositories providing the persistence layer required for connecting the automation tools.
Next, we have the CI/CD applications. As stated before, we’re using built-in services which are part of OpenShift’s DevOps ecosystem. SAS Model Manager does not directly interact with these tools. While its’ model repository is used for storing model metadata and tracking model performance over time, Model Manager can push the model to a Git server and to a Container registry (at the same time). This is an important capability required for supporting an automation workflow as this push event can trigger activity in the CI/CD applications. The Git repository keeps the model source code under revision control while the Container Registry stores the binary execution runtime (SCR container image).
|
SAS Container Runtime (SCR) For this blog we’ll be using the SAS Container Runtime (SCR) framework to create the model container images. SCR is a native capability available on the SAS Viya platform and part of SAS Model Manager. It’s used when publishing an analytical model (written in SAS or in Python) to a container registry destination: a lightweight Open Container Initiative (OCI) compliant container is created on the fly which is the execution environment for your model. Each SCR image contains a single model or decision. SCR images are self-contained (no dependencies on SAS Viya platform services) and highly scalable. |
In our case, OpenShift Pipelines acts on the push event to the Git repository and executes a workflow labeled as the “model serving pipeline”. It analyzes the model metadata pushed to Git and creates a deployment manifest for the model which it then commits back to the same Git repository. Why is this step important? It decouples the training layer from the serving layer: SAS Model Manager does not need to know the details of the execution infrastructure of the model. A DevOps engineer can easily modify the pipeline without interfering with the data scientist’s work.
Once the model’s deployment manifest has been pushed back to Git, it is picked up by OpenShift GitOps. This tool then takes responsibility for syncing the new content to the target namespace(s) – simply put it deploys the new model as a Kubernetes pod and adds the required Service and Route to make it accessible.
Once a model has been deployed to the model serving layer, it can finally be used by consumers (apps) sending requests to the deployed models at runtime. For real-time serving scenarios it’s a common practice that model containers (such as SAS SCR container images) offer a REST API to external apps.
Infrastructure setup
SAS Viya
Deploying SAS Viya on OCP requires a few steps which are not needed for other Kubernetes distributions, mostly to take account for OpenShift’s additional security layers and its’ use of the HAProxy ingress controller. Check the sections about OpenShift in the SAS Viya Operations Guide for more details.
For our scenario you need to make sure to include a specific transformer patch to the deployment, which is required for enabling the model publishing destination we need. See the instructions found in $deploy/sas-bases/examples/sas-model-publish/kaniko/README.md for more information and pay attention to the additional security requirement. SAS uses the kaniko tool to create the SCR container images on the fly.
|
Kaniko Kaniko is an open-source project originally developed by Google. It’s a tool to build container images from a Dockerfile and runs inside a Kubernetes cluster. SAS has packaged it as a microservice and deploys kaniko as part of the SAS Viya software stack. |
kaniko cannot run on OpenShift using the default restricted Security Context Constraint (SCC) but needs elevated permissions (check this Git issue for some additional background). For that reason, you must bind the anyuid SCC to the sas-model-publish-kaniko service account:
$ oc -n <name-of-namespace> adm policy add-scc-to-user anyuid -z sas-model-publish-kaniko
OpenShift Pipelines
The Pipelines app can easily be installed via OpenShift’s OperatorHub using the default settings. I’d also recommend installing the Tekton CLI to manage the pipelines. The main building blocks of Tekton pipelines are Tasks and the Tekton project offers a repository of pre-defined Tasks to choose from. We need two of them for the pipeline we’re about to deploy:
$ kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/git-clone/0.9/git-clone.yaml
$ kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/git-cli/0.4/git-cli.yaml
OpenShift GitOps
Like before, the OperatorHub provides the easiest way to deploy the GitOps tool. You need to grant cluster-admin permissions to the openshift-gitops-argocd-application-controller service account to allow it to run deployments on your behalf:
$ oc -n openshift-gitops adm policy add-cluster-role-to-user cluster-admin \
-z openshift-gitops-argocd-application-controller
And again, I’d recommend that you install the ArgoCD CLI for managing the environment.
Git server
Prepare a Git repository
You should prepare an (empty) Git repository which can be used for storing your model assets. I called mine sas-model-repository, but any name will do. There are 2 approaches to choose from:
- have one Git repository per model (1:1)
- have one Git repository for all models (1:n)
For this blog I chose the 2nd approach, but this is certainly not a requirement. When creating the Git repository, make sure to add a folder to keep the generated deployment manifests. I called it deploy-manifest in my system. Here’s a screenshot showing how this repository could look like (note that it already contains two models):
Figure 3
Create a Git webhook
You need to define a webhook connecting the Git repository with the CI pipeline. A simple setup like this would be sufficient (all push events, all branches). The webhook URL needs to match the address used for the HAProxy route when defining the pipeline’s event listener (see below).
Figure 4
OpenShift cluster
Automation account authentication
If you’re planning to use model validation tests in SAS Model Manager (not covered in this blog), you must ensure that the automation account used by SAS can deploy pods to the validation namespace. SAS authenticates to OpenShift using a client certificate which needs appropriate permissions for the target namespace.
Allow image pulls from your registry
Depending on your system, you might need to import your container registry’s CA certificates to OpenShift to allow ImagePulls. For example, the following commands add my registry’s CA certificates to OpenShift:
$ oc -n openshift-config create configmap registry-ca-certs \
--from-file=harbor-registry.myspace.com=/tmp/certs/harborCA.crt
$ oc patch image.config.openshift.io/cluster \
--patch '{"spec":{"additionalTrustedCA":{"name":"registry-ca-certs"}}}' \
--type=merge
Connecting the bits and pieces together
With the infrastructure in place, it’s time to discuss the configuration needed to set the process in motion. Let’s start with defining the model publish destination in SAS Model Manager and then proceed to the model serving pipeline (Tekton) and the ArgoCD sync app.
Configuring the model publish destination
A publishing destination is a concept used by SAS Model Manager. It provides a modular way of defining target environments into which models can be pushed to, for example Azure Machine Learning, Apache Hadoop, Teradata, Git and many more.
In our case, the main publishing destination will be a Container Registry, but we’ll also use a Git repository which is hooked into the CI/CD pipeline. When publishing a model to a Container Registry, SAS Model Manager will automatically create a container image for the model using the kaniko tool. Publishing destinations can be created by using the SAS Viya CLI. The command to create a Container Registry destination is rather complex, so let’s break it down to its’ individual sections:
Let me explain the “weird” lines at the end of the command first – this is known in Linux as a “here” document. It’s a useful trick to run a command in batch-mode which expects interactive input. The sas-viya CLI plugin for creating publishing destinations automatically creates a credentials domain and will prompt you for the name. The “here” document provides that name (“ContainerRegistryPublishingDomain”) and thus saves you from typing it in.
Model validation
You might wonder about the group of settings related to Kubernetes in the command shown above. This refers to a Kubernetes namespace which can be used for running a validation test on your model. This check can be triggered from the Model Manager user interface once the model container image has been published to the container registry. It’s an optional feature and if preferred it’s certainly possible to set up validation checks as part of the automation flow instead.
Setting up the model serving pipeline
Let’s take a look at the Tekton pipeline. Tekton is fully integrated into Kubernetes and extends the Kubernetes API with components needed for defining pipelines. I’ve already mentioned the use of Tasks, but there is a lot more. Again, take a look at the code snippets I have provided in the attached ZIP file to see how a sample pipeline could look like. In my simplified setup, these Tasks will perform the following steps:
- Clone the contents from the Git model repository to an internal workspace
- Create deployment manifests for all models found in the Git repository
- Commit the new manifests back to the Git repository
Once the pipeline is set up, you need to define an EventListener so that it can be triggered by a Git push event. Finally, the listener needs to be exposed using a Service and a Route (remember to make the Git webhook point to the Route URL).
You can either use the Tekton CLI or the OpenShift web interface to review your pipeline:
Figure 6
Setting up the ArgoCD syncing app
Compared to the Tekton pipeline, the ArgoCD app is actually a lot simpler to create. The following code snippets use the ArgoCD CLI for this task:
# switch to the right namespace
$ oc project openshift-gitops
# Add repository credentials to ArgoCD (for Git login credentials)
$ argocd repo add https://my-gitserver.server.com/my-model-repository.git \
--username <user1> --password <password1>
# Create application
$ argocd app create sas-modelops --insecure \
--repo https://my-gitserver.server.com/my-model-repository.git \
--dest-namespace sas-modelops-deployments \
--dest-server https://kubernetes.default.svc \
--path deploy-manifest --sync-policy automated \
--revision main --sync-option Prune=true
Note that the ArgoCD app is exclusively monitoring the “deploy-manifest” folder in the Git repository – we only want the app to wake up after the Tekton model serving pipeline has committed new deployment manifests to this folder. Here’s a screenshot of the sync app taken from the OpenShift web console:
Figure 7
Testing the configuration
Now that the environment has been fully configured, it’s time to do some testing. I would suggest to take these steps:
- Start in Model Studio by creating a simple pipeline project. Once the pipeline has completed, click on “Register” to transfer the model to Model Manager.
- In Model Manager, highlight the new model, click on “Publish” and select the Container Registry destination.
- Now move over to the Pipelines user interface in the OpenShift web console - if you’re quick enough you should be able to see the pipeline executing.
- Finally, when switching to the ArgoCD user interface you should see that the sync app starts running right after the pipeline has finished.
- Send data to the model to test that it’s working. You could use PostMan for this or simply run the curl utility:
$ curl --location --request POST 'https://mymodel.apps.openshift.cluster.com/model' \
--header 'Content-Type: application/json' \
--data-raw ' {
"CLAGE" : 100,
"CLNO" : 20,
"DEBTINC": 20,
"REASON" : "REFINANCE",
...
"VALUE" : 1000000,
"YOJ" : 10
}' | jq .
Conclusion
In this blog I discussed how OpenShift can be leveraged as a model serving platform for analytical models created by SAS Viya. Using the CI/CD tools provided by the OpenShift ecosystem - OpenShift GitOps and OpenShift Pipelines - in conjunction with SAS Viya provide a robust way to automate the lifecycle management of models.
I hope you enjoyed reading this blog and please feel free to use the comments section below if you have any feedback or questions.
