Using the SAS Container Runtime for publishing SAS models to Kubernetes on the Azure cloud

4 Likes

With the 2021.1.3 STABLE release of SAS Viya 2 new publishing destinations have been added to SAS Model Manager: you now have the ability to publish SAS models to Azure’s Machine Learning service (AML) as well as to a new container destination called the SAS Container Runtime (SCR). In this blog we will not discuss the AML integration but focus on the configuration steps required to set up the SAS Container Runtime for Viya environments deployed in the Azure cloud. Want to start with an overview first before diving knee-deep into technology? - then this blog is what you need. If you’re interested in the AML integration specifically, you should take a look at this blog describing the new feature in more detail. We also highly recommend that you spend a few minutes reading this blog which gives you a great overview of all available publishing destinations (including the new ones).

To cite the official documentation: the SAS Container Runtime is a “lightweight OCI (Open Container Initiative) compliant container that scores SAS models. This enables you to run models on any OCI compliant compute system, including clusters that are based on Docker and Kubernetes.” (SAS Container Runtime: Programming and Administration Guide).

Simply put: SCR allows you to package your SAS model into a container image which has been specifically designed for executing SAS analytical models (as of now – a future release will bring the support for models written in Python and R as well). Each SCR container image hosts a single SAS model. The image does not have any dependencies to the outside world so it will work right away, no matter where you deploy it. External clients send requests to the SCR container using REST calls by communicating with the built-in REST server in each container. The container approach comes with a lot of benefits: there’s no dependency on the runtime environment (on premise, cloud …), it’s stateless (so it can be deployed using multiple replicas to address scalability and availability requirements), it’s easy to update without downtime (rolling updates) and much more.

This blog will guide you through the configuration steps needed for setting up the SCR destination for the Azure cloud. Apart from publishing to a container registry, SAS Model Manager also supports running a publishing validation check, which allows you to test if publishing your model actually worked. In our case, a Kubernetes cluster is used for running the validation and throughout this blog, we’ll use the terms “Viya cluster” and “validation cluster” rather often. Keep in mind that this is a conceptual thing – nothing is keeping you from using a single Kubernetes cluster on which you deploy SAS Viya and validate your models at the same time (e.g. for testing purposes). However, even then you still need to follow all of the configuration steps written down below.

The conceptual architecture looks like this:

For the rest of this blog, we’re assuming that you have already created the AKS cluster(s) and an instance of the Azure container registry (ACR). We also assume that you can use the Azure CLI for submitting commands to Azure. We’re not going to cover the details of a SAS Viya deployment (this is documented here), but make sure to read the next section before you start deploying Viya.

Here’s a brief overview on the configuration steps we are going to cover:

We need to establish a service principal in both the container registry and the validation cluster with the appropriate role assignments to allow the principal to push/pull images to/from the container registry and to deploy the container image on the validation cluster
And of course, we also need to register the ACR registry and the validation cluster as a new publishing destination in SAS Model Manager

Apply the kaniko storage patch to your Viya deployment

While we said that we would not cover the deployment of SAS Viya, we’re including this section to remind you that there is one important kustomize patch which you should not forget if you want to publish a SAS model as a container image. This patch (and README) can be found in

$deploy/sas-bases/examples/sas-model-publish/kaniko/

Where $deploy points to the directory where you unpacked the Viya deployment assets.

This patch assigns storage to the model publishing microservice which is needed for the kaniko tool. In case you have not heard about kaniko before: it’s an open-source tool “to build container images from a Dockerfile, inside a container or Kubernetes cluster” (copied from the kaniko homepage). The model publishing microservice uses kaniko to build the SCR image on-the-fly when you publish a model to the SCR destination from Model Manager.

Applying the patch should be pretty straight-forward:

cd $deploy
mkdir -p site-config/sas-model-publish/kaniko

cp sas-bases/examples/sas-model-publish/kaniko/*.yaml site-config/sas-model-publish/kaniko
chmod 644 site-config/sas-model-publish/kaniko/*

# set capacity of RWX storage
sed -i "s|{{ STORAGE-CAPACITY }}|20Gi|" site-config/sas-model-publish/kaniko/storage.yaml

# set storage class providing the RWX storage (NFS, Azure Files ...)
MY_RWX_SC=myrwx
sed -i "s|{{ STORAGE-CLASS-NAME }}|$MY_RWX_SC|" site-config/sas-model-publish/kaniko/storage.yaml

Don’t forget to include the patch to your kustomization.yaml:

resources:
...
- site-config/sas-model-publish/kaniko

transformers:
...
- sas-bases/overlays/sas-model-publish/kaniko/kaniko-transformer.yaml

Assigning a service principal to the Azure container registry and the validation AKS cluster

First of all create a new service principal and keep it’s ID and password:

# this command will return the tenant and the subscription ID
az account list -o table \
   --query "[].{name: name, tenantid: tenantId, subscriptionid: id}"

SUBSCR_ID=/subscriptions/<your-subscription-id>

# create the service principal
az ad sp create-for-rbac --scopes $SUBSCR_ID --skip-assignment true -n azure-scr-sp

# store AppID
SP_APPID=$(az ad sp list --display-name azure-scr-sp --query "[].appId" -o tsv)

Important: Be sure to also write down the password generated by this step. You will need it later when creating the publishing destination in SAS.

Before we can create the role assignments we need to retrieve the IDs of the container registry and the validation AKS cluster. This could be the AKS where you deployed Viya, but it could also be a different cluster (e.g. a “production cluster” while Viya has been deployed on a “development cluster”).

# replace these with meaningful values
RESOURCEGRP=myresgrp
VALIDATE_AKS_CLUSTER=myaks
ACR_NAME=myacr

# get the ACR ID
ACR_REGISTRY_ID=$(az acr show --name $ACR_NAME --query id --output tsv)

# get the AKS ID
AKS_ID=$(az aks show --resource-group $RESOURCEGRP \
   --name $VALIDATE_AKS_CLUSTER --query id -o tsv)

# assign ACR Push and ACR Pull roles to the service principal
# (acrpush includes acrpull)
az role assignment create --assignee $SP_APPID \
   --scope $ACR_REGISTRY_ID --role acrpush

# assign the cluster admin role to the service principal
az role assignment create --assignee $SP_APPID --scope $AKS_ID \
   --role "Azure Kubernetes Service Cluster Admin Role"

Opening the validation cluster’s firewall for accessing NodePort services

If you ask Model Manager to run a publishing validation on your new model after you’ve pushed it to the registry, it will deploy a new SCR instance on the target cluster for the duration of the validation test. Once the new SCR instance is running, Model Manager needs to be able to connect to it in order to send scoring requests to it.

From a technical point of view the Model Manager pod issues REST calls to the SCR pod. By default the SCR pod is deployed to the validation AKS using the NodePort service type, which is a mechanism provided by Kubernetes to make pods accessible to external clients. Pods exposed via NodePort can be reached from any Kubernetes node using the same port address (so a client does not need to know on which particular node a pod has been deployed as long as it knows on which port it has been exposed). NodePort uses a port range of 30000-32767, so we have to allow inbound traffic for this range. Luckily we do not need to open this range to just anyone: because we know that Model Manager will make the request, we can restrict incoming connections from the public IP address of the AKS cluster where Viya is deployed to the public IP address of the Load Balancer serving the cluster where the validation is supposed to happen. This configuration step is required even if you are just using a single cluster.

You can either make these changes manually or by using a few shell commands. Either way, this is how it should look like after applying the changes (this picture shows the network security group used by the AKS nodes):

You can use the following snippet to create the inbound security rule programmatically. Careful: this snippet assumes that there is only one AKS cluster, so you need to make changes to this if you have separate ones.

RESOURCEGRP=myresgrp

# find the internal AKS resource group
AKS_RG=$(az aks list -g $RESOURCEGRP --query [].nodeResourceGroup -o tsv)
echo "AKS resource group: $AKS_RG"

# find the LB IP address of the AKS cluster (outbound)
OUTBOUND_IP=$(az network public-ip list --query "[].{tags: tags.type, address: ipAddress}" -o tsv -g $AKS_RG | grep aks-slb | cut -f2)
echo "AKS outbound IP: $OUTBOUND_IP"

# find the IP address of the AKS cluster (inbound)
INBOUND_IP=$(az network public-ip list --query "[].{tags: tags.type, address: ipAddress}" -o tsv -g $AKS_RG | grep None | cut -f2)
echo "AKS inbound IP: $INBOUND_IP"

AKS_NSG=$(az network nsg list -g $AKS_RG --query [].name -o tsv)
echo "AKS NSG: $AKS_NSG"

az network nsg rule create -g $AKS_RG --nsg-name $AKS_NSG -n AllowNodePortRange \
   --priority 100 \
   --source-address-prefixes $OUTBOUND_IP/32 \
   --source-port-ranges '*' \
   --destination-address-prefixes $INBOUND_IP \
   --destination-port-ranges '30000-32767' --access Allow \
   --protocol Tcp --description "Allow access to pods via nodeport"

Register your Azure Container Registry instance as a publishing destination in Model Manager

With the Azure related preparation steps in place, it’s now time to turn to the SAS configuration. Before we can publish SAS models to Azure (that is: to the instance of the Azure Container Registry (ACR) which you want to use), we first need to define the ACR instance as a publishing destination in Model Manager.

Creating a publishing destination actually consists of two steps as you need to create a Credentials Domain (also referred to as a Base64 Domain) and the Publishing Destination. Luckily both steps can be completed by running just a single command using the SAS Viya CLI. If you have not worked with the SAS Viya CLI before, this would be a good chance to make yourself familiar with it as it greatly simplifies many administration tasks in SAS Viya.

Assuming that you have downloaded the CLI, installed the plugins, created a profile and logged in, creating a new publishing destination is done using this command:

sas-viya models destination createAzure --name Azure \ 
  --baseRepoURL youracregistry.azurecr.io \             # the ACR login server URL
  --tenantId a1b1c1d1-... \                             # Azure tenant ID
  --subscriptionId 1a3b5c7d-.... \                      # Azure subscription ID
  --resourceGroupName yourviyarg \                      # Azure resource group
  --kubernetesCluster yourakscluster \                  # name of AKS cluster
  --region yourregion \                                 # e.g. eastus
  --validationNamespace validation-ns                   # namespace for validation

There is one additional parameter which has not been set in this example: “credDomainID”. The context help of the CLI explains this parameter like this:

--credDomainID, -c          Specifies the credential domain ID for the publishing destination.
                            If this option is not specified, you are prompted to create a domain.

So, if you do not specify this parameter you will have to answer a few prompts which are needed to create the Credentials Domain for you. However, if you already have created the domain before (or if you want to re-use the domain for another publishing destination), you can simply include this parameter and refer to the existing domain by specifying the domain ID.

If you want to learn more about this powerful CLI plugin, head over to this Git repository which not only offers a lot of information related to SAS Model Manager but also includes examples of how to create publishing destinations using the CLI.

Wrapping up

That’s it for the configuration steps and you’re now ready for a test run. Why don’t you download some sample models from here, register the SAS models with Model Manager and then test the new publishing destination? We’ve recorded a short video clip showing the new SCR publishing destination “in action”.

(view in My Videos)

Thanks for following us through this blog. We hope you found it helpful and don’t hesitate to reach out in case you have any question!