How to Publish a SAS Model to Azure with SCR: A Start-to-Finish Guide

4 Likes

SAS Container Runtime (SCR) allows you to publish a SAS model as a container image in Azure. SAS Container Runtime (SCR, pronounced "soccer") and Azure as a publishing destination is available as of since the SAS Viya version 2021.1.3. The post details all the configuration steps in Azure and SAS to publish a SAS model as an Azure container image and validate the publishing.

The Big Picture

Publishing

(view in My Videos)

Publishing Validation

(view in My Videos)

Why the SAS Container Runtime?

SAS Container Runtime (SCR) benefits:

Self-contained: once the image is published, it includes all the libraries and files that are required to score the decision. It has no dependencies on SAS Viya anymore for execution.
Minimal disk footprint: around 370 MB for a bare SCR image. The size is increased by the decision files. (The whole container image size used in this post is 391 MB.)
Fast start-up: once the image pulled, the container starts in 12-15 seconds.
Immutable design: the container image is designed for execution. If you must change something in your decision, you need to publish a new decision version. That means a new container image.
Flexible deployment options: you can deploy it to any compute environment that supports a docker pull command. In Azure, you can run your SCR image in an Azure Container Instance (ACI), Azure Web App or App Service Plan, Azure Kubernetes Service (AKS) pod. Deployment is not limited to Azure. You can start your container on a Linux machine with docker. Push the image to an Amazon Elastic Container Registry, deploy it on AWS, etc.
High scalability: because each decision exists in its own container image, each running container can be scaled independently. I will focus on a later post on how to scale SCR containers in Azure.
Has a much smaller runtime compute cost. Execution not being tied to hundreds of running microservices has its advantages. The immediate one is lower running cost / scoring request.

SAS Container Image Publishing and Validation

10 SAS Viya on Azure Architecture - Model Publishing and Validation with SCR.png

The red-blue floating cube is the SCR container image published from SAS.
The blue arrow represents publishing to Azure Container Registry.
The red arrow highlights publishing validation.

Graphical background, SAS Viya on Azure source: SAS Research & Development.

Steps to Publish a Model to Azure

SAS Viya Deployment
Azure Components
SAS Viya Components
Publish a model to Azure
Published model validation

Remarks:

Firstly, I am publishing to an Azure Container Registry outside of SAS Viya's deployment resource group.
Secondly, the type of models you can publish today are limited to SAS code (Data Step). Read more about it in SCR documentation.
Thirdly, I will keep it simple and reuse the same Azure Kubernetes Service cluster, where SAS Viya is running. It is a simple, yet realistic scenario. The whole point of validating the SCR container image is to make sure it produces the expected results, before deploying the images further. Deployment of the container image from the Azure Container Registry will be covered in future posts.

SAS Viya Deployment

You will need to deploy SAS Viya stable version 2021.1.3 or later (the first when SCR was available) in Azure.

Before deployment, you must apply the Kaniko patch. If you do not perform this step, model publishing will fail. You would need these files:

kaniko-transformer.yaml  kustomization.yaml  podtemplate.yaml  storage.yaml

from your deployment folder:

# export PRJ=fill-your-path-to-deployment-folder-here 
ls $PRJ/sas-bases/examples/sas-model-publish/kaniko/

To apply the kaniko patch, you can follow the steps below:

# 1. Using the files in the `$PRJ/sas-config/sas-model-publish/kaniko` directory
echo "kaniko folder permissions"
cd ~
chmod -R 755 $PRJ/site-config/sas-model-publish/kaniko

# 2. Modify the parameters in the podtemplate.yaml file, if you need to implement customized requirements, such as the location of Kaniko image. 
# Nothing to change

# 3. Modify the parameters in storage.yaml. For more information about PersistentVolume Claims (PVCs), see [Persistent Volume Claims on Kubernetes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims).

# Azure storage: 4Gi should be enough ~ 10 SCR models at the same time
# storageClassName: sas-azurefile
# You can find out the Azure storage class by checking other PVCs e.g.  kubectl get pvc -n ${current_namespace} -o wide | { head -1; grep "cas"; }

echo "Update storage.yaml"
cd ~
cd $PRJ/site-config/sas-model-publish/kaniko/

echo "Update the Azure storage capacity '{{ STORAGE-CAPACITY }}'"
sed -i 's/{{ STORAGE-CAPACITY }}/4Gi/' ./storage.yaml

echo "Update the storage class name"
sed -i 's/{{ STORAGE-CLASS-NAME }}/sas-aks/' ./storage.yaml

echo "List changes in storage.yaml"
cat storage.yaml
cd ~

# 4. Make the following changes to the base kustomization.yaml file in the $deploy directory.

echo "Remove readme.md file"
cd ~
cd $PRJ/site-config/sas-model-publish/kaniko/
rm -Rf *.md

echo "Backup kustomization.yaml first"
cd ~
cd $PRJ
cp kustomization_template.yaml kustomization_template_before_kaniko.yaml

#   * Add site-config/sas-model-publish/kaniko to the resources block.
#   ```yaml
#   resources:
#   - site-config/sas-model-publish/kaniko

echo "Add Kaniko to resources"
cd ~
printf "
- command: update
  path: resources[+]
  value:
    site-config/sas-model-publish/kaniko # 2021.1.3 Kaniko mount patch
" | yq -I 4 w -i -s - $PRJ/kustomization_template.yaml

#   * Add sas-bases/overlays/sas-model-publish/kaniko/kaniko-transformer.yaml to the transformers block.
#   transformers:
#   - sas-bases/overlays/sas-model-publish/kaniko/kaniko-transformer.yaml

echo "Add Kaniko to transformers"
cd ~
printf "
- command: update
  path: transformers[+]
  value:
    sas-bases/overlays/sas-model-publish/kaniko/kaniko-transformer.yaml   # 2021.1.3 Kaniko transformer patch
" | yq -I 4 w -i -s - $PRJ/kustomization_template.yaml

cd ~
echo "Kaniko patch end"

Hans-Joachim Edert has published similar notes Using the SAS Container Runtime for publishing SAS models to Kubernetes on the Azure cloud.
Launch the SAS Viya deployment and validate the deployment.

Optional, after the deployment, you can check the Kaniko Mount:

# 1. Verify pvc
kubectl get pvc | grep kan 

# You should see
# sas-model-publish-kaniko  Bound    pvc-91494c0d-07c1-466e-b45a-925934a72ae2   4Gi   RWX     sas-aks        102m

# 2. Run the following command to verify whether the overlays have been applied:
kubectl get pods | grep sas-model-publish

# You should see something similar to
# sas-model-publish-5d96cd6f44-22h5m 1/1     Running       0          28m

# get the name of the pod an add it in the command below
kubectl describe pod   | grep models

# 3. Verify that the output contains the following mount directory paths:
#     /models/kaniko from kaniko (rw)

Azure Components

Publishing

From your Azure subscription, Azure Active Directory, retrieve your Tenant ID:

600 Azure Tenant.png

Get your Azure Subscription ID:

610 Azure Subscription.png

Check you have sufficient rights in your Azure subscription. As an Owner of the subscription; therefore it should be fine for the rest of the setup. At my knowledge, you have to be at least a Contributor to perform the next steps.
You will require a machine to execute Azure Command-Line Interface scripts. The easiest option is to use cloud-shell (top-right of your Azure portal):

620 Azure Cloud Shell.png

If you are connecting from a Linux machine, you will need to Install the Azure CLI on Linux.

Create an Azure Container Registry. You can create the Azure Container Registry via the Azure Portal, or by scripting it:

# Check you have the AZ CLI
az version

# Customize variables
RG=MYSCR # choose a name for your resource group
LOCATION=eastus # change it to suit your needs
ACR=sascontainers # choose a name for your Azure Container Registry

# Create an Azure resource group RG
az group create --name $RG --location $LOCATION

# Create an Azure Container Registry ACR
az acr create --resource-group $RG --name $ACR --sku Basic --location $LOCATION

630 Azure Container Registry create.png

Write down the Azure Container Registry login server: sascontainers.azurecr.io. In SAS Viya, when defining the Azure publishing destination, this is called the Base Repository URL.
If you are planning to pull images from this registry, from a machine or application, you need to authenticate with an Azure Container Registry. The easiest option is to enable access keys and an admin user for the registry.

640 Azure Container Registry access.png

Next, register an Azure Secure App. Register an application, perhaps, SCRApp and a secret to provide access to the container registry and the validation cluster.
The application ID and the secret will be used later, in SAS as a client id and client secret. A SAS token will be tied to the client id and secret (the Azure app) and linked to a SAS user or group, and a credential domain. This effectively establishes a bridge between SAS and Azure. SAS can then push the SCR image containing a model or decision to the Azure Container Registry. SAS can also create a deployment in the AKS publishing validation cluster from the image.

650 Azure Register Application.png

Register and note the Application Client ID: 21df95c2-d562-4b1d-9c1c-7562bc1f9b36

Create a new client secret. Copy the client secret before moving away from the screen, as you will not be able to recover it: 2sY7Q~...

Before sharing, it is best to check that your notebooks, scripts, documents or external git projects do not contain any confidential elements or credentials. Keep your resources and the company safe.
With the Azure app created, assign it roles to push and pull images in the Azure Container Registry. Go to your Azure Container Registry > Access control > Role assignments > Add > Search the app you registered and add the roles:
- Owner (full access to the resource). I opted for this one, as it is the only one working in my subscription.
The SAS documentation suggests using:
- AcrPull – pull images from the registry or,
- AcrPush – pull / push images from the registry (includes AcrPull). I think this is the absolute minimum for successful publishing.

680 Assign App Role Azure Container Registry.png

Publishing Validation

Assign the Azure app roles to access to access the Azure Kubernetes Service validation cluster. Select the AKS instance (I opted for the same where SAS Viya is running) and click access control. Add role assignments: Contributor.
The above method is just one way of doing it. If you are an Azure tenant administrator, you could use Hans-Joachim Edert’s approach. Create a service principal, as in Using the SAS Container Runtime for publishing SAS models to Kubernetes on the Azure cloud, then assign roles to this service principal. His approach seems to be in line with Azure Active Directory Integration referred in SAS Model Manager’s documentation.

Open the validation cluster’s firewall to access NodePort services (required even on a single cluster). Add an Inbound Security Rule in the Network Security Group linked to the AKS cluster. The rule will allow traffic from the public IP address of the AKS cluster where Viya is deployed, to the public IP address of the Load Balancer serving the cluster where the validation will happen. The required port range is 30000-32767. The script below was provided by Hans-Joachim Edbert in his post Using the SAS Container Runtime for publishing SAS models to Kubernetes on the Azure cloud:

RESOURCEGRP=sasuser-azuredm-rg # choose the resource group name where the AKS cluster is deployed

# find the internal AKS resource group
AKS_RG=$(az aks list -g $RESOURCEGRP --query [].nodeResourceGroup -o tsv)
echo "AKS resource group: $AKS_RG"

# find the LB IP address of the AKS cluster (outbound)
OUTBOUND_IP=$(az network public-ip list --query "[].{tags: tags.type, address: ipAddress}" -o tsv -g $AKS_RG | grep aks-slb | cut -f2)
echo "AKS outbound IP: $OUTBOUND_IP"

# find the IP address of the AKS cluster (inbound)
INBOUND_IP=$(az network public-ip list --query "[].{tags: tags.type, address: ipAddress}" -o tsv -g $AKS_RG | grep None | cut -f2)
echo "AKS inbound IP: $INBOUND_IP"

# Get NSG of AKS resource group
AKS_NSG=$(az network nsg list -g $AKS_RG --query [].name -o tsv)
echo "AKS NSG: $AKS_NSG"

# Create inbound nsg rule
az network nsg rule create -g $AKS_RG --nsg-name $AKS_NSG -n AllowNodePortRange \
   --priority 100 \
   --source-address-prefixes $OUTBOUND_IP/32 \
   --source-port-ranges '*' \
   --destination-address-prefixes $INBOUND_IP \
   --destination-port-ranges '30000-32767' --access Allow \
   --protocol Tcp --description "Allow access to pods via nodeport"

# Rule is created in aks-agentpool-59212252-nsg 
# from Load Balancer IP of AKS cluster (OUTBOUND IP) to IP address of the AKS Cluster (INBOUND IP).

After running the scripts check the Network Security Group:

700 Azure Kubernetes Service Network Security Group ports.png

Integrate the validation Azure Kubernetes Service cluster with the Azure Container Registry. This is required, so that the AKS cluster can pull its own images from the ACR.

# Integrate an existing ACR with existing AKS clusters by supplying valid values for acr-name or acr-resource-id as below.
RESOURCEGRP= sasuser-azuredm-rg
VALIDATE_AKS_CLUSTER=sasuser-azuredm-aks
ACR_NAME=sascontainers

az aks update -n $VALIDATE_AKS_CLUSTER -g $RESOURCEGRP --attach-acr $ACR_NAME

SAS Viya Components

Define Azure as a Publishing Destination

You cna use the sas-viya models destination createAzure CLI. You cna find several SAS official examples on GitHub:
Code example to define the Azure publishing destination:

# login with the sas-viya cli profile already created
cd ~
export SAS_CLI_PROFILE=${namespace}
export SSL_CERT_FILE=~/.certs/${namespace}_trustedcerts.pem
sas-viya -k auth login --user sasuser --password **********

# Use the CLI - invoke the help
sas-viya --profile ${SAS_CLI_PROFILE} models destination createAzure --help

# define the needed variables
NS=$GELENV_NS # namespace
INGRESS_FQDN=${PREFIX}.gelenablesyou.sas.com # SAS Viya URL
SASUSER=sasadmin@gelenablesyou.sas.com # SAS User
DOMAIN_NAME="AzureDomain" # credentials domain name
ACR_DEST_NAME="Azure" # destination name
ACR_DEST_DESC="Azure SCR"
ACR_SERVER=${PREFIXNODASH}acr.azurecr.io # the Azure Container Registry you will publish to
AKS_NAME=$PREFIX-aks # the SAS AKS cluster where you will perform publishing validation. You integrated it with the ACR.

# list variables which will be used
printf "\nThe destination will be created with these parameters \n"
export ACR_DEST_NAME_CLI=AzureCLI
echo "Name: $ACR_DEST_NAME_CLI"
echo $INGRESS_FQDN
echo $PREFIX && echo $PREFIXNODASH
echo ${PREFIXNODASH}acr
echo "baseRepoURL: $ACR_SERVER"
echo "Subscription: $SUBSCRPTN"
echo "Tenant: $TENANTID"
echo "Region: $LOCATION"
echo "Cluster Name: $PREFIX-aks"
echo "Resource group: $RG"
echo "Identity: $SASUSER"
echo "App Client ID: $APP_CLIENT_ID" # app registration client id
echo "App Client Secret: $APP_CLIENT_SECRET" # app registration client secret

sas-viya --profile ${SAS_CLI_PROFILE} models destination createAzure --name ${ACR_DEST_NAME_CLI} --description "ACR with SAS Viya CLI" --baseRepoURL ${ACR_SERVER} --subscriptionId ${SUBSCRPTN} --tenantId ${TENANTID} --region ${LOCATION} --kubernetesCluster ${PREFIX}-aks --resourceGroupName ${RG} --credDomainID "ACRCredDomainCLIRomeo" --credDescription "Azure ACR credentials CLI Romeo" --clientId ${APP_CLIENT_ID} --clientSecret ${APP_CLIENT_SECRET} --identityType user --identityId ${SASUSER}

Finally, create the destination and confirm the creation in SAS Environment Manager:

710 SAS Created Azure Publishing Destination.png

Publish a model to Azure

To publish a model, you would need a model first. Use the Quick Start to import a and test a few models in SAS Model Manager. Stop before Publish the Champion Model.
The champion and challenger model were selected in the video above.

Publishing Validation

In SAS Model Manager > Project.
Go to Scoring > Publishing Validation.
Select the model published to Azure and an input table.
Run. At the end, click on the Results.
Check the Output, Code and Log.

Conclusions

It was a long journey, but by this time you learnt how to configure Azure as a publishing validation for SAS models, how to publish a model to Azure and how to validate the publishing.

Now that your models have been published as container images in Azure, you could choose to stop the SAS Viya deployment in Azure, to save cloud costs. To score the published SAS models, you no longer require all the SAS resources and the SAS Viya Azure Kubernetes cluster up and running. You can deploy your container in a number of ways and score the model. Read more about it in future posts.

SAS Container Runtime is certainly an important step allowing us to reimagine how we can work with SAS Viya in the cloud. Models can now be shipped outside SAS Viya as "containerized intelligence". To paraphrase Neil Armstrong, SCR is one small step for SAS, one giant leap for the SAS customers.

Acknowledgements

This post builds on very good work done previously by:

Deva Kumar, who wrote very clear notes about how to exploit the container images once in Azure.
Hans-Joachim Edert wrote this excellent post. He made me realize that publishing a model to Azure and publishing validation are somehow independent concepts or operations.
Glenn Clingroth described very well the Azure configuration.
The GEL Team for many aspects related to deployment in Azure and SAS Viya configuration.

Want to Know More?

Thank you for your time reading this post. If you found it useful, like it. Please comment and tell us what you think about the new SCR and Azure publishing destination.