How to Publish a SAS Model to Azure with SCR: A Start-to-Finish Guide
SAS Container Runtime (SCR) allows you to publish a SAS model as a container image in Azure. SAS Container Runtime (SCR, pronounced "soccer") and Azure as a publishing destination is available as of since the SAS Viya version 2021.1.3. The post details all the configuration steps in Azure and SAS to publish a SAS model as an Azure container image and validate the publishing.
The Big Picture
Publishing
Publishing Validation
Why the SAS Container Runtime?
SAS Container Runtime (SCR) benefits:
- Self-contained: once the image is published, it includes all the libraries and files that are required to score the decision. It has no dependencies on SAS Viya anymore for execution.
- Minimal disk footprint: around 370 MB for a bare SCR image. The size is increased by the decision files. (The whole container image size used in this post is 391 MB.)
- Fast start-up: once the image pulled, the container starts in 12-15 seconds.
- Immutable design: the container image is designed for execution. If you must change something in your decision, you need to publish a new decision version. That means a new container image.
- Flexible deployment options: you can deploy it to any compute environment that supports a docker pull command. In Azure, you can run your SCR image in an Azure Container Instance (ACI), Azure Web App or App Service Plan, Azure Kubernetes Service (AKS) pod. Deployment is not limited to Azure. You can start your container on a Linux machine with docker. Push the image to an Amazon Elastic Container Registry, deploy it on AWS, etc.
- High scalability: because each decision exists in its own container image, each running container can be scaled independently. I will focus on a later post on how to scale SCR containers in Azure.
- Has a much smaller runtime compute cost. Execution not being tied to hundreds of running microservices has its advantages. The immediate one is lower running cost / scoring request.
SAS Container Image Publishing and Validation
- The red-blue floating cube is the SCR container image published from SAS.
- The blue arrow represents publishing to Azure Container Registry.
- The red arrow highlights publishing validation.
Graphical background, SAS Viya on Azure source: SAS Research & Development.
Steps to Publish a Model to Azure
- SAS Viya Deployment
- Azure Components
- SAS Viya Components
- Publish a model to Azure
- Published model validation
Remarks:
- Firstly, I am publishing to an Azure Container Registry outside of SAS Viya's deployment resource group.
- Secondly, the type of models you can publish today are limited to SAS code (Data Step). Read more about it in SCR documentation.
- Thirdly, I will keep it simple and reuse the same Azure Kubernetes Service cluster, where SAS Viya is running. It is a simple, yet realistic scenario. The whole point of validating the SCR container image is to make sure it produces the expected results, before deploying the images further. Deployment of the container image from the Azure Container Registry will be covered in future posts.
SAS Viya Deployment
- You will need to deploy SAS Viya stable version 2021.1.3 or later (the first when SCR was available) in Azure.
- Before deployment, you must apply the Kaniko patch. If you do not perform this step, model publishing will fail. You would need these files:
kaniko-transformer.yaml kustomization.yaml podtemplate.yaml storage.yaml
from your deployment folder:# export PRJ=fill-your-path-to-deployment-folder-here ls $PRJ/sas-bases/examples/sas-model-publish/kaniko/
- To apply the kaniko patch, you can follow the steps below:
# 1. Using the files in the `$PRJ/sas-config/sas-model-publish/kaniko` directory echo "kaniko folder permissions" cd ~ chmod -R 755 $PRJ/site-config/sas-model-publish/kaniko # 2. Modify the parameters in the podtemplate.yaml file, if you need to implement customized requirements, such as the location of Kaniko image. # Nothing to change # 3. Modify the parameters in storage.yaml. For more information about PersistentVolume Claims (PVCs), see [Persistent Volume Claims on Kubernetes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims). # Azure storage: 4Gi should be enough ~ 10 SCR models at the same time # storageClassName: sas-azurefile # You can find out the Azure storage class by checking other PVCs e.g. kubectl get pvc -n ${current_namespace} -o wide | { head -1; grep "cas"; } echo "Update storage.yaml" cd ~ cd $PRJ/site-config/sas-model-publish/kaniko/ echo "Update the Azure storage capacity '{{ STORAGE-CAPACITY }}'" sed -i 's/{{ STORAGE-CAPACITY }}/4Gi/' ./storage.yaml echo "Update the storage class name" sed -i 's/{{ STORAGE-CLASS-NAME }}/sas-aks/' ./storage.yaml echo "List changes in storage.yaml" cat storage.yaml cd ~ # 4. Make the following changes to the base kustomization.yaml file in the $deploy directory. echo "Remove readme.md file" cd ~ cd $PRJ/site-config/sas-model-publish/kaniko/ rm -Rf *.md echo "Backup kustomization.yaml first" cd ~ cd $PRJ cp kustomization_template.yaml kustomization_template_before_kaniko.yaml # * Add site-config/sas-model-publish/kaniko to the resources block. # ```yaml # resources: # - site-config/sas-model-publish/kaniko echo "Add Kaniko to resources" cd ~ printf " - command: update path: resources[+] value: site-config/sas-model-publish/kaniko # 2021.1.3 Kaniko mount patch " | yq -I 4 w -i -s - $PRJ/kustomization_template.yaml # * Add sas-bases/overlays/sas-model-publish/kaniko/kaniko-transformer.yaml to the transformers block. # transformers: # - sas-bases/overlays/sas-model-publish/kaniko/kaniko-transformer.yaml echo "Add Kaniko to transformers" cd ~ printf " - command: update path: transformers[+] value: sas-bases/overlays/sas-model-publish/kaniko/kaniko-transformer.yaml # 2021.1.3 Kaniko transformer patch " | yq -I 4 w -i -s - $PRJ/kustomization_template.yaml cd ~ echo "Kaniko patch end" - Hans-Joachim Edert has published similar notes Using the SAS Container Runtime for publishing SAS models to Kubernetes on the Azure cloud.
- Launch the SAS Viya deployment and validate the deployment.
- Optional, after the deployment, you can check the Kaniko Mount:
# 1. Verify pvc kubectl get pvc | grep kan # You should see # sas-model-publish-kaniko Bound pvc-91494c0d-07c1-466e-b45a-925934a72ae2 4Gi RWX sas-aks 102m # 2. Run the following command to verify whether the overlays have been applied: kubectl get pods | grep sas-model-publish # You should see something similar to # sas-model-publish-5d96cd6f44-22h5m 1/1 Running 0 28m # get the name of the pod an add it in the command below kubectl describe pod | grep models # 3. Verify that the output contains the following mount directory paths: # /models/kaniko from kaniko (rw)
Azure Components
Publishing
- From your Azure subscription, Azure Active Directory, retrieve your Tenant ID:
- Get your Azure Subscription ID:
- Check you have sufficient rights in your Azure subscription. As an Owner of the subscription; therefore it should be fine for the rest of the setup. At my knowledge, you have to be at least a Contributor to perform the next steps.
- You will require a machine to execute Azure Command-Line Interface scripts. The easiest option is to use cloud-shell (top-right of your Azure portal):
- If you are connecting from a Linux machine, you will need to Install the Azure CLI on Linux.
- Create an Azure Container Registry. You can create the Azure Container Registry via the Azure Portal, or by scripting it:
# Check you have the AZ CLI az version # Customize variables RG=MYSCR # choose a name for your resource group LOCATION=eastus # change it to suit your needs ACR=sascontainers # choose a name for your Azure Container Registry # Create an Azure resource group RG az group create --name $RG --location $LOCATION # Create an Azure Container Registry ACR az acr create --resource-group $RG --name $ACR --sku Basic --location $LOCATION
- Write down the Azure Container Registry login server: sascontainers.azurecr.io. In SAS Viya, when defining the Azure publishing destination, this is called the Base Repository URL.
- If you are planning to pull images from this registry, from a machine or application, you need to authenticate with an Azure Container Registry. The easiest option is to enable access keys and an admin user for the registry.
- Next, register an Azure Secure App. Register an application, perhaps, SCRApp and a secret to provide access to the container registry and the validation cluster.
- The application ID and the secret will be used later, in SAS as a client id and client secret. A SAS token will be tied to the client id and secret (the Azure app) and linked to a SAS user or group, and a credential domain. This effectively establishes a bridge between SAS and Azure. SAS can then push the SCR image containing a model or decision to the Azure Container Registry. SAS can also create a deployment in the AKS publishing validation cluster from the image.
- Register and note the Application Client ID: 21df95c2-d562-4b1d-9c1c-7562bc1f9b36
- Create a new client secret. Copy the client secret before moving away from the screen, as you will not be able to recover it: 2sY7Q~...
-
Before sharing, it is best to check that your notebooks, scripts, documents or external git projects do not contain any confidential elements or credentials. Keep your resources and the company safe.
-
With the Azure app created, assign it roles to push and pull images in the Azure Container Registry. Go to your Azure Container Registry > Access control > Role assignments > Add > Search the app you registered and add the roles:
- Owner (full access to the resource). I opted for this one, as it is the only one working in my subscription.
- The SAS documentation suggests using:
- AcrPull – pull images from the registry or,
- AcrPush – pull / push images from the registry (includes AcrPull). I think this is the absolute minimum for successful publishing.
Publishing Validation
- Assign the Azure app roles to access to access the Azure Kubernetes Service validation cluster. Select the AKS instance (I opted for the same where SAS Viya is running) and click access control. Add role assignments: Contributor.
- The above method is just one way of doing it. If you are an Azure tenant administrator, you could use Hans-Joachim Edert’s approach. Create a service principal, as in Using the SAS Container Runtime for publishing SAS models to Kubernetes on the Azure cloud, then assign roles to this service principal. His approach seems to be in line with Azure Active Directory Integration referred in SAS Model Manager’s documentation.
- Open the validation cluster’s firewall to access NodePort services (required even on a single cluster). Add an Inbound Security Rule in the Network Security Group linked to the AKS cluster. The rule will allow traffic from the public IP address of the AKS cluster where Viya is deployed, to the public IP address of the Load Balancer serving the cluster where the validation will happen. The required port range is 30000-32767. The script below was provided by Hans-Joachim Edbert in his post Using the SAS Container Runtime for publishing SAS models to Kubernetes on the Azure cloud:
RESOURCEGRP=sasuser-azuredm-rg # choose the resource group name where the AKS cluster is deployed # find the internal AKS resource group AKS_RG=$(az aks list -g $RESOURCEGRP --query [].nodeResourceGroup -o tsv) echo "AKS resource group: $AKS_RG" # find the LB IP address of the AKS cluster (outbound) OUTBOUND_IP=$(az network public-ip list --query "[].{tags: tags.type, address: ipAddress}" -o tsv -g $AKS_RG | grep aks-slb | cut -f2) echo "AKS outbound IP: $OUTBOUND_IP" # find the IP address of the AKS cluster (inbound) INBOUND_IP=$(az network public-ip list --query "[].{tags: tags.type, address: ipAddress}" -o tsv -g $AKS_RG | grep None | cut -f2) echo "AKS inbound IP: $INBOUND_IP" # Get NSG of AKS resource group AKS_NSG=$(az network nsg list -g $AKS_RG --query [].name -o tsv) echo "AKS NSG: $AKS_NSG" # Create inbound nsg rule az network nsg rule create -g $AKS_RG --nsg-name $AKS_NSG -n AllowNodePortRange \ --priority 100 \ --source-address-prefixes $OUTBOUND_IP/32 \ --source-port-ranges '*' \ --destination-address-prefixes $INBOUND_IP \ --destination-port-ranges '30000-32767' --access Allow \ --protocol Tcp --description "Allow access to pods via nodeport" # Rule is created in aks-agentpool-59212252-nsg # from Load Balancer IP of AKS cluster (OUTBOUND IP) to IP address of the AKS Cluster (INBOUND IP). - After running the scripts check the Network Security Group:
- Integrate the validation Azure Kubernetes Service cluster with the Azure Container Registry. This is required, so that the AKS cluster can pull its own images from the ACR.
# Integrate an existing ACR with existing AKS clusters by supplying valid values for acr-name or acr-resource-id as below. RESOURCEGRP= sasuser-azuredm-rg VALIDATE_AKS_CLUSTER=sasuser-azuredm-aks ACR_NAME=sascontainers az aks update -n $VALIDATE_AKS_CLUSTER -g $RESOURCEGRP --attach-acr $ACR_NAME
SAS Viya Components
Define Azure as a Publishing Destination
- You cna use the sas-viya models destination createAzure CLI. You cna find several SAS official examples on GitHub:
- Code example to define the Azure publishing destination:
# login with the sas-viya cli profile already created
cd ~
export SAS_CLI_PROFILE=${namespace}
export SSL_CERT_FILE=~/.certs/${namespace}_trustedcerts.pem
sas-viya -k auth login --user sasuser --password **********
# Use the CLI - invoke the help
sas-viya --profile ${SAS_CLI_PROFILE} models destination createAzure --help
# define the needed variables
NS=$GELENV_NS # namespace
INGRESS_FQDN=${PREFIX}.gelenablesyou.sas.com # SAS Viya URL
SASUSER=sasadmin@gelenablesyou.sas.com # SAS User
DOMAIN_NAME="AzureDomain" # credentials domain name
ACR_DEST_NAME="Azure" # destination name
ACR_DEST_DESC="Azure SCR"
ACR_SERVER=${PREFIXNODASH}acr.azurecr.io # the Azure Container Registry you will publish to
AKS_NAME=$PREFIX-aks # the SAS AKS cluster where you will perform publishing validation. You integrated it with the ACR.
# list variables which will be used
printf "\nThe destination will be created with these parameters \n"
export ACR_DEST_NAME_CLI=AzureCLI
echo "Name: $ACR_DEST_NAME_CLI"
echo $INGRESS_FQDN
echo $PREFIX && echo $PREFIXNODASH
echo ${PREFIXNODASH}acr
echo "baseRepoURL: $ACR_SERVER"
echo "Subscription: $SUBSCRPTN"
echo "Tenant: $TENANTID"
echo "Region: $LOCATION"
echo "Cluster Name: $PREFIX-aks"
echo "Resource group: $RG"
echo "Identity: $SASUSER"
echo "App Client ID: $APP_CLIENT_ID" # app registration client id
echo "App Client Secret: $APP_CLIENT_SECRET" # app registration client secret
sas-viya --profile ${SAS_CLI_PROFILE} models destination createAzure --name ${ACR_DEST_NAME_CLI} --description "ACR with SAS Viya CLI" --baseRepoURL ${ACR_SERVER} --subscriptionId ${SUBSCRPTN} --tenantId ${TENANTID} --region ${LOCATION} --kubernetesCluster ${PREFIX}-aks --resourceGroupName ${RG} --credDomainID "ACRCredDomainCLIRomeo" --credDescription "Azure ACR credentials CLI Romeo" --clientId ${APP_CLIENT_ID} --clientSecret ${APP_CLIENT_SECRET} --identityType user --identityId ${SASUSER}
- Finally, create the destination and confirm the creation in SAS Environment Manager:
Publish a model to Azure
- To publish a model, you would need a model first. Use the Quick Start to import a and test a few models in SAS Model Manager. Stop before Publish the Champion Model.
- The champion and challenger model were selected in the video above.
Publishing Validation
- In SAS Model Manager > Project.
- Go to Scoring > Publishing Validation.
- Select the model published to Azure and an input table.
- Run. At the end, click on the Results.
- Check the Output, Code and Log.
Conclusions
It was a long journey, but by this time you learnt how to configure Azure as a publishing validation for SAS models, how to publish a model to Azure and how to validate the publishing.
Now that your models have been published as container images in Azure, you could choose to stop the SAS Viya deployment in Azure, to save cloud costs. To score the published SAS models, you no longer require all the SAS resources and the SAS Viya Azure Kubernetes cluster up and running. You can deploy your container in a number of ways and score the model. Read more about it in future posts.
SAS Container Runtime is certainly an important step allowing us to reimagine how we can work with SAS Viya in the cloud. Models can now be shipped outside SAS Viya as "containerized intelligence". To paraphrase Neil Armstrong, SCR is one small step for SAS, one giant leap for the SAS customers.
Acknowledgements
This post builds on very good work done previously by:
- Deva Kumar, who wrote very clear notes about how to exploit the container images once in Azure.
- Hans-Joachim Edert wrote this excellent post. He made me realize that publishing a model to Azure and publishing validation are somehow independent concepts or operations.
- Glenn Clingroth described very well the Azure configuration.
- The GEL Team for many aspects related to deployment in Azure and SAS Viya configuration.
Want to Know More?
- SAS documentation: SAS Container Runtime.
- The SAS GitHub CLI Examples to create the Azure publishing destination.
- How to Score a SAS Model in an Azure Container Instance.
- Using the SAS Container Runtime for publishing SAS models to Kubernetes on the Azure cloud.
- How to Publish a Decision to Azure with SAS Container Runtime.
- How to Score a SAS Decision Published to Azure with SCR.
- What's New with SAS Model Manager? SAS Runtime Container.
- ModelOps and DecisionOps for the Modern World.
- Using Private Docker Publishing Destinations With SAS Model Manager and SAS Open Model Manager.
Thank you for your time reading this post. If you found it useful, like it. Please comment and tell us what you think about the new SCR and Azure publishing destination.
