After reading a comprehensive series from my colleague @RobCollum, in which he explains AWS Fargate and its role in serverless computing, I wanted to experiment with the idea of deploying SAS scoring runtime containers through AWS Fargate. By encapsulating SAS analytical models within lightweight, ephemeral containers and orchestrating their execution through Fargate, I envisioned a scenario where computational resources could be provisioned dynamically, on an as-needed basis, without the overhead of managing underlying infrastructure.   Let me guide you through a demo where I started with publishing a SAS analytical model to an AWS Elastic Container Registry (AWS ECR). This comes down in publishing the SAS model to a SAS Container Runtime image. After I used the SCR image as the source image in a Kubernetes deployment-service yaml that was applied on the same AWS EKS Kubernetes cluster where Viya is running. But I deployed that in a dedicated Kubernetes namespace named "sas-modelops-deployments". One of the reasons I used that namespace is because I configured it in such way that pods running here will launch Kubernetes nodes by using AWS Fargate. These nodes spin up quickly and once the pod is terminated the Fargate node will be terminated within seconds. I will launch the scoring pod and call the REST API endpoint with a "scoring.sh" script that is executed by launching a container with AWS Elastic Container Service (AWS ECS). By making use of AWS ECS, I'm copying the technique Rob used in his AWS serverless series . When you read the articles you will see that an AWS Batch Job and AWS Batch Queue is created. Also, you will understand it's another usage of AWS Fargate as the AWS Batch Job will run the Docker container in a AWS Fargate Batch Compute environment. The data that will be scored is coming from an AWS S3 bucket and the output scoring result is copied as well to the same AWS S3 bucket. Finally the scoring process can be monitored with AWS Cloudwatch as any event originating from the scoring job is sent to a specific log file that we can tail to see the latest updates.   Here's the AWS architecture that was used for the demo. For the rest of the article I will go in a bit more detail to each step of the process.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   1. Publishing a SAS analytical model to a AWS ECR SCR destination.   A SAS Gradient Boosting model was registered in Model Manager and published to a target destination named AWS Demo   Checking the details of the publishing destination one sees it's an AWS ECR container repository. How to create that is explained here.     Once it's published is ready to be consumed as container image. Copy the correct url so you can re-use that one in step 3 when deploying the SCR image as a pod/service on the Kubernetes cluster.         2. Configure an EKS Kubernetes "sas-modelops-deployments" namespace with AWS Fargate   On the EKS Kubernetes cluster a namespace "sas-modelops-deployments" is created.   As mentioned before EKS can be used to run pods on Amazon Fargate by integrating with upstream K8s APIs. At that moment you will run serverless pods using AWS Fargate. Before you can do you will have to extend your EKS cluster using AWS Fargate. I followed instructions from this AWS EKS getting started with Fargate link and this article. If everything went well you can inspect on the Compute tab of your EKS if you can see the Fargate profile.     Checking the details of that EKS Fargate Profile note I've associated it with my new created modelops namespace. Optional I could have added specific Kubernetes labels to the selector but here it's decided to run every pod from that namespace on AWS Fargate.     3. Deploy the SCR images as a Kubernetes deployment into AWS EKS   Based on a yaml  file that I received from my colleague Hans-Joachim Edert I created and applied a similar yaml file below on the AWS EKS cluster. Please take a special note of the SAS_SCR_REST_API_TYPE environment variable. It's set to BATCH. That's a feature that's not yet documented but should come soon. It allows to execute/score batch payload. Another attention point is the AWS-loadbalancer annotation "service.beta.kubernetes.io/aws-load-balancer-scheme". Make sure it's set to internal like explained here. That way we request for an internal AWS-loadbalancer and as such the DNS name associated with the loadbalancer will resolve to an internal IP. In a later step we will make sure that the incoming scoring request will come from a container running in the same VPC. Pay attention as well to "service.beta.kubernetes.io/aws-load-balancer-subnets". I used the same subnets that are reserved for Kubernetes pods. The last extra thing that I had to do is adding the IP CIDR range of the VPC to the LoadbalancerSourceRange of the Kubernetes service.   apiVersion: apps/v1 kind: Deployment metadata: name: hmeqtestgradientboosting namespace: sas-modelops-deployments labels: app-owner: viya_admin app.kubernetes.io/name: hmeqtestgradientboosting annotations: CapacityProvisioned: 1vCPU 2GB spec: replicas: 1 selector: matchLabels: app-owner: viya_admin app.kubernetes.io/name: hmeqtestgradientboosting template: metadata: labels: app-owner: viya_admin app.kubernetes.io/name: hmeqtestgradientboosting spec: containers: - name: hmeqtestgradientboosting image: xxxxxxxxxx.dkr.ecr.eu-west-2.amazonaws.com/hmeqtestgradientboosting:latest ports: - name: http containerPort: 8080 protocol: TCP imagePullPolicy: IfNotPresent env: # - name: "SAS_SCR_LOG_LEVEL_App.tk.MAS" # value: "INFO" # - name: "SAS_SCR_LOG_LEVEL_App.TableServices.DS2.Runtime.SQL" # value: "INFO" # - name: "SAS_SCR_LOG_LEVEL_App.TableServices.DS2.Runtime.Log" # value: "INFO" - name: "SAS_SCR_APP_PATH" value: "/score" - name: "SAS_SCR_REST_API_TYPE" value: "BATCH" securityContext: capabilities: drop: - ALL privileged: false runAsUser: 1001 runAsNonRoot: true allowPrivilegeEscalation: false restartPolicy: Always securityContext: {} imagePullSecrets: - name: ecr-secret affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.azure.com/mode operator: NotIn values: - system --- apiVersion: v1 kind: Service metadata: name: hmeqtestgradientboosting namespace: sas-modelops-deployments labels: app-owner: viya_admin app.kubernetes.io/name: hmeqtestgradientboosting annotations: service.beta.kubernetes.io/aws-load-balancer-type: internal service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip service.beta.kubernetes.io/aws-load-balancer-scheme: internal service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-xxx, subnet-yy spec: ports: - name: http protocol: TCP port: 80 targetPort: 8080 selector: app-owner: viya_admin app.kubernetes.io/name: hmeqtestgradientboosting type: LoadBalancer loadBalancerSourceRanges: - 192.168.0.0/16   So after the yaml is applied to your Kubernetes cluster you should see an internal AWS loadbalancer is created. When pinging the DNS check if it's resolving to an internal IP address.       You can also double check by inspecting the Kubernetes service in the sas-modelops-deployment namespace. It should have an "external IP" pointing to DNS name of the AWS loadbalancer.     Finally have a detail look to where the scoring pod is running:     So there's a new node that has a prefix fargate in it's name. So with that we confirm that the pod is running on Fargate.     4. Create Dockerfile with a customized "scoring.sh" script.   In a next step scoring.sh script was created. A   payload.json  is copied from an AWS S3 bucket. The kubeconfig of the EKS is downloaded. With that one can connect to the EKS cluster and can scale up the scoring deployment. After we wait till the scoring pod is pulled and running. Once running we can submit scoring request by executing the curl command sending the payload to the correct AWS loadbalancer endpoint that was created when the deployment was applied in step 3. Finally, results are sent to the same AWS S3 bucket, and the scoring deployment can be scaled back to 0.   aws s3 ls aws s3 cp s3://xxx/aws-summit-london/sample_scr_batch_payload.json /tmp/sample_scr_batch_payload.json aws eks --region eu-west-2 update-kubeconfig --name summit-london-eks kubectl get nodes ls -ltr /tmp kubectl scale deployment -n sas-modelops-deployments hmeqtestgradientboosting --replicas=1 kubectl wait pods -n sas-modelops-deployments -l app.kubernetes.io/name=hmeqtestgradientboosting --for condition=Ready --timeout=120s echo $(curl ident.me) echo "50sec sleep start" sleep 50 echo "sleep over" cd /tmp timestamp=$(date +""%d%m%y_%H%M%S) echo $timestamp cat /etc/hosts curl -X POST -H "Content-Type: application/json" -m 10 -d @sample_scr_batch_payload.json http://k8s-sasmodel-hmeqtest-xxxx-yyyy.elb.eu-west-2.amazonaws.com/score -o /tmp/out.json cat out.json | jq '.' > out_$timestamp.json aws s3 cp /tmp/out_$timestamp.json s3://xxx/aws-summit-london/out_$timestamp.json cat /tmp/out_$timestamp.json kubectl scale deployment -n sas-modelops-deployments hmeqtestgradientboosting --replicas=0 #sleep 120   The scoring script is used in a  Dockerfile.   #Build this container to run in AWS Batch #Start with the latest AWS-CLI v2 image FROM public.ecr.aws/aws-cli/aws-cli:latest #Consider using "--platform=linux/amd64" if building on Apple Silicon #Install additional software as needed RUN yum update -y && yum install -y wget unzip WORKDIR /tmp RUN curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.20.4/2021-04-12/bin/linux/amd64/kubectl RUN chmod 755 ./kubectl RUN mv ./kubectl /usr/local/bin/kubectl RUN kubectl version --short --client #Copy your project code into the container RUN yum install -y jq COPY scoring.sh / #Default execution ENTRYPOINT ["/bin/bash","/scoring.sh"] #Default parameters CMD ["aws"]   You can build the container with this command:   docker build -t scoring .   5. Create AWS ECS, AWS Batch Queue and AWS Batch Job definition.   This step is more or less exactly what I've learnt from @RobCollum   Serverless Operations in AWS, part 2: Deploy and Run .   If you followed Rob's series you will end up with an AWS Batch-hello-aws-fargate AWS Elastic Container Service. If you then repeat all his steps but using the scoring Docker image created in step4 you will have a 2nd AWS ECS like I have:     And with that when checking AWS Batch you should see extra scoring compute environment and and extra scoring queue.       Finally an extra AWS Job Definition was created which refers to the scoring Docker Container image that is pushed to the correct AWS ECR:     6. Launch AWS Batch Job running on AWS ECS using AWS Fargate.   It's time to launch the AWS Batch job and see the scoring in action. You can submit the job from a terminal window where AWS CLI is installed.   PROJECT=scoring JOBDEF_NAME="$PROJECT-jobdef" QUEUE_NAME="$PROJECT-queue" NOW=`date +"%H%M%S"` # current time showing only MMSS. JOB_NAME="$PROJECT-$NOW" echo -e "\naws batch submit-job --job-name ${JOB_NAME} --job-definition ${JOBDEF_NAME} --job-queue ${QUEUE_NAME}" aws batch submit-job --job-name scoring-110406 --job-definition scoring-jobdef --job-queue scoring-queue   If you check from in AWS Console the AWS Batch Dashboard you should see that a new job is starting:     7. Monitor scoring execution with AWS CloudWatch   Going to AWS Cloud Watch allows you to follow all output events from the JOB launched in step6. AWS Fargate launch a container with an AWS role having the permission to interact with AKS. So remember in the scoring.sh script we download the Kubernetes kubeconfig and as such we can interact with the EKS Kubernetes cluster. Which means we can scale hmeqtestgradientboosting deployment to 1. That's also the moment that an AWS Fargate EKS node will start.  So same happened when we deployed the SCR pod in step 3.          Once the scoring pod is running the incoming data can be scored. Note that I added a sleeping time of 50sec allowing some time to the SCR pod to startup it's scoring application.       Once scoring is done the special Kubernetes AWS Fargate node will disappear.    Thanks for reading. ... View more

Azure spot instances are a cost-effective option provided by Microsoft Azure for running virtual machines (VMs) and workloads. Under traditional Azure virtual machine pricing, you pay a fixed rate. Spot instances allow you to take advantage of spare capacity in Azure’s data centers at significantly reduced prices. However, there’s a trade-off: Azure can reclaim the spot instance if the capacity is needed by paying customers. This therefore gives lower-cost compute resources but with the possibility of interruption. Wouldn’t it be great if we could use spot instances to run low-priority SAS Viya workloads? The key is to find workloads that can be restarted after an interruption without causing a problem.   The idea I wanted to test the idea of creating a “spot host type” within SAS Workload Management deployed on an Azure AKS service. This new host type would be required by a “spot” queue. If workload is submitted to the spot queue, Azure spot instances should be launched via Azure autoscaling. On Kubernetes, these Azure spot instances would have to be Kubernetes nodes running within an Azure spot node pool. We therefore start by creating an Azure spot node pool on the Azure Kubernetes service where we deployed SAS Viya.   Azure Spot Instances offer substantial cost savings, but they are not suitable for all workloads. Those that require uninterrupted, constant resources should be run on regular on-demand instances to avoid potential interruptions. When using Azure Spot Instances with SAS Viya, careful workload planning, job monitoring, and resource management are essential to maximize the benefits of cost savings and scalability, but manage the risk of interruption   Azure Spot node pool How to add an Azure Spot node pool is explained here. I used the command:   az aks nodepool add --resource-group hackinsas-viya202307-rg --cluster-name hackinsas-viya202307-aks -s Standard_L8s_v2 --name computespot8 --priority Spot --eviction-policy Delete --spot-max-price -1 --enable-cluster-autoscaler --min-count 0 --max-count 1 -c 1 --labels wlm=spot k8s\.azure\.com\/aks-local-ssd=true workload\.sas\.com\/class=compute launcher\.sas\.com\/prepullImage=sas-programming-environment --node-taints=workload\.sas\.com\/class=compute:NoSchedule --no-wait   If you set the max price to -1, the instance won't be evicted based on price. As long as there's capacity and quota available, the price for the instance will be the lower price of either the current price for a Spot instance or for a standard instance.The minimal node count is set to 0. That means that initially, you see one node created. However, shortly after you launch this command, the nodes will probably scale to 0. That might sound surprising, especially if you were expecting SAS compute pods to use this node to schedule workloads. The node is correctly labeled with “workload.sas.com/class=compute”, so in theory SAS compute pods could land here. However, spot instances have an additional label that stops scheduling: “kubernetes.azure.com/scalesetpriority=spot”. What do you need to do to be able to submit a SAS batch program to run on these instances?   Installing Kyverno We can use a tool called Kyverno to add the required toleration (“kubernetes.azure.com/scalesetpriority=spot”) to the SAS pods. Kyverno enhances Kubernetes by providing a policy-as-code framework that helps you enforce and manage various aspects of your workloads and resources in a Kubernetes cluster. Kyverno can be installed with these instructions. You can choose to install Kyverno with Helm or just download and apply an install.yaml:   kubectl create -f https://github.com/kyverno/kyverno/releases/download/v1.10.0/install.yaml   Adapting the SAS Workload Orchestrator and the SAS Image Staging Configuration DaemonSets We can use Kyverno to apply an extra toleration to the SAS Workload Orchestrator (SWO) Daemonset and to the SAS “prepull-ds” of the Staging Configuration process. An exampe yaml file below is available in the manifests folder. When pre-conditions are met, this code will mutate the selected resources. The mutation adds the missing toleration to two DaemonSets: the sas-workload-orchestrator DaemonSet and any DaemonSet starting with prepull-ds in its name.     kubectl apply -f ds-add-toleration-scalesetpriority.yaml -n <viya-namespace> Once this toleration has been added, sas-workload-orchestrator pods will be able to run on the spot nodes that were labeled with “workload.sas.com/class=compute”.  If the Kubernetes cluster starts any spot nodes in future, they will also automatically get sas-workload-orchestrator Daemonset pods. We have therefore used Kyverno to mutate the SWO daemonset to give it the necessary “kubernetes.azure.com/scalesetpriority=spot” toleration to get past the taint. It can now chase the “workload.sas.com/class=compute” label that was applied when provisioning the spot nodes. The node is now electable as candidate to host SAS compute workloads, and will be displayed as such in the workload orchestrator in SAS Environment Manager.   Looking at the details of the “computespot” host, all expected labels are set:     We added a toleration to any DaemonSet starting with “prepull-ds-*” because we want to let the SAS Image Staging process start required pods on spot nodes. That SAS Image Staging process will use a daemonset at a given time interval to ensure that relevant images have been pulled to hosts. Relevant images mainly means the sas-programming container. This will only be useful if at least one spot instance will run before you schedule workload on it. In practice, if the Azure autoscaler balances from 0 to n nodes, you will probably start with 0 nodes. After you submit workloads intended to run on spot instances, you will probably have to wait before the sas-programming container is pulled on the node. This reminds us again that spot instances should only be used for low priority jobs, when waiting five minutes doesn’t matter.   Adapting the SAS Workload Manager scaling pod. SAS Workload Orchestrator can integrate with the Kubernetes Cluster Autoscaler using this documentation. This means that the SAS Workload Orchestrator can use the Kubernetes Cluster Autoscaler to launch new compute nodes. To do this, a sas-workload-orchestrator-scaling-pod will be scheduled on a compute node. That scaling pod should run on a compute node with the correct resource properties specified via the SAS Workload management queue. We therefore need to know how to create a “spot queue”. Jobs submitted via a “spot queue” will only run on Azure spot instances. If no spot nodes are available, they can be autoscaled via a sas-workload-orchestrator-scaling-pod. However, as before, the scaling pods requesting a specific node can’t run on our spot nodes. We know why by now: they do not tolerate the spot taint. We need to create an extra Kyverno cluster policy via pod-add-toleration.yaml that will add that toleration to any scaling-pod in our Kubernetes Viya namespace.   kubectl apply -f pod-add-toleration-scalesetpriority.yaml -n <viya-namespace>     Creating new sas-batch-pod-template-spot podtemplate To make sure sas-batch-server pods hosting workloads meant for spot instances can run, we need to add the same extra scalesetpriority toleration needs to the podTemplate linked to the SAS Batch service launcher context. We need to create an extra sas-batch-pod-template for that. We can copy the original sas-batch-pod-template and modify the name and tolerations in the definition. An example yaml is available in the manifests folder.   kubectl apply -f sas-batch-pod-template-spot.yaml -n <viya-namespace> Adding spot host type to SAS WLM Creating a new SAS WLM host type is described in this documentation link. The host type created in our example here is named “spot”. In each host type, definition tags used to identify the hosts are specified. These tags can identify host characteristics that are relevant to SAS Workload Orchestrator. In the screenshot, the distinct tag used to identify Azure spot nodes is “wlm=spot”.   Adding spot queue to SAS WLM Rather like we created host types, we can create a queue “spot”. You can assign different hosts or host types to each queue. Each queue definition includes a list of the host types and tags that identify the hosts allowed to process jobs from the queue. If the queue definition includes tags, then when the SAS Workload Orchestrator manager evaluates which host to use to process a job from the queue, it checks the tag values on the queue and host, and sends the job to a host with a matching tag. By assigning the group of spot host type we created before, you can ensure that low-priority jobs will be processed on the Azure spot nodes.         Create new SAS Batch service launcher context spot The next step is creating a new SAS Batch service launcher context spot. From in SAS Environment Manager click on the new launcher context icon:     Fill in the details for the new launcher context as shown in the screenshot.     Creating new spot Batch context The final step is to create an extra Batch context. In the view where you defined the launcher context, switch to Batch contexts. Create a new one and name it ‘spot’. For the Launcher context, choose the one created in Step 7.     You can associate server contexts with SAS Workload Orchestrator queues. For information about associating a queue, see Contexts Page. Don’t forget to choose the dedicated “spot” queue as your SAS Workload Orchestrator queue.   Time for action As a test, you can launch workloads via a simple test script. Here is an example script that you can run from in a Linux session. You can submit any program, just make use of the parameter “c” that allows you to provide the context, which should be specified as “spot”. This will transfer the workload to a “spot” queue that is supposed to run only on Azure Spot Instances.   export SSL_CERT_FILE=/<ANY_DIR>/trustedcerts.pem INGRESS_URL=https://viya.sas.com /home/ubuntu/sas-viya --profile default profile set-endpoint "${INGRESS_URL}" /home/ubuntu/sas-viya --profile default profile toggle-color off /home/ubuntu/sas-viya --profile default profile set-output text /home/ubuntu/sas-viya --profile default auth login -u <user> -p <password> for i in {1..50} do echo "Welcome $i times" /home/ubuntu/sas-viya --profile default batch jobs submit-pgm -c spot --pgm prog1_hmeq.sas --restart-job /home/ubuntu/sas-viya --profile default batch jobs submit-pgm -c spot --pgm prog2_baseball.sas --restart-job /home/ubuntu/sas-viya --profile default batch jobs submit-pgm -c spot --pgm prog2_baseball.sas --restart-job done Before we kick off this shell script, we see four compute hosts. Looking closer, one is inactive. This is a spot compute host that is no longer part of the Kubernetes service. This may have been scaled down, possibly because no workload was submitted.   As soon the sas-workload orchestrator re-scans the Kubernetes services, inactive nodes are no longer displayed: Lets launch the shell script and see what happens. As soon the script is started, some simple SAS programs are submitted in batch: If we correctly set up the “spot” context, you will see the number of pending jobs rapidly growing in the “spot” queue: When there is no node available with the right criteria (meaning a Kubernetes node with a label wlm=spot), SAS Workload Management will launch a sas-workload-orchestrator-scaling pod and a Kubernetes node with the right criteria to schedule that pod.     The new node has the right criteria (it is tainted and labeled with “workload.sas.com/class=compute”) and SAS Workload Management will therefore recognize it as compute host. The sas-workload-orchestrator DaemonSet will launch a sas-workload-orchestrator pod on the Azure spot node and then the process will be confirmed, with the host displayed in the orchestration dashboard in SAS Environment Manager. Note status is OPEN-FULL, which is correct: immediately after starting the node, it will accept the maximum number of jobs permitted to start running.   However before the first programs can run, the sas-programming container image needs to be pulled onto the new compute node. This was explained and configured in Step 2, and in theory is done by the SAS Image staging process. However, there has not yet been time for that process.   We can confirm shortly after the first sas-batch-servers are scheduled that a prepull-ds pod is running on the aks-computespot node. The time cycle when these prepull pods are running can be configured.     Five minutes after the first SAS batch jobs were scheduled, they were running on the new compute nodes. Within those five minutes, the nodes were launched and all required pods were able to run on it. For a job that can “wait” and doesn’t require the highest priority, I think that’s a good result. This is especially true once you realize that these compute nodes can have discounts of up to 90 percent compared to pay-as-you-go prices.   The WLM dashboard confirms that the system is configured to run at most five jobs. All other jobs will stay in a pending state until a slot becomes available.   You can follow up the status of the submitted workload. This is important because programs can be interrupted if the spot instance is claimed back for a paying customer. You can automate this checking and also automatically restart the job if required.     When all goes well all your submitted workloads will be executed and all counters will be reset to 0 on the spot queue:   The autoscaler will then scale back the spot node pool to the minimum. In our example, that’s 0. We then go back to our start position, where the single aks-computespot in the WLM dashboard has a status of OPEN-INACTIVE.     Finally, if you check Kubernetes Nodes via kubectl or Lens, you will see confirmation that you have no aks-computespot hosts:   It means as well your batch job is completed.    Spot instances are a great way to get more benefit from the flexibility offered by your cloud vendor to reduce your overall cloud spent. While the above example is completely built for Azure, similar capabilities exist in both AWS and GCP.  This post illustrates the concept, to make it operate in a more customized production environment additional testing and configuration might be needed.   Unfortunately using Spot instances is not supported.     ... View more

Introduction In part1 of this series we created a customer connector via the Microsoft Power Automate platform (part1). We continued this series with a part2 where we created Power Automate Flow consuming the customer connector from part1.  Now let's integrate this Power Automate Flow in a Power App and demonstrate how SAS Viya REST endpoints can be consumed in our customer business applications.   Create a Power App Go to https://powerapps.com/. Let's start by creating an empty app.   You can start with adding Text labels:     After you can add 2 Text input fields next to each label. Make sure you switch format to Number.     To make next step easier change the name of those 2 input fields to 'limit' and 'start':     As next step add a button and label as you want.   t's now time to add the Power Automate flow we created. Click on the Power Automate and from there you can add your flow:     Now change the Function of the 'Fetch Rows' button to:     Then drag and drop the 'Data table (preview)' from the Layout section to the canvas:   Make sure you select the table you referenced in the function of the button. If you followed the example it should be 'tbl'. After that you can click 'Edit fields'   There you can click 'Add field' and select any number of fields that are returned by the PowerAutomate flow.   You should be able to preview the app.   ... View more

Introduction   On developer.sas.com it says that the SAS Viya REST APIs are designed for enterprise application developers, who intend to build on the work of model builders and data scientists, to deliver apps based on SAS Viya technology. With the SAS Viya REST APIs, you can create and access SAS resources using any third party application that supports http(s) calls.   Microsoft Power Platform connectors are wrappers or proxies around an API that allow the underlying service to talk to Microsoft Power Automate, Microsoft Power Apps, and Azure Logic Apps. SAS offers a SAS Decisioning Connector to run SAS Intelligent decisions that are deployed on your SAS Viya platform.   However Microsoft also gives you the option to build your own custom connectors which allows you to communicate with services that aren't available as prebuilt connectors. That way you could for instance connect from within PowerApps to a SAS model published in a SAS Scoring Container.   In this blog series that is split up in to three different parts, we will explain how you can create a Power App application that displays data retrieved from your SAS Viya environment. In the first part of the blog we will start with creating custom connector that list data loaded into the SAS Cloud Analytic Services (CAS)   Prerequisites The SAS Viya environment use TLS to secure data in motion.  You need a certificate from a commercial vendor like Digicert or Verisign.  Self signed certificates are not be trusted by the Power Automate of Power App platforms of Microsoft. And you are not able to import a self signed certificate.   In case the SAS Viya environment is protected by a firewall, you will need to whitelist the IP from which Power Automate requests are sent. More info about these IP addresses can be found here.   Create a Power Platform custom connector     Go to make.powerautomate.com and sign in to your Microsoft account. Click on New custom connector and select Create from blank.     Give the connector a name.     Now you can use a wizard that will guide you through the steps necessary to create your customer connector in Microsoft Power Automate. We will guide you through each of these steps and call out what needs to be setup for your custom connector.     General   Fill in your Viya hostname in the Base URL field.     Security   Now click on Security. Choose API key as authentication and fill the fields like you see below.     Definition   Click Definition and in the next screen you can determine actions that users will be able to perform. In our example we will only add 1 action: the retrieval of a number of rows from a CAS table. We start by clicking on New action and then we will define some general properties of the actions.     After that it's time to specify the details of the Request, for that you can click Import from sample.     Enter the following URL. We are going to modify the values between the brackets.   https://{viya-fqdn}/casRowSets/servers/cas-shared-default/caslibs/{cas-lib}/tables/{cas-table}/rows?limit=a&start=b   Parameter Description viya-fqdn The fully qualified domain name of the SAS Viya server. cas-lib The CAS library where the table is located. cas-table The CAS table name.   *) Leave the limit and start parameter to the respective values as shown above   Click Import.     Test   Before you are able to test the newly created customer connector you need to first save it by clicking on Create connector.     Once the custom connector has been saved we can start our test. But first we have to create a new connection. If no connection is listed click on "New connection"     To establish the connection to the SAS Viya environment you need to provide an API key. This API key is something that needs to be generated within the SAS Viya environment. How to generate this API key can be found here.        The API key needs to be in this format: Bearer <access_token>.  An example of what the API key could look like is shown here.   Bearer eyJhbGciOiJSUzI1NiIsImprdSI6Imh0dHBzOi8vbG9jYWxob3N0L1NBU0xvZ29uL3Rva2VuX2tleXMiLCJraWQiOiJsZWdhY3k tdG9rZW4ta2V5IiwidHlwIjoiSldUIn0.***********************6InVpZD12aX***********l9zaWciOiI4YTdjYzE4Ni0xZjE1L T***************************31HN-Upuhs-PKk7SEn_FWRyL0tasi0nNb_H0f****************VARJo67cpzyBxyv1gQg   You can validate if the connection was created successful by selecting connections:   You can go back to customer Custom connectors to test the connection. Click Test operation.   If the Access Token was generated correctly you will be able to run a successful test.     That concludes the creation for the custom connector. Now you're all set to create a Power Automate Flow which can be used in a later step in a Power App. We will explain how to do that in the second part of our blog series.   Let us know if you have any questions in the comments! ... View more

Kubernetes inherently comes with workload management controls. These are great for an IT administrator to maintain stability and harmony in their environment. These controls can extend to SAS. You may ask yourself, "since we have native workload management in our infrastructure, do I need more?" The answer is, "Yes."  In this article I'd like to introduce a new solution introduced by SAS, for SAS cloud deployments: SAS Workload Management.   Like my colleague @EdoardoRiva explains in his community article, Kubernetes provides workload management. In addition, he points out the latest release of SAS Viya is deployed on Kubernetes. Edoardo provides a high level overview of the advantages on SAS Workload Management. And in a very recent article Edoardo explains how the architecture is impacted when SAS WLM is added.  Having a curious mind, I was interested in discovering what SAS Workload Management brings in addition to the existing Kubernetes workload capabilities.   I wanted to see it live and experience myself how it all works, so let me explain how I configured a demo covering the essentials of SAS Workload Management.   By reading the SAS Viya Workload Management documentation I started to learn about pre-emption, queues, compute contexts, host-type, etc. Now let's put it all together.   Spin up Kubernetes Cluster with multiple compute nodes   I started with creating a SAS Viya Kubernetes environment in Azure by using the instructions in the viya4-iac-azure GitHub repository. In contrast with what I normally do (single node), I created an AKS cluster with four compute nodes in the SAS Compute Node Pool. I wanted to play with spreading workload over the different compute hosts and reserving a host for specific tasks. Figure 1 displays the organization of the cluster members.   Figure 1 - Kubernetes cluster organization for SAS Viya deployment   Remember, if you’re using the SAS IaC scripts to create your AKS cluster it’s very easy to configure a AKS cluster with four compute nodes.  Specify the min_nodes and max_nodes parameters in the Terraform.tfvars file as seen in Figure 2 below.   Figure 2 - min_ and max_nodes parameter settings in the Terraform.tfvars file   SAS Environment Manager & Workload Orchestrator   After deploying SAS Viya I navigate to the SAS Environment Manager, where I open the Workload Orchestrator. Consider the Dashboard tab in Figure 3 below.   Figure 3 - Workload Orchestrator Dashboard in SAS Viya Environment Manager   Figure 3 shows me I have three Queues available. I also get confirmation I have four compute servers available, which I can use to spread the SAS workload.   Checking the current SAS workload details on those queues is done via the Queue tab (Figure 4 below). Similar there’s a Hosts tab to see the current workload on all the hosts (Figure 5 below).   Figure 4 - Workload Orchestrator Queues   Figure 5 - Workload Orchestrator Hosts   Configuring Queues   Before launching any workload, let’s check the Configuration tab (Figure 6 below). Here we have a lot of options to define where the jobs submitted to a specific queue will run.   Figure 6 - Workload Orchestrator Configuration tab     Consider the Queue Configuration in Figure 7. First of all, you can assign a priority to the queue. Jobs submitted to queues with a higher priority will run before jobs submitted to queues with a lower priority.  Look back to Figure 4 and it's clear I have configured three queues in my demo environment: default, priority and interactive, respectively having priority 10, 20 and 30.   Figure 7 - Workload Orchestrator Queue Configuration   There’s also possibility to specify a list of other queues the queue can preempt.      Note that I selected for the interactive queue as Host Type SAS Studio interactive. This means for the interactive queue I would like to run the jobs that are submitted to this queue on hosts reserved purely for this type of work. More about configuring Host Types in the next section.     As my goal is to do a basic SAS Workload Management demo, so for now, that’s enough configuration for me. However, it should be clear there are many more to configurable queue parameters. For example, how many jobs can run per user and per host, who can submit jobs to the queue, or who can administer this queue.  Also, worth mentioning is the possibility to specify a completely new group of settings based on the time of the day.    Please  the SAS Workload documentation for further details.   Configuring Host Types   As mentioned earlier, I configured the interactive queue to run jobs on hosts that are of type SAS Studio Interactive.  These Host Types are configured by an administrator when selecting in the same configuration tab the Host Types (see Figure 8).   Figure 8 - Workload Orchestrator Hosts Configuration   For the SAS Studio Interactive Host Type, I assigned one specific compute node.  Here, I selected it by name. But in the field Host Properties, you can also make use of Kubernetes Node Labels.  I did that as well.  It should be in sync with a label on my compute nodes.   By running the following kubectl command I found my compute nodes and validated a "wlm" label exists.    kubectl get nodes -o=custom-columns=NODENAME:.metadata.name,LABELS:.metadata.labels | grep compute | grep wlm   As you can see in the results in Figure 9, three out four nodes have "wlm=default" but one is labelled with "wlm=interactive".  That’s the node I’m targeting for Host Type SAS Studio Interactive and where I would like to run jobs submitted to the interactive queue. I will discuss shortly how to configure so only SAS Studio jobs land & run on this interactive queue and associated compute host labelled "wlm=interactive".   Figure 9 - compute nodes   Lastly, I want to focus on another configuration difference between my default host type and the SAS Studio interactive host type on my demo environment (see Figures 10 and 11).   Figure 10 - default Host Type       Figure 11 - SAS Studio Interactive Host Type   I specified a different number for maximum jobs allowed. For the default hosts it’s set to four, whereas for the SAS Studio interactive I allow five jobs. During the demo it will help me to quickly discover which node is recognized as compute node to run SAS Studio jobs.   Configuring contexts   Let's now explore how to make sure a SAS Studio job is ending on the interactive queue and the associated compute node reserved for that queue.  Please refer to another article from Edoardo to learn all the details about the SAS runtime environment with the latest SAS Viya release on Kubernetes.  For now it’s important to know when requesting a SAS programming environment a Kubernetes SAS programming pod launches.  Depending how or from where the request is coming, a different context will be used: Compute context (with SAS Studio, SAS Model Studio, ..) Connect context  (with SAS/Connect) Batch context (when submitting program via sas-viya-client) Simply said, the program launches with a specific context, as seen in Figure 12.   Figure 12 - List of contexts   And with each compute, batch, or connect context a launcher context is associated, as seen in Figure 13.   Figure 13 - Launcher contexts SAS Studio compute context   Let’s take the example of SAS Studio: in Figure 14, a SAS Studio compute context will go together with SAS Studio Launcher context.   Figure 14 - SAS Studio compute and launcher contexts   And as seen in Figure 15, in the SAS Studio Launcher context, one specifies the SAS Workload Orchestrator queue. I assigned the interactive queue.  Remember jobs submitted to this queue will launch on computes nodes of host type SAS Studio interactive.   Figure 15 - SAS Studio launcher context   SAS viya-cli batch-context   As I will also use sas-viya-cli to launch programs from batch in the demo let me share how to configure the possibility to run priority batch jobs.    First, I go to the batch contexts. Next, to the predefined default context. I added an extra one here, a new, customized context named priority (see Figure 16). Figure 16 - Batch context   Checking the details, notice I specify in this context the SAS Workload Orchestrator queue. I assigned the priority queue as it has a higher priority than the default queue. I could have specified the Workload Orchestrator queue on the associated SAS Launcher context (that would be the SAS Batch Service Launcher context), but I preferred to make use of this Batch Context Workload Orchestrator field.    The bottom line is, I now have the choice to launch batch jobs with a default or priority context. As the default context  runs on the default queue, jobs submitted on that queue have to wait for an available slot on the compute nodes or might be even preempted by jobs with a higher priority.   Demo script   Time to see this in action. The following script launches SAS batch jobs with the  SAS Viya CLI.  I’m using three different profiles, each authenticating with a different user.   #!/bin/bash export SSL_CERT_FILE=/data/config/vdm/security/cacerts/** INGRESS_URL=https://myhost.sas.com /opt/sas/viya/home/sas-viya --profile default profile set-endpoint "${INGRESS_URL}" /opt/sas/viya/home/sas-viya --profile default profile toggle-color off /opt/sas/viya/home/sas-viya --profile default profile set-output text /opt/sas/viya/home/sas-viya --profile default auth login -u frederik -p password /opt/sas/viya/home/sas-viya --profile prod profile set-endpoint "${INGRESS_URL}" /opt/sas/viya/home/sas-viya --profile prod profile toggle-color off /opt/sas/viya/home/sas-viya --profile prod profile set-output text /opt/sas/viya/home/sas-viya --profile prod auth login -u edoardo -p password /opt/sas/viya/home/sas-viya --profile test profile set-endpoint "${INGRESS_URL}" /opt/sas/viya/home/sas-viya --profile test profile toggle-color off /opt/sas/viya/home/sas-viya --profile test profile set-output text /opt/sas/viya/home/sas-viya --profile test auth login -u sundaresh -p password123   After the authentication script, I loop over a set of commands, each launching SAS Programs.  Each loop launches two jobs with the default profile, one job with the test profile, and a series of jobs launch SAS programs with the production profile.  If you look carefully, you will see when using the prod profile I add "-c priority".  This means the request launches with the priority context of the batch context and consequently submits it to the priority queue.   for i in {1..50} do echo "Welcome $i times" /opt/sas/viya/home/sas-viya --profile default batch jobs submit-pgm -c default --pgm prog1_hmeq.sas --restart /opt/sas/viya/home/sas-viya --profile test batch jobs submit-pgm -c default --pgm prog2_baseball.sas --restart /opt/sas/viya/home/sas-viya --profile default batch jobs submit-pgm -c default --pgm prog2_baseball.sas --restart /opt/sas/viya/home/sas-viya --profile prod batch jobs submit-pgm -c priority --pgm prog2_baseball.sas --restart /opt/sas/viya/home/sas-viya --profile prod batch jobs submit-pgm -c priority --pgm prog3_baseball.sas --restart /opt/sas/viya/home/sas-viya --profile prod batch jobs submit-pgm -c priority --pgm prog4_baseball.sas --restart /opt/sas/viya/home/sas-viya --profile prod batch jobs submit-pgm -c priority --pgm prog5_baseball.sas --restart done   Queue and Host statuses   Soon after the initial loop runs, the priority queue will run the maximum of 12 jobs. A the same time the default queue has all jobs in pending state (see Figure 17).   Figure 17 - Queue Status   This is exactly what we expected to be happen because we defined max number of four running jobs and we also requested three out of four compute nodes for normal workload use.  So, once you have 12 jobs you have reached the maximum and jobs submitted to the priority queue will take precedence over the default queue.   Also, note that one host has nothing to do because we reserved it for SAS Studio sessions.  That’s why the first three hosts have status OPEN-FULL and the last one OPEN-OK (see Figure 18).   Figure 18 - Host Status  If a SAS Studio session is launched, a running program will be show on the interactive queue (see Figures 19 and 20).   Figure 19 - SAS Studio session   Figure 20 - Updated interactive queue Recorded demo   If you want to see a demo of the SAS Viya Workload Management, refer to the video below by my colleague Mark Schneider (@Mark_sas)  SAS Advisory Product Manager.  Mark talks about SAS WLM and introduces the demo described in this blog.  Have a look and let us know your thoughts.     ... View more

I think a lot of us know the feeling: you have a great idea and want to try it immediately on a SAS Viya environment.  So you need a deployment up and running in the next hour.  But then there's no server or nobody has the knowledge to install the software.   So before you know the momentum to execute your idea is over.   And yes I'm thinking about running an Analytics Sprint, or discussing a potential proof of concept in the meeting and starting with it the day after, or you want to test a brand new feature that has just been released with the latest SAS Viya cadence, or simple testing the connection of your environment to an IoT device.    In that case it would be very good to have something ready that automatically installs and configures such a SAS Viya deployment. The environment needs to be up and running quickly and consistent with the previous time you used it. Without human errors or mistakes that typically happen during a manual deployment. That's exactly what we developed here, and since we created this it already saved us a lot of time for use cases as described here above. How did we do it: by making use of Azure DevOps pipeline and installing the latest release on Azure AKS clusters.  Have a look to the following video to see it in action.         ... View more

This is part2 of a video recording how I perform a Blue/Green deployment on an existing demo Viya environment. My goal was to update Viya from stable 2021.1.3 to stable 2021.1.4. and transfer my end-users smoothly from the old version to the new version.     In part1 I recorded a video where I demonstrated the backup/restore process.  At the start of the next video I assume that the update has been done successfully by following the documentation.  So what's left is making sure that my users are now switched to the new Viya 2021.1.4 cadence.  The user will continue to use the same url but will be redirected to the new environment as soon we request a DNS switch of our INGRESS URL Hostname.  At SAS we can use internal application for that.   Watch the following video for the details how I did that. ... View more

As my colleague @RobCollum wrote in his post on communities.sas.com the combination of CI/CD and DevOps has led to a whole new world how software today is built, packaged and delivered.  Since Viya 2020 SAS is releasing software that way.  More specifically SAS Viya has two release cadences: Stable or Long-Term Support cadence.   Especially with the Stable cadence the updates come quickly: at least once a month and probably more when patches are released. While it’s nice to have continuous updates to the SAS software, production environments where users are actively creating content are often protected from frequent, ad-hoc changes.   That’s why I like the idea of a Blue/Green Deployment. I can make use of the backup/restore functionalities in SAS Viya to first create an identical box as my source environment.  Working this way we can say it's a Viya 4 to Viya 4 Full System migration.  But make sure you use the same Ingress Hostname.   After restoring the backup content I can safely apply the update on the new target environment.  If tests can confirm that everything is still working and all my Data, SAS Models, SAS VA reports, .. are still available I can execute the Blue/Green switch by changing the DNS entry of my Ingress Hostname to a new LoadBalancer IP address.   In this blog I wanted to share a video how I run the first part.  So how I restore a backup on a fresh Viya environment.   For the the update process I refer to the Viya Operation guide.  How to do the Blue/Green switch I will post in a follow up of this article but don't hesitate to drop me an email if you want to learn more about it. Let me start with introducing the flow that you will see in this video.   I will start with checking if a scheduled or ad-hoc backup completed successfully.  With the following kubectl get jobs command I can retrieve all backup jobs. (please note I’m using the alias k for kubectl)   k get jobs.batch -n hackinsas-eugtp -l "sas.com/backup-job-type=scheduled-backup" -L "sas.com/sas-backup-id,sas.com/backup-job-type,sas.com/sas-backup-job-status,sas.com/sas-backup-persistence-status" We will need the SAS-BACKUP-ID later when we do the restore of the content.   If you want to restore the backup content to a new target environment you will need to access the physical location of your Kubernetes Persistent volumes of the current Viya environment.  In my case the persistent volumes are provisioned via a NFSServer that I can access via a jumpbox. On that jumpbox I create two zip files (BACKUP_ID_cas.zip and BACKUP_ID_common.zip ) containing all the files I need to restore a backup.     The BACKUP_ID_cas.zip and BACKUP_ID_common.zip needs to be migrated to the similar Persistent Volumes of the target environment.  There you can unzip the files so that you have the backup available on the target environment.  Don’t forget to allow sufficient permissions to the extracted folders. Once  your BACKUP is on the target Persistent Volumes you can start following the ‘Perform a Restore’ like described here in the SAS Viya Operations guide.  (Make sure your select the guide for your release and cadence).   Watch the following video demonstrating backup and restore of SAS Viya content.     Big thanks to @RobCollum @GerryNelson @RPoumarede  for their feedback and help in creating this article. Also have a look to this articles from my colleague @GerryNelson:  a generic method for copying backup packages for migration or disaster recovery (introduced with 2021.1.4) a first look to backup and restore in SAS Viya on Kubernetes   ... View more

I was recently involved in work to design and set up the architecture for the SAS Global Hackathon, and especially setting up the SAS Viya sandboxes for all the teams. My main area of work—and interest—is analytics deployment, so it was interesting to see a rapid deployment and operational use in action. There were several user experience design factors that were considered in the deployment.  You can find a detailed technical description of my work done for the Hackathon in this blog but here’s what I learned from the experience.   Open source integration was a key requirement for users Users had some very important requirements. First, they needed open source integration with the sandbox. We provided them with code that would enable them to access the sandbox from Python or R-Code. We know that easy and robust integration of different technologies is essential for hackathons. However, it can also be important in everyday analytics deployments too. Data scientists often have different preferences for programming languages, but still need to be able to work together, and you need an environment that facilitates this.   It is essential to create trust in the environment We wanted hackathon participants to be able to get their data into the sandbox easily. Each team therefore had access to a Microsoft Azure Storage Account. We also carefully considered the rules for sandbox access, including firewall rules, whitelisting IP addresses, and authentication. These options meant that participants were confident about using the sandbox. This created a level of trust that was essential for success in the hackathon. This is very definitely a lesson that can be applied more widely: users need to be able to trust their environment and know that it is safe to use.   Flexibility can be embedded in several ways We designed three types of architecture to provide the right environment for different types of use case. This allowed us to support multiple use cases without having to create bespoke environments each time. To control costs, we switched the environments on and off automatically each day. However, we added flexibility by encouraging teams to email if they needed access at different times. We could then extend the time, or change the opening hours. The use of automated switching made life easier for the support team, and meant that we had time to provide the additional flexibility needed.   Self-service analytics provides multiple ‘wins’ for both users and organisations For part of the teams we opted for a self-service environment for many reasons. The first was pragmatic: with just a small team providing support, it was the most viable option to allow teams to do as much as possible for themselves. Analysts also often expect self-service, so it makes sense. However, another nice quick win from the self-service approach is self-education. We effectively gave hackathon participants a hands-on lesson in how a SAS Viya installation on an Azure Kubernetes Service cluster works. Self-service therefore provides faster learning—and so faster deployment—than other analytics environments. It also has the benefit that user acceptance is higher.   Automation, like self-service, provides several benefits for both users and organisations I mentioned the use of automatic switching to turn the environment on and off. We chose to automate several aspects of the environment for the hackathon, for several reasons. First because self-service tools for provisioning architecture are impossible without automation. We needed to be able to provide the environment without placing a big burden on IT operations. We also needed it to be both agile and efficient. Automation, especially with the cloud environment, offered both those aspects.   Speed of deployment can also be a crucial aspect of analytics architecture One of the key lessons for us—and for any architectures of analytics environments—was about the ease of providing SAS Viya sandboxes in the cloud. This is essential for a hackathon, where users need to get up and running fast. However, it would also be helpful to organisations or teams setting up analytical ‘sprints’ or demonstrations. It could therefore be used for conferences and events, or for students—or, indeed, many other uses across a wide range of organisations and sectors. I think one of the most useful aspects of supporting a hackathon is that we, as architects, can try new things. As my colleague James Ochiai-Brown commented on Twitter, “Our experience shows it is possible to set up self-service provisioning for the latest version of SAS Viya on Kubernetes. It really works!” At the SAS Global Hackathon, it was certainly not just the competitors who were experimenting or learning. ... View more

SAS is sponsoring a global hackathon, #HackinSAS. The event is designed to explore how to use data for social good in new and creative ways. I won’t go into details on the hackathon in this article, but I highly encourage you to refer to the #HackinSAS page as well as the Hacker’s Hub on the SAS Communities for more information. What I do want to cover in this post is how we went about creating the environment we’ll provide for each team participating in the hackathon.   Article Mission Introduce the possibilities Azure DevOps Pipelines bring in the context of automating SAS Viya deployments.   Pre-requisites This article relies heavily on two separate sassoftware GitHub repositories: SAS Viya 4 IaC for Azure SAS Viya 4 Deployment   These repositories contain tools, scripts, and code for SAS Viya deployment in an Azure environment. I highly recommend becoming familiar with these concepts.   Foundations I have been using the powerful SAS Viya Infrastructure as Code (IaC) repository (referenced above) to create the necessary Azure compute resources for a SAS Viya deployment.  The project contains Terraform scripts to provision Microsoft Azure Cloud infrastructure resources required to deploy SAS Viya 4 products. After the Infrastructure is created, I deploy SAS Viya with the code in the other referenced repository.     Azure DevOps pipelines The Azure infrastructure for #HackinSAS requires environments for 100 teams. Each team needs access to his own SAS Viya environment.  We are currently researching how to provide all that infrastructure in the most agile and efficient way, but I decided to start looking at Azure DevOps pipelines to automate this process as much as possible.   Azure Pipelines combine continuous integration (CI) and continuous delivery (CD) to constantly and consistently test and build your code and ship it to any target.  Consider it an automation server, similar to Jenkins.  My goal is to share with you how you can integrate the Viya4 IaC Terraform & Viya 4 deployment scripts in these pipelines. This requires some familiarity with automation servers and the SAS Viya IaC project.   The graphic below represents the entire deployment process explained. I’ll refer to it in the Pipeline initialization section.     Pipeline initialization Consider the first DevOps pipeline the initialization process before we can kick off the main pipeline.  It first creates a container to store the terraform.tfstate file within an existing Azure Storage Account.  Each participating team will have its unique terraform.tfstate file because it contains all the information about the Azure Cloud Infrastructure created for that team. The creation of that storage container is the first task of the pipeline.    The second task of the starters pipeline is to build a Docker Container Image, tag it with the name of the team and push it to an Azure Container Registry (step 3). This Docker image runs a few times in the second pipeline when we plan and apply the Terraform plan. Here are the Docker commands executed in the first pipeline to prepare the Docker image:   docker build -t viya4-iac-azure-hack:$(team) . docker tag viya4-iac-azure-hack:$(team) $(location).azurecr.io/viya4-iac-azure-hack:$(team) docker push $(location).azurecr.io/viya4-iac-azure-hack:$(team)   As you can see, we are making use of variables to identify team and locations.   The pipeline can't start without these required parameters. The Dockerfile (represented bellow) in use for the container image build is the same as what is found in the SAS IaC Github project. Note that one additional parameter,  ARM_ACCESS_KEY is required. This value provides access to the Azure Storage Account.   I also modified the Terraform init command by adding some backend parameters of the location of the terraform.tfstate file.  During the pipeline execution, the team string is replaced with the name of the hackathon team.   ... ENV ARM_ACCESS_KEY=__storagekey__ RUN apk --update --no-cache add git openssh \ && terraform init -backend-config="key=__team__-terraform.tfstate" -backend-config="container_name=__team__" /viya4-iac-azure ...   To get this working you also need to add the rest of the backend details to the main.tf of the Terraform scripts.   terraform { backend "azurerm" { resource_group_name = "tstate" storage_account_name = "tstatehackathon" } }   The pipeline execution (step 4) normally finishes in around two minutes. Below is a representation of the results of the pipeline execution.     Main pipeline Now it's time to do the bulk of the work which is setting up an Azure AKS Kubernetes cluster (step 5) and deploy Viya.  We set up the AKS cluster with the Docker image we pushed before to the Azure Container Registry. To deploy SAS Viya we make use of another very handy project on GitHub, SAS Viya4 deployment.   Just as with the first pipeline the idea is to check-in all necessary & parameterized files in an Azure DevOps project with an associated Azure Repos Git. azure_docker_creds.env terraform.tfvars ansible-vars-iac-azure.yaml TLS certificates   The first file contains the authentication information we will source during a specific stage in the pipeline.  It allow the Azure DevOps pipeline to authenticate Terraform to access our Azure subscription as described in the instructions for the SAS Viya 4 IaC for Azure project. Again, we need to add an additional variable to the azure_docker_creds.env file.  The ARM_ACCESS_KEY provides access the backend Azure storage so the Terraform state file can be saved in there.  Below is a representation of the file contents.   TF_VAR_subscription_id="00000000-0000-0000-0000-000000000000" TF_VAR_tenant_id="00000000-0000-0000-0000-000000000000" TF_VAR_client_id="00000000-0000-0000-0000-000000000000" TF_VAR_client_secret="00000000-0000-0000-0000-000000000000" ARM_ACCESS_KEY=__storagekey__   The pipeline queries the value of the storagekey with the appropriate credentials during the run.  In the terraform.tfvars and the ansible-vars-iac-azure.yaml I've added a few parameters to replace values during the run.   Below is a representation of the files.   terraform.tfvars # **************** REQUIRED VARIABLES **************** # These required variables' values MUST be provided by the User prefix = "hackinsas-__team__" location = "__location__" # e.g., "eastus2" # **************** REQUIRED VARIABLES ****************   ansible-vars-iac-azure.yaml ## Cluster NAMESPACE: hackinsas-__team__ .... JUMP_SVR_HOST: __jmpip__ JUMP_SVR_USER: jumpuser JUMP_SVR_PRIVATE_KEY: /data/auto-private-key-__team__.pem ... ## Ingress V4_CFG_INGRESS_TYPE: ingress V4_CFG_INGRESS_FQDN: __team__-hackinsas.vectorlabs.sas.com V4_CFG_TLS_MODE: full-stack # [full-stack|front-door|disabled] V4_CFG_TLS_CERT: /data/vector.crt V4_CFG_TLS_KEY: /data/vector.key V4_CFG_TLS_TRUSTED_CA_CERTS: /data/vector-cacerts.crt ..   With all of this in place, I kick off the Azure DevOps pipeline. This process takes about 30 minutes to complete.  During this time the setup of the infrastructure and the SAS Viya deployment is done automatically for me. This allows me to quickly spin up the necessary environments as teams are enrolling for the hackathon.  Here (graphic below) I start the pipeline manually but it can also be kicked off by using specific triggers.       When the execution completes, the pipeline creates an AKS cluster with six nodepools and kicks off a full SAS Viya deployment. About an hour later you can start working on the environment.   Details of the AKS cluster System node Pool default_nodepool_vm_type = "Standard_D4_v2" Node Pool Vm_type Min nodes Max Nodes system D4_v2 2 2 CAS E16ds_v4 1 1 compute E16ds_v4 1 1 connect E8ds_v4 1 1 stateless E16ds_v4 1 2 stateful E16ds_v4 1 3   As you can see from the screenshot below, I split the work in the pipeline over different stages and jobs.  Accept the challenge to provide more structure and better naming conventions 😊.  I also added the option to destroy the IaC and clean all the Azure resources when the Hackathon is finished.     And if you checked the details of the image below, you might see I'm publishing two artifacts.  The build artifact is a build-team.zip file that contains all the manifests, kustomization.yaml and site-config folder of the deployment.  The drop folder has the necessary files to connect to the Kubernetes cluster and Jump server created by the IaC terraform scripts. Finally, with that I can access and customize the SAS Viya environment if required.       Finally If this all is a bit abstract for you, I recorded a video demonstrating the Azure DevOps pipeline.  So, yes, we are getting ready for the Hackathon! ... View more