When most of us think about ‘IT support’ or ‘IT admin’, we think about the person who answers the phone when our work computer goes wrong. Indeed, the phrase that most of us are most likely to associate with IT support is “Have you tried turning it off and turning it on again?”   This view, however, seriously underestimates the importance of IT support teams. Far from simple troubleshooters, they are architects, managers and coordinators, responsible for decisions about how to keep vital applications running. They have to decide when and how to upgrade IT systems to ensure that organisations can run smoothly and efficiently—and all without disrupting the day-to-day operations. This is crucial in any business, but perhaps even more so in banking and financial services, where customers require both good security and instant access to their accounts.   A complex environment   Raiffeisen Banking Group is Austria’s largest banking group, with around half of all Austrians as its customers. Bernhard Lainer is SAS Application Manager at Raiffeisen Informatik, and leads a small team supporting SAS users at the Raiffeisen banks and UNIQA Insurance. His team is in turn supported by a team of SAS consultants on a contract basis which help managing standard administrative tasks. The banking group uses a wide range of SAS products, including Credit Risk, Market Risk and Risk Management for Banking, on both SAS 9.4 and SAS Viya environments.   As the SAS Application Manager, Bernhard has to support several hundred SAS users across the many subsidiaries of the Raiffeisen banking group in Austria, such as the Raiffeisen Banken Niederösterreich-Wien, Burgenland, Vorarlberg and the UNIQA Insurance. He notes that the IT landscape is quite complex, even when you only take account of the SAS elements.   “We still have various SAS 9.4 environments on AIX, zOS and Linux. We also have a SAS Viya 3.5 cluster, but this is being replaced by SAS Viya 4 on Kubernetes. We are currently in the process of migration, which we’re doing in stages because not all solutions are available on Viya 4. We started with the reporting system, so that we can use the latest versions of Visual Analytics.”   Raiffeisen Informatik has a clear strategy of consolidating their huge SAS infrastructure into SAS Viya on Kubernetes to reduce maintenance efforts and streamline administration tasks. These SAS Viya environments are run on a Red Hat OpenShift platform. Lucas Kranawetter is the Product Owner of the OpenShift platform at Raiffeisen Informatik. He comments,   “Red Hat OpenShift was chosen for SAS Viya 4 because there was already a team that takes care of the OpenShift clusters and keeps them up to date. We were one of the first providers in Austria to introduce OpenShift back in 2016. Productive applications have been running on this Kubernetes stack since 2018. We run OpenShift on an on-premise virtualised infrastructure.”   Lucas adds that the group’s strategy is to be infrastructure-neutral wherever possible, because this gives maximum flexibility both now and in future.   “This helps us to be able to react more quickly to changing market situations and new orders and to have to reserve fewer resources for high availability.”    Managing the SAS Viya environments   IT support teams have to consider how best to manage updates and upgrades. Bernhard is clear about how and why these decisions are taken for SAS products.   “We decided in favour of monthly STABLE releases so that we are always up to date and have all the new features. I think it is generally better to install several smaller updates rather than a single major changeover each year. This spreads the effort evenly over the year and improves security. However, I’ve noticed that unlike SAS 9, we have virtually no vulnerabilities in Viya 4 because the Viya services are updated much more frequently.”   Bernhard notes that one of the key differences between SAS Viya on Kubernetes and the older environments is the level of automation. He comments that there are far more opportunities to automate processes using a much wider range of tools in the newer environment:   “A lot of the support tasks are now automated. For example, authorisations are controlled by the customer in a self-service manner. I don’t have to do anything here, apart from explaining to new users every now and then that they can request this themselves. I still deploy the reports from the Test to the Prod environment manually but I am planning to automate this soon as well.”   Likewise, he was able to automate the software deployment and update process as well by using the SAS Deployment Operator and the sas-orchestration utility, which are optional components for managing the SAS Viya platform. Using these tools reduces the work needed for updating a release significantly. Lucas adds some details about the SAS Viya environments from a Kubernetes perspective:   “The SAS Viya environments currently run on several shared clusters together with other customer applications. However, we maintain exclusive resources for environments to avoid interference and other disruptions. We use ResourceQuotas at namespace level for this. For the future, however, we are considering dedicated clusters for SAS Viya, as we will then be able to better harmonise our operating activities on the platform, such as updates or similar, with the SAS environment.”   It's important to mention that automation also helps improving the way how Bernhard and Lucas can work together:   “Initially, there was a lively exchange and a good introduction to the world of containers given to me by our OpenShift administrators. Now, with a new deployment, the OpenShift administrator only has to update the Deployment Operator, which then executes the remaining tasks for me with extended rights. This is an advantage for me as I don't always have to ask for additional rights and also an advantage for the OpenShift administrators and security who don't have to give me all the rights straight away.”   Without doubt, this is one of the major reasons that make it possible to manage an infrastructure as large and complex as the SAS Viya environments at Raiffeisen Informatik by Bernhard’s small team.   A changing view   Bernhard notes that the high level of automation means that his job is changing from traditional ‘IT support’ to a much more customer-focused role.   “Far more of my job now is about communicating with customers. For example, we upgrade the STABLE release every two to three months, which isn’t a huge job. I then have to look at the new features and communicate these to the customers. I also spend a lot more time on further development and optimisation, working directly with customers.”   For him, the step towards SAS Viya on Kubernetes has come with many improvements that make his everyday working life easier:   “No more Linux updates, no more security vulnerabilities that I have to deal with, no more discussions with Linux admin and security about which extra tools you need, the data is no longer stored on the server but on a database or a NAS share, starting/stopping of services is much faster and so on …”   As he describes it, the move to SAS Viya on Kubernetes has been a journey for him, which is still ongoing:   “I started my journey 2,5 years ago with no prior experience of Kubernetes. Kubernetes requires a different kind of thinking and especially automation becomes a key principle. While the architecture of the SAS platform might have changed, at the core, it‘s still the same SAS. So far, it has been a steep, but rewarding learning curve.”   As a conclusion, the advice he would give other SAS administrators who find themselves at a similar starting point in their journey to SAS Viya is to have faith in their own expertise.  Adapting to new tools and capabilities can feel challenging, but we have all done this before, sometimes several times.   Learn more   Bernhard’s recent presentation at the SAS Innovate conference in Las Vegas is available for a more in-depth look at his team’s experience.   ... View more

  SAS Container Runtime (SCR) provides a way to publish your analytical models to any environment which is able to run OCI compliant Docker containers. For example, you could deploy a model to a managed cloud service such as AWS Elastic Container Service (ECS) or Azure Container Apps or to just any Kubernetes cluster running either in the cloud or on-premises. The SCR images generated by SAS Model Manager are free from any external dependencies (except from requiring a container runtime environment of course) and can be accessed via standardized REST API calls.   While there is a lot of documentation describing the process of building and deploying a SCR model container (see here and here for blogs discussing a ModelOps approach), there is not much material discussing what happens after you’ve successfully deployed your model. Which is, after all, just one step in your model’s lifecycle and not the end of it.   In this blog I’d like to focus on how to enable auditing and model governance specifically for models deployed as SCR images. The main challenge with this task is that you often need a way to trace your model’s actions and decisions. Let’s see how this can be achieved …     A little bit of background   It has probably become common knowledge in the analytics space that deploying a model to a production environment is not the final state in the model’s lifecycle. Rather, it’s crucial to keep track of the model’s performance over time to detect when the accuracy of the predictions it generates have degraded below a threshold where you want to retrain or even retire the model. Besides of monitoring the model drift, you might have other reasons to track your model as well. For example, you might be required to store a record of all decisions the model has taken for auditing purposes. Note that in both cases you probably need the full record, i.e. the metadata (who called at which time etc.) and the data (which input data was sent and which probability score was returned).   If you’re in the lucky situation of having full control over the score client which communicates with the model container, you could simply log all scoring requests and responses on the client side. But what if this is not the case? If you need to do the logging on the server side? Well, as long as you’re within the SAS infrastructure, this isn’t a challenging task – it’s a built-in feature of the publishing destinations for CAS and MAS (note that you have to perform a custom configuration step to enable this feature for MAS).   However, the picture changes once you leave the SAS-managed infrastructure: a SCR model container is fully autonomous and can be deployed to environments without having any other SAS platform component within reach. Even “worse” (so to say): the model container is stateless, which means that it’s not writing any information to a persistent volume for example.   As a DevOps person, you certainly could follow your primal instincts and crack the container image open to add your own layers to it, but that’s not exactly a recommended approach (to put it nicely). Luckily there are better solutions available. Let’s discuss a simple approach first, followed by a more sophisticated one.     A built-in solution: SAS_SCR_LOG_LEVEL_SCR_IO   Being quite a mouthful, SAS_SCR_LOG_LEVEL_SCR_IO is one of the SCR core loggers. Out-of-the-box it’s more or less muted but can be enabled by setting the appropriate log level. When set to TRACE, it will make the model container log the input and output values for every request it receives.   For Kubernetes, this logger is activated by setting an environment variable in the Pod manifest, for example like this (abbreviated for clarity):   apiVersion: apps/v1 kind: Deployment metadata:   name: my-scr-scoring-pod spec:   template:     spec:       containers:       - name: my-scr-scoring-model         env:         - name: SAS_SCR_LOG_LEVEL_SCR_IO           value: TRACE   Once activated, your container will start printing out lines like this one to the standard Kubernetes log (abbreviated and formatted for clarity):   2023-11-06 17:21:06.724 DEBUG --- [nio-8080-exec-4] SCR_IO  stud01_steel score : Executed module step my_model->score: {   "version" : 1,      "inputs" : [      { "name" : "X_Minimum", "value" : 1498 },      { "name" : "X_Maximum", "value" : 1150 }, (...)      ]  } -> {  "metadata" : { "module_id" : "my_model", ... },   "version" : 1,   "outputs" : [   { "name" : "EM_EVENTPROBABILITY", "value" : 0.15030613001686055 },   { "name" : "EM_CLASSIFICATION", "value" : "           0" }, (...)   ] }   As you can see, you’ll get one line for each scoring request which the container has received, along with a timestamp and the generated response (probability). While this is basically all you need, it comes with potential disadvantages which might prevent you from using this approach. One important gotcha to consider is that you might disclose sensitive information to a log which is not under your control: the information is collected and processed by a cluster-wide logging subsystem and might be surfaced at places you don’t want it to appear (say, a web app like Kibana/OpenSearch etc.). Additionally, you still need to find a way to send back that information to SAS, as you need to ingest it for tracking your model’s performance.     Introducing the sidecar container pattern   Let’s take a look at the second approach. On Kubernetes, our models will be deployed as Pods, which is a wrapper object for one or more containers. By default, the model runtime will be the only container of the model Pod, but that’s not carved in stone. We can add a sidecar container running a proxy web server which can “intercept” the traffic directed to the model container. This allows us to capture the inbound and outbound data as it passes by and write it to a log on a shared volume. So we’re bypassing the standard Kubernetes logging facility - but only for the information we’re interested in.   The use of “sidecar” containers is a common deployment pattern which has been around since the early days of Kubernetes. The main purpose of a sidecar is to add supporting features to the main container in a pod without the need of modifying it. By using sidecars, you can avoid having to rebuild your main container when it lacks a feature you need.   The sidecar container(s) and the main container share the pod storage and network, which means that a sidecar can use “localhost” (127.0.0.1) to communicate with the main container. Thanks to this close relationship, sidecar containers are quite often used as proxies of some kind and this is exactly the very reason why we’re using them in this blog.     A custom solution: the web proxy server sidecar   Take a look at this diagram to better understand the basic idea:         Starting on the top left, there’s a client submitting a scoring request to a model instance deployed as SCR. As you can see, there are actually 2 containers deployed as one “model-pod”: the “logproxy” sidecar and the main model container, i.e. their manifest roughly looks like this:   apiVersion: apps/v1 kind: Deployment spec:   template:     spec:      volumes:       - name: shared-volume (...)       containers:       - name: logproxy-sidecar         image: ...         ports:         - containerPort: 8081         volumeMounts:         - name: shared-volume (...)       - name: score-app-main         image: ...         ports:         - containerPort: 8080   (BTW – I have attached the full deployment manifests as a ZIP file to this blog). The service object in front of the pod redirects incoming traffic to port 8081 which is the port of the sidecar container.   apiVersion: v1 kind: Service spec:   ports:   - name: http     protocol: TCP     port: 80     targetPort: 8081   The sidecar container (which is basically just a web server configured as a proxy) passes the request to the main model container using port 8080. Once the calculation is done, the response is sent back to the client on the same route, giving the sidecar a chance to write out a log record to the persistent volume.   Obviously, this persistent volume should be shared with SAS Viya so that you can set up a scheduled SAS compute job which will read the log records and add them to a CAS table for example (maybe for building a custom Visual Analytics dashboard or for feeding the model performance report in SAS Model Manager).   With the basic flow covered, let’s take a deeper look at that mysterious “logproxy” thingy …     The logproxy configuration   I hope you won’t be disappointed when learning that the sidecar container (in this blog at least) is a simple nginx server with the Lua scripting module enabled. Sorry, no magic here …   You might already be familiar with the nginx web server as it is commonly used as the default ingress controller for many Kubernetes distributions. It’s easy to package nginx into a container image and it’s pretty lightweight as well. The nginx functionality can be extended and the Lua module is the usual approach for embedding scripting capabilities to nginx. If you’re not familiar with Lua – don’t worry too much: the snippets which you need to understand are fairly simple. They are needed for transforming the request and response data into a clean output format which can be easily consumed later (by SAS for example).   It's not required (but certainly possible) to build the Nginx+Lua container image from scratch. There are a couple of open-source solutions available to choose from. For example, I tested the openresty and the nginx-lua images and they both work nicely. In the end, I decided to use the nginx-lua image as it allowed me to run the sidecar as a non-root user, but the decision is of course up to you.   I also decided that I want the sidecar to generate the log records in the JSON format, so my code snippets are written for that purpose. The log output generated by the sidecar is supposed to look like that (as usual, abbreviated and formatted for clarity):   {     "timestamp": "29/Nov/2023:10:23:58 +0000",     "req_id": "805054d675e19c17d31b1b04056b7351",     "request": {         "inputs": [             { "name": "CLAGE", "value": 94.36666667 },             { "name": "CLNO",  "value": 9           }, (...)         ]     },     "response": {         "metadata": {             "module_id": "hmeq_score",             "elapsed_nanos": 513119,             "step_id": "score",             "timestamp": "2023-11-29T10:23:58.131916257Z"         },         "version": 1,         "outputs": [             { "name": "P_BAD1", "value": 0.6131944552855646 },             { "name": "EM_PROBABILITY", "value": 0.6131944552855646 }, (...)         ]     } }   You can easily see the input and output data along with some metadata like a unique request ID, the model name and more. In order to create this output, some Lua code is required which can be executed by nginx for each scoring request. These code snippets can be stored in a custom configuration file which is read by nginx during startup. I used the following snippet for generating the above log records:   log_format scorelog escape=none '{ '   '"timestamp":"$time_local", '   '"req_id":   "$request_id", '   '"request":  $req_body, '   '"response": $resp_body '   '},'; upstream webapp {     server 127.0.0.1:8080; } server {   listen 8081;   location /score {       set $req_body '';       set $resp_body '';       access_by_lua_block  {         ngx.req.read_body()         ngx.var.req_body = ngx.req.get_body_data() or "{}"         if not ngx.var.req_body:match "^{" then ngx.var.req_body = "{}" end       }       body_filter_by_lua_block {         ngx.var.resp_body = (ngx.var.resp_body or "") .. ngx.arg[1]       }       proxy_pass         http://webapp;       proxy_redirect     off;       access_log /var/log/nginx/shared-data/model-logs/log-$server_addr.json scorelog;   } }   I can’t discuss all the details here, so I’ll focus on 3 important points: access_by_lua_block and body_filter_by_lua_block are predefined “directives” you can use to execute code at a specific point in time. access_by_lua_block is called first to process the incoming request, followed by body_filter_by_lua_block which is called when nginx has access to the response returned by the main model container (see here for a more detailed explanation). Note the definition of the upstream webapp as 127.0.0.1:8080 – this is actually the port where the main model container listens to. As said above, a sidecar shares the pod storage and network, so it can just use “localhost” (127.0.0.1) to reach the container “next door”. Finally note the output storage location: /var/log/nginx/shared-data/model-logs is the mount point I’m using for the persistent volume. I’m also adding the IP address of the model container to the file name: log-$server_addr.json. The reason for this is that attributes such as the model ID or the date would not be unique in case you have deployed multiple replicas of the SCR pod.   Closing the loop – if this configuration is stored in a ConfigMap, you can “mount” it to the logproxy sidecar to make it available to the nginx server:   apiVersion: v1 kind: ConfigMap metadata:   name: scoreapp-logproxy-nginx-conf   namespace: default data:   nginx-logproxy.conf: |-     log_format scorelog escape=none '{ '       '"timestamp":"$time_local", '       '"req_id":   "$request_id", ' (...) --- apiVersion: apps/v1 kind: Deployment spec:   template:     spec:       volumes:       - name: nginx-proxy-config         configMap:           name: scoreapp-logproxy-nginx-conf       containers:       - name: logproxy-sidecar         volumeMounts:         - name: nginx-proxy-config           mountPath: /etc/nginx/conf.d/nginx-logproxy.conf           subPath: nginx-logproxy.conf (...)   Again, remember that I have attached the complete manifests to this blog.     Conclusion   Before closing this blog, I’d like to mention a caveat concerning log rotation. In Kubernetes you’re usually not bothered with this because it is handled for you automatically as long as the pods are using the default stdout destination for their log output.   Unfortunately, that’s not the case with the sidecar container, so there might be a risk that the log files exceed the available disk space at some point. If you choose nginx as your web proxy (like I did in this blog), you should be aware that it doesn’t have native log rotation capabilities and relies on external tools like logrotate, which might not be available in the nginx docker image you’re using. In other words: you might need to add another sidecar container with logrotate to manage your log files volume.   I hope you liked the blog and I’m curious to hear your feedback. Please feel to send in your thoughts and comments!   ... View more

Introduction   The Red Hat OpenShift Container Platform (OCP) is a popular choice for many SAS customers running SAS Viya in a private cloud infrastructure. OpenShift not only offers a hardened, enterprise-ready Kubernetes distribution but also comes with a rich ecosystem of 3 rd -party applications and extensions to the core platform.   In this blog I’d like to discuss how tools taken from this ecosystem can be used for automating the delivery process for analytical models developed with SAS Viya. Unlike other blogs, which have focused on model training and model governance aspects, this blog discusses the steps needed to make analytical models available to applications (or users) for consumption. Typically, “model serving” topics rather belong to the domain of DevOps engineers, not data scientists.   Why at all should you bother about automating the lifecycle management of your models? It has become clear over the past years that decisions, driven by analytical models, have become more and more relevant to everyday business processes. We’ve not only seen a heavy increase in the sheer number of analytical models, but also in the frequency in which these models need to be revised and updated. Automating these processes is key to keep the pace without losing stability and robustness.   This blog describes an example setup for model automation on the OCP platform, using the capabilities of SAS Viya in conjunction with tools taken from the OCP ecosystem: OpenShift Pipelines (Tekton) and OpenShift GitOps (ArgoCD). When used in combination, these apps form the CI/CD (“Continuous Integration / Continuous Delivery”) toolchain which typically powers all DevOps systems.   This diagram shows the major steps which will be discussed in this blog: Figure 1    The blog focuses on the steps shown in the center and on the right-hand side of the picture – the preparation and deployment of analytical models trained on the SAS Viya platform. The process steps shown above are actually “distribution-agnostic” - however this blog is “distribution-aware”. You could replace a tool in the CI/CD toolchain with another (e.g. Tekton with Jenkins) without changing the process flow, but there are distribution-specific specialties to be considered when choosing OpenShift as your model delivery platform.     Infrastructure overview   Figure 2 should help to get a better understanding of the infrastructure. Like most “*opsy” architectures, the infrastructure we must set up for ModelOps automation might look confusing at the beginning, but it will soon start making sense hopefully.   Figure 2   The typical automation flow is shown on the horizontal axis from training (left) to serving models (right), while on a vertical axis the picture shows the key “systems” starting from users and apps (i.e. producers and consumers), followed by the Kubernetes infrastructure and the external repositories providing the persistence layer required for connecting the automation tools.   Next, we have the CI/CD applications. As stated before, we’re using built-in services which are part of OpenShift’s DevOps ecosystem. SAS Model Manager does not directly interact with these tools. While its’ model repository is used for storing model metadata and tracking model performance over time, Model Manager can push the model to a Git server and to a Container registry (at the same time). This is an important capability required for supporting an automation workflow as this push event can trigger activity in the CI/CD applications. The Git repository keeps the model source code under revision control while the Container Registry stores the binary execution runtime (SCR container image).   SAS Container Runtime (SCR) For this blog we’ll be using the SAS Container Runtime (SCR) framework to create the model container images. SCR is a native capability available on the SAS Viya platform and part of SAS Model Manager. It’s used when publishing an analytical model (written in SAS or in Python) to a container registry destination: a lightweight Open Container Initiative (OCI) compliant container is created on the fly which is the execution environment for your model. Each SCR image contains a single model or decision. SCR images are self-contained (no dependencies on SAS Viya platform services) and highly scalable.   In our case, OpenShift Pipelines acts on the push event to the Git repository and executes a workflow labeled as the “model serving pipeline”. It analyzes the model metadata pushed to Git and creates a deployment manifest for the model which it then commits back to the same Git repository. Why is this step important? It decouples the training layer from the serving layer: SAS Model Manager does not need to know the details of the execution infrastructure of the model. A DevOps engineer can easily modify the pipeline without interfering with the data scientist’s work.   Once the model’s deployment manifest has been pushed back to Git, it is picked up by OpenShift GitOps. This tool then takes responsibility for syncing the new content to the target namespace(s) – simply put it deploys the new model as a Kubernetes pod and adds the required Service and Route to make it accessible.   Once a model has been deployed to the model serving layer, it can finally be used by consumers (apps) sending requests to the deployed models at runtime. For real-time serving scenarios it’s a common practice that model containers (such as SAS SCR container images) offer a REST API to external apps.     Infrastructure setup   SAS Viya   Deploying SAS Viya on OCP requires a few steps which are not needed for other Kubernetes distributions, mostly to take account for OpenShift’s additional security layers and its’ use of the HAProxy ingress controller. Check the sections about OpenShift in the SAS Viya Operations Guide for more details.   For our scenario you need to make sure to include a specific transformer patch to the deployment, which is required for enabling the model publishing destination we need. See the instructions found in $deploy/sas-bases/examples/sas-model-publish/kaniko/README.md for more information and pay attention to the additional security requirement. SAS uses the kaniko tool to create the SCR container images on the fly.   Kaniko Kaniko is an open-source project originally developed by Google. It’s a tool to build container images from a Dockerfile and runs inside a Kubernetes cluster. SAS has packaged it as a microservice and deploys kaniko as part of the SAS Viya software stack.   kaniko cannot run on OpenShift using the default restricted Security Context Constraint (SCC) but needs elevated permissions (check this Git issue for some additional background). For that reason, you must bind the anyuid SCC to the sas-model-publish-kaniko service account:   $ oc -n <name-of-namespace> adm policy add-scc-to-user anyuid -z sas-model-publish-kaniko   OpenShift Pipelines   The Pipelines app can easily be installed via OpenShift’s OperatorHub using the default settings. I’d also recommend installing the Tekton CLI to manage the pipelines. The main building blocks of Tekton pipelines are Tasks and the Tekton project offers a repository of pre-defined Tasks to choose from. We need two of them for the pipeline we’re about to deploy:   $ kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/git-clone/0.9/git-clone.yaml $ kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/git-cli/0.4/git-cli.yaml   OpenShift GitOps   Like before, the OperatorHub provides the easiest way to deploy the GitOps tool. You need to grant cluster-admin permissions to the openshift-gitops-argocd-application-controller service account to allow it to run deployments on your behalf:   $ oc -n openshift-gitops adm policy add-cluster-role-to-user cluster-admin \     -z openshift-gitops-argocd-application-controller   And again, I’d recommend that you install the ArgoCD CLI for managing the environment.   Git server   Prepare a Git repository You should prepare an (empty) Git repository which can be used for storing your model assets. I called mine sas-model-repository, but any name will do. There are 2 approaches to choose from:   have one Git repository per model (1:1) have one Git repository for all models (1:n)   For this blog I chose the 2 nd approach, but this is certainly not a requirement. When creating the Git repository, make sure to add a folder to keep the generated deployment manifests. I called it deploy-manifest in my system. Here’s a screenshot showing how this repository could look like (note that it already contains two models):   Figure 3   Create a Git webhook You need to define a webhook connecting the Git repository with the CI pipeline. A simple setup like this would be sufficient (all push events, all branches). The webhook URL needs to match the address used for the HAProxy route when defining the pipeline’s event listener (see below).   Figure 4   OpenShift cluster   Automation account authentication If you’re planning to use model validation tests in SAS Model Manager (not covered in this blog), you must ensure that the automation account used by SAS can deploy pods to the validation namespace. SAS authenticates to OpenShift using a client certificate which needs appropriate permissions for the target namespace.   Allow image pulls from your registry Depending on your system, you might need to import your container registry’s CA certificates to OpenShift to allow ImagePulls. For example, the following commands add my registry’s CA certificates to OpenShift:   $ oc -n openshift-config create configmap registry-ca-certs \   --from-file=harbor-registry.myspace.com=/tmp/certs/harborCA.crt $ oc patch image.config.openshift.io/cluster \   --patch '{"spec":{"additionalTrustedCA":{"name":"registry-ca-certs"}}}' \   --type=merge     Connecting the bits and pieces together   With the infrastructure in place, it’s time to discuss the configuration needed to set the process in motion. Let’s start with defining the model publish destination in SAS Model Manager and then proceed to the model serving pipeline (Tekton) and the ArgoCD sync app.   Configuring the model publish destination   A publishing destination is a concept used by SAS Model Manager. It provides a modular way of defining target environments into which models can be pushed to, for example Azure Machine Learning, Apache Hadoop, Teradata, Git and many more.   In our case, the main publishing destination will be a Container Registry, but we’ll also use a Git repository which is hooked into the CI/CD pipeline. When publishing a model to a Container Registry, SAS Model Manager will automatically create a container image for the model using the kaniko tool. Publishing destinations can be created by using the SAS Viya CLI. The command to create a Container Registry destination is rather complex, so let’s break it down to its’ individual sections:   Figure 5   Let me explain the “weird” lines at the end of the command first – this is known in Linux as a “here” document. It’s a useful trick to run a command in batch-mode which expects interactive input. The sas-viya CLI plugin for creating publishing destinations automatically creates a credentials domain and will prompt you for the name. The “here” document provides that name (“ContainerRegistryPublishingDomain”) and thus saves you from typing it in.   Model validation You might wonder about the group of settings related to Kubernetes in the command shown above. This refers to a Kubernetes namespace which can be used for running a validation test on your model. This check can be triggered from the Model Manager user interface once the model container image has been published to the container registry. It’s an optional feature and if preferred it’s certainly possible to set up validation checks as part of the automation flow instead.     Setting up the model serving pipeline   Let’s take a look at the Tekton pipeline. Tekton is fully integrated into Kubernetes and extends the Kubernetes API with components needed for defining pipelines. I’ve already mentioned the use of Tasks, but there is a lot more. Again, take a look at the code snippets I have provided in the attached ZIP file to see how a sample pipeline could look like. In my simplified setup, these Tasks will perform the following steps:   Clone the contents from the Git model repository to an internal workspace Create deployment manifests for all models found in the Git repository Commit the new manifests back to the Git repository   Once the pipeline is set up, you need to define an EventListener so that it can be triggered by a Git push event. Finally, the listener needs to be exposed using a Service and a Route (remember to make the Git webhook point to the Route URL).   You can either use the Tekton CLI or the OpenShift web interface to review your pipeline:   Figure 6       Setting up the ArgoCD syncing app   Compared to the Tekton pipeline, the ArgoCD app is actually a lot simpler to create. The following code snippets use the ArgoCD CLI for this task:   # switch to the right namespace $ oc project openshift-gitops # Add repository credentials to ArgoCD (for Git login credentials) $ argocd repo add https://my-gitserver.server.com/my-model-repository.git \     --username <user1> --password <password1> # Create application $ argocd app create sas-modelops --insecure \     --repo https://my-gitserver.server.com/my-model-repository.git \     --dest-namespace sas-modelops-deployments \     --dest-server https://kubernetes.default.svc \     --path deploy-manifest --sync-policy automated \     --revision main --sync-option Prune=true   Note that the ArgoCD app is exclusively monitoring the “deploy-manifest” folder in the Git repository – we only want the app to wake up after the Tekton model serving pipeline has committed new deployment manifests to this folder. Here’s a screenshot of the sync app taken from the OpenShift web console:   Figure 7       Testing the configuration   Now that the environment has been fully configured, it’s time to do some testing. I would suggest to take these steps:   Start in Model Studio by creating a simple pipeline project. Once the pipeline has completed, click on “Register” to transfer the model to Model Manager. In Model Manager, highlight the new model, click on “Publish” and select the Container Registry destination. Now move over to the Pipelines user interface in the OpenShift web console - if you’re quick enough you should be able to see the pipeline executing. Finally, when switching to the ArgoCD user interface you should see that the sync app starts running right after the pipeline has finished. Send data to the model to test that it’s working. You could use PostMan for this or simply run the curl utility:   $ curl --location --request POST 'https://mymodel.apps.openshift.cluster.com/model' \     --header 'Content-Type: application/json' \     --data-raw ' {          "CLAGE"  : 100,          "CLNO"   : 20,          "DEBTINC": 20,          "REASON" : "REFINANCE", ...          "VALUE"  : 1000000,           "YOJ"    : 10    }' | jq .     Conclusion   In this blog I discussed how OpenShift can be leveraged as a model serving platform for analytical models created by SAS Viya. Using the CI/CD tools provided by the OpenShift ecosystem - OpenShift GitOps and OpenShift Pipelines - in conjunction with SAS Viya provide a robust way to automate the lifecycle management of models.   I hope you enjoyed reading this blog and please feel free to use the comments section below if you have any feedback or questions.   ... View more

Introduction   In this blog I’d like to talk about some approaches worth taking a close look at when you’re set out to tune the performance of your SAS compute sessions in SAS Viya on Kubernetes. These sessions are still using a SAS 9.x runtime, so you can expect a high backward compatibility when you’re planning to migrate your existing SAS codes. However, they’re now executing in Kubernetes pods and that means a fundamental change in the way how they are provisioned and configured.   Understanding how Kubernetes resource management works might not have been on the list of your favorite To-Dos for the rest of 2022 (so far at least), but it’s definitely an important topic. Keep in mind that a successful deployment of SAS Viya will get you a running but certainly not an optimized environment, so most of the configuration tasks discussed in this blog will be quite obligatory. The following list shows the major areas which determine if your compute sessions will run with adequate performance:   I/O – namely the I/O of your SASWORK drive CPU and memory SAS system options Pod startup latency     Make sure to turn the right knobs   … and there are quite a few of these “knobs”. Performance tuning is not a task which can be done in one spot only. Rather there are at least 3 layers to be considered: the (cloud) infrastructure, the Kubernetes cluster and the SAS configuration. Combine this with the areas I mentioned above and you get this matrix:   “Where to do what” Disk I/O (SASWORK) CPU and memory SAS system options Pod startup latency (Cloud) Infrastructure X       Kubernetes cluster   X     SAS configuration     X X     Disk I/O (SASWORK)   SAS compute workloads are known to be I/O intensive and the SASWORK location is especially important in this regard as SAS programs usually use it quite heavily. There are a couple of options available for SAS Viya on Kubernetes (depending on whether you run in the cloud or not) and instead of covering this here, I’d like to refer you to another blog which is dedicated to this topic.     CPU and memory assignments   Kubernetes is often referred to as a “container orchestration software” and resource management capabilities are not surprisingly at the very core of its’ capabilities. The most obvious resource types to manage are CPU and memory utilization. Kubernetes pods can declare their requirements in their deployment manifests. The Kubernetes service is responsible for making sure that pods are scheduled to nodes which meet the requested minimum and that they to not exceed their declared limits (e.g. the maximum declared amount of memory).   SAS compute sessions start with rather low default settings for CPU and memory. These settings can and usually need to be modified. Compute sessions are based on PodTemplates and the SAS Launcher service is responsible for dynamically creating the session pods on request (e.g. when you log in to SAS Studio or when a batch job is submitted). The Launcher service will also add the resource usage definitions to the session pods it starts. From an administrator’s point of view, you set these definitions as annotations in the metadata section of the corresponding PodTemplate(s). For this blog we’re focusing on the “sas-compute-job-config” PodTemplate as it is the one used by the applications we’re interested in (SAS Job Execution, but also SAS Studio and SAS Model Studio).   The full documentation for these annotations can be found here in the SAS Administration Guide and there is also a README in the deployment assets at $deploy/sas-bases/examples/sas-launcher/configure/README.md. Here are the most important ones which should be set:   Resource Type Annotation  Purpose CPU launcher.sas.com/default-cpu-request guaranteed (min) CPU value CPU launcher.sas.com/default-cpu-limit max CPU limit (max threshold) Memory launcher.sas.com/default-memory-request guaranteed (min) memory Memory launcher.sas.com/default-memory-limit max memory limit (max threshold)   (all using the usual notation for Resource units in Kubernetes).   These annotations should be set when deploying the environment (see the link above), but it’s also possible to patch the PodTemplate afterwards (which comes in handy for testing things). Here are some sample commands:   # set the max CPU limit for SAS Studio sessions $ kubectl -n viya4 annotate PodTemplate sas-compute-job-config \   --overwrite launcher.sas.com/max-cpu-limit=4   # delete the max CPU limit setting (so the default applies again) $ kubectl -n viya4 annotate PodTemplate sas-compute-job-config \   --overwrite launcher.sas.com/max-cpu-request- # display the current settings (if available) $ kubectl -n viya4 describe PodTemplate sas-compute-job-config | \   grep " launcher.sas.com" | grep "cpu\|memory" Annotations:  launcher.sas.com/default-cpu-limit: 4               launcher.sas.com/default-cpu-request: 1               launcher.sas.com/default-memory-limit: 32Gi               launcher.sas.com/default-memory-request: 2Gi               launcher.sas.com/max-cpu-limit: 4               launcher.sas.com/max-cpu-request: 1               launcher.sas.com/max-memory-limit: 32Gi               launcher.sas.com/max-memory-request: 2Gi    (make sure that you restart your SAS sessions (e.g. SAS Studio -> “reset SAS session”) after you made changes to the PodTemplate to let them come into effect).   That’s easy enough, isn’t it? Yes, but there are a few considerations to be aware of. Let’s look at them before continuing.   CPU resource assignment   Setting the default-cpu-request and default-cpu-limit annotations usually defines a min-max range, e.g. say 1-4 CPUs. During execution time the SAS session will request the CPU time it needs and can handle.   What does that mean? Well, not all SAS PROCs are multithreaded, so some of them might stick to a single CPU regardless of what you allow them to use. Take a look at this screenshot showing the resource usage of 2 compute session pods running in parallel:     When this screenshot was taken, one of the sessions was busy executing a DATA step while the other one did a PROC SORT. Guess which is which? Right – the session which used 961m CPU (~ 1 CPU) is the one running the DATA step, while the PROC SORT utilizes ~ 1.5 CPUs because this PROC supports multithreading. This is not something you can influence, but it’s still good to know when trying to understand performance metrics.   Kubernetes uses the default-cpu-request setting to decide on which node the new session pod can be scheduled. As the pod is guaranteed to be able to use at least this amount of CPU, the node needs to have at least the requested free capacity. Which means that in the worst case a high default-cpu-request value might prevent sessions from starting at high-traffic times. On the other hand: being (overly) conservative by setting a too low value for default-cpu-limit will lead to CPU throttling which negatively impacts the compute performance.   Memory resource assignment   default-memory-request and default-memory-limit, the memory related resource management settings, can be used just like the CPU settings. One notable difference is that setting the memory limit too low is risky, as pods trying to use more memory than declared will get evicted (in plain words: they are “OOMkilled”, i.e. killed because they’ve run “out-of-memory”). Which is a much more unfriendly behavior than just throttling CPU usage …   Finding the right range for CPU and memory usage can be tricky and it’s a good idea to use observability tools such as Grafana to monitor the cluster to better understand the resource usage. These tools also provide a good overview of the concurrency of your compute workload, which is important to detect negative effects that emerge from overcommitting the Kubernetes worker node – for example when multiple sessions max out their limits at the same time.     SAS System options   Compute sessions in SAS Viya on Kubernetes are still based on SAS 9.4 and so it seems to be logical that the same set of system options which helped tuning the performance on previous SAS releases is still valid. Typical candidates with high impact on the compute performance are:   MEMSIZE SORTSIZE BUFSIZE   (see the SAS documentation for an explanation of each option or just google them) These configuration options can be set in the “Contexts” plugin of Environment Manager. The SAS Viya Administration Guide briefly summarizes the concept of Compute Contexts like that:   A SAS Compute Server is run under a compute context. (Contexts are analogous to SAS 9® SAS Application Servers.) A compute context is a specification that contains the information that is needed to run a compute server. The information that is contained in a compute context is the user identity and any SAS options or autoexec file parameters to be used when starting the server.   As you can see from the screenshot below, different applications in SAS Viya use different contexts:     For the discussion in this blog, these contexts are the most relevant ones:   SAS Studio compute context – used when launching a SAS session from SAS Studio. Data Mining compute context – for SAS sessions launched from SAS Model Studio pipelines. Reducing the startup latency is an important consideration for this context. SAS Job Execution compute context – for SAS sessions launched as SAS Jobs from the Job Execution Framework (e.g. from the /SASJobExecution UI). Reducing the startup latency is important for this one as well.   Changes made to the configuration of a context are picked up immediately for any new session launched after the change was saved.     (Environment Manager -> Contexts -> Compute contexts -> (pick one context) -> Advanced -> SAS options)   The SAS system options should be aligned to the Kubernetes resource management settings to avoid unpleasant surprises. Let’s briefly discuss the MEMSIZE option, probably the most prominent “usual suspect” when it comes to performance tuning. At first look it might be tempting to simply rely on the Kubernetes settings and just let SAS use all the memory which is available to the pod:   -MEMSIZE MAX   However, this will most certainly lead to errors during program execution with SAS complaining that there was insufficient memory available. It’s important to keep in mind that pods are not virtual machines and setting resource limits will not “magically” change the pod’s view of the hardware environment it runs on. Instead, Kubernetes will pass the container’s resource settings to the underlying container runtime which “translates” this information into Linux kernel cgroups. This however might not be transparent to the application running in the container.   For example, this screenshot shows the output of the Linux top command from a shell inside a SAS session pod which has a max-memory-limit of 8GB but runs on a worker node with 256 GB of memory.     Not surprisingly this also affects SAS when MEMSIZE is set to MAX. Run PROC OPTIONS to validate that the memory of the worker node is used in this case:   80   proc options option=memsize define value lognumberformat; 81   run;     SAS (r) Proprietary Software Release V.04.00  TS1M0 Option Value Information For SAS Option MEMSIZE     Value: 253,556,663,040     Scope: SAS Session   While the top command simply reported the node’s amount of memory, a different command reveals the actual memory limit of the pod (8 GB in that case) and this is a more suitable value to be used for MEMSIZE:   $ CPOD=$(kubectl -n viya4 get pods -l \   "launcher.sas.com/requested-by-client=sas.studio" -o name) # display the cgroups setting for limiting the pod’s memory $ kubectl -n viya4 exec -it $CPOD -c sas-programming-environment \   -- cat /sys/fs/cgroup/memory/memory.limit_in_bytes 8589934592   To summarize the above: it’s recommended to set the MEMSIZE system option as it is critical for SAS performance. However, avoid setting it to MAX - instead match it to what is declared in the max-memory-limit annotation for the PodTemplate.     Pod startup latency   This section is concerned with reducing the startup latency of SAS compute sessions (pods). “Startup latency” refers to the combined time it takes for a) Kubernetes to schedule a new pod to a node and b) for this pod to reach the “Running” state.   Unfortunately these steps take a few moments – depending on the cluster infrastructure it could be just a few seconds to something close to a minute. The sas-prepull service, which is active by default after deployment, tries to minimize this delay by pre-pulling the SAS runtime container image (“sas-programming-environment”) to all compute nodes. In addition to that, SAS Viya introduced the concept of “reusable servers” a few releases ago. “Reusable servers” are compute servers which are not shut down after the session terminates (which is the default behaviour). Instead, they stay around for some time and can be reused by a later session, thus saving the startup time. This feature can even be enhanced by configuring a minimum number of servers to be available at any time. If you feel reminded to what previous SAS releases called a “Pooled Workspace server”, you are on the right track.   As you can imagine, this feature is especially interesting for the SAS Job Execution Framework (i.e. the successor of the Stored Process technology was is available in former SAS releases) and also for the Data Mining Compute context (heavily used by pipelines created in SAS Model Studio). “Reusable compute servers” require a shared account to run them. Here’s an example of how a shared account can be configured using the SAS Viya CLI:   $ sas-viya compute credentials create \   --user viyademo01 --password password1 \   --description "Shared account for reusable compute server"   Once the shared account is available, re-configuring a compute context is rather trivial. Add these options to the right Compute Context definition in Environment Manager:   reuseServerProcesses = true runServerAs = <the shared account> serverMinAvailable = <1 – x>   The last option defines the pool size and should be adjusted to the expected workload (i.e. the level of concurrency). Here’s how the final configuration should look like:     You should see the new compute session(s) starting right after the configuration has been updated.   In conclusion, here’s a short test I did to see the effects of turning on the reusable servers. I simply stopped the round-trip time it takes when calling a SAS Job using a HTTP request (i.e. what a user would do in a web browser):   time \   curl "https://viya.host.com/SASJobExecution/?_program=/Public/ScoringTest&_action=execute" \     --header "Content-Type: application/json" \     --header "Authorization: Bearer $CLIENT_ACCESS_TOKEN" \     -s -o /dev/null -w "%{http_code}"   Running this command a few times with and without the “reusable servers” configuration returned these results:   # / real time (curl) Default (new session pod) Compute server reused 1 0m22.528s 0m6.153s 2 0m19.968s 0m5.919s 3 0m18.019s 0m5.965s   As you can see, the performance gain has been quite remarkable.     Conclusion   In this blog I shared some ideas for tuning the performance of SAS compute sessions in SAS Viya on Kubernetes: how to configure CPU and memory resources and how to minimize (or even eliminate) the startup latency of SAS compute sessions. I apologize for bothering you with Kubernetes details you probably never had wished to know, but the key message has hopefully become clear: you need to take action on this (don’t rely on the defaults) and you have to be aware that some of the “knobs” you’re looking for are found on the Kubernetes level, not within SAS.   I hope this text was useful for you and let me know if you have any feedback or questions.   Helpful resources   SAS® Viya® Administration: SAS Viya Server Contexts: Overview https://go.documentation.sas.com/doc/en/sasadmincdc/v_034/calcontexts/n01003viyaprgmsrvs00000admin.htm   SAS® Viya® Operations: Programming Run-Time Servers https://go.documentation.sas.com/doc/en/itopscdc/v_034/itopssrv/p0wvl5nf1lvyzfn16pqdgf9tybuo.htm   Kubernetes documentation: Resource Management for Pods and Containers https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/   Where to configure the SAS Programming Run-time with broader or narrower scope https://communities.sas.com/t5/SAS-Communities-Library/Where-to-configure-the-SAS-Programming-Run-time-with-broader-or/ta-p/846124   Where SAS Viya Relies on Kubernetes for Workload Placement https://communities.sas.com/t5/SAS-Communities-Library/Where-SAS-Viya-Relies-on-Kubernetes-for-Workload-Placement/ta-p/814780   Some SASWORK storage options for SAS Viya on Kubernetes https://communities.sas.com/t5/SAS-Communities-Library/Some-SASWORK-storage-options-for-SAS-Viya-on-Kubernetes/ta-p/839275     ... View more

This blog can be seen as an “extension” to my overview describing common options for SASWORK storage on Azure AKS (see here). While the other text compares various storage options, I’d like to provide more details about a specific new Kubernetes API called “generic ephemeral volumes” in this blog.   Generic ephemeral volumes have been introduced to Kubernetes in release 1.23. As the name suggests, the lifetime of these volumes is coupled to the lifetime of the pod which makes the request. As the Kubernetes documentation puts it: a generic ephemeral volume is a “per-pod directory for scratch data“ and for that reason it is a prime candidate for being used as SASWORK and CAS disk cache data storage. In case you are wondering why you should bother at all about this I’d like to refer you to the blog I mentioned above. Just as a short summary: the default configuration for SASWORK (emptyDir) will not meet the requirements for any serious workload and the alternative hostPath configuration (albeit providing sufficient performance) is often objected by Kubernetes administrators for security reasons.   As we will see in this blog generic ephemeral volumes provide a good alternative to both options but some caution is advised to make sure that the I/O performance meets your expectations. While the Kubernetes API defines how generic ephemeral volumes can be requested and how their lifecycle looks like, it is up to the specific Kubernetes CSI provisioners how this API is implemented. This blog describes how generic ephemeral volumes can be used in the managed Kubernetes service on Azure (also known as “AKS”).  The most straightforward implementation of this API in AKS can be found in provisioners based on the CSI driver named disk.csi.azure.com. A StorageClass like managed-csi-premium, which is provided by default with each AKS cluster, is a good example for this. All these provisioners make use of Azure managed disks which are mounted and unmounted to the Kubernetes worker nodes.   Introduction to generic ephemeral volumes   The basic functionality of generic ephemeral volumes is simple to explain. A pod can request a volume by adding a volumeclaimTemplate to its’ manifest. The template contains all relevant information about the requested storage such as which CSI driver to use (a.k.a. the StorageClass), the access mode(s) and the volume size. In case you picked one of the default storage classes on AKS which are backed by the Azure Disk service, the CSI driver will attach a new disk to the Kubernetes host and mount it to the requesting pod. Once the pod terminates, the disk will be detached and discarded. Take a look at this picture for a better understanding:     This seems simple enough, however it gets a little more complicated if we want to use this approach for SASWORK, because a SAS compute pod is based on a PodTemplate and these PodTemplates are dynamically built using the kustomize tool.   Using generic ephemeral volumes for SASWORK   It might be easier to work backwards and start by looking at the resulting manifest which we want to create. Here’s a super-simplified PodTemplate describing how a SAS compute pod should look like:   apiVersion: v1 kind: PodTemplate metadata: name: sas-compute-job-config template: spec: containers: name: sas-programming-environment volumeMounts: - mountPath: /opt/sas/viya/config/var name: viya volumes: - name: viya ephemeral: volumeClaimTemplate: spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "managed-csi-premium" resources: requests: storage: 128Gi   Note the volumes section and the use of the “ephemeral” keyword which tells Kubernetes that a generic ephemeral volume is requested. This example uses a default StorageClass on AKS which will provision Premium_SSD disks.   Secondly, note the name of the volume – “viya” – and the mount path which is used inside the pod. By default a SAS session started in SAS Viya on Kubernetes will create it’s WORK storage in a folder below /opt/sas/viya/config/var. Submit the following SAS commands in a SAS Studio session to see where your SASWORK can be found:   %let work_path=%sysfunc(pathname(work)); %put "SASWORK: &work_path"; "SASWORK: /opt/sas/viya/config/var/tmp/compsrv/default/932f131e-...-f0bcce9ded4f/SAS_work1C5C000000B9_sas-launcher-9dfdf749-...-r4ltl"   The name of the volume configuration used here is not arbitrary: it turns out that the SAS developers have created a kind of “plugin” mechanism. We can plug in any storage configuration for SASWORK we want (be it emptyDir, hostPath or a generic ephemeral volume) – as long as our configuration uses the name “viya” it will work without problems.   The SAS deployment assets bundle contains a suitable patch which you can easily modify to define your desired storage configuration (check the README at $deploy/sas-bases/examples/sas-programming-environment/storage for a detailed explanation). Here’s how the kustomize patch should look like in order to generate a PodTemplate which matches the one we just saw above:   apiVersion: v1 kind: PodTemplate metadata: name: change-viya-volume-storage-class template: spec: volumes: - $patch: delete name: viya - name: viya ephemeral: volumeClaimTemplate: metadata: labels: type: ephemeral-saswork-volume spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "managed-csi-premium" resources: requests: storage: 128Gi   In case you’re wondering about this line “$patch: delete” – that’s a special label used by Kubernetes patches which will simply remove any existing content before the new configuration is added. After you’ve created this file (e.g. in the site-config folder), it needs to be added to the kustomization.yaml like this:   patches: - path: site-config/change-viya-volume-storage-class.yaml target: kind: PodTemplate labelSelector: "sas.com/template-intent=sas-launcher"   Note how this patch is applied to all PodTemplates used by the launcher service by setting the labelSelector attribute.   Done! That’s it? Well, let’s check …   Let’s assume you’ve configured SASWORK to use the generic ephemeral volumes and have deployed your SAS Viya environment on AKS. How can you confirm that the configuration is working as you’d expect? There are several checkpoints to be reviewed:   The application (SAS): nothing changed actually. The SAS code snippets shown above still shows that the same folder is used for SASWORK (/opt/sas/viya/config/var). Not surprising actually … The pod: here’s a change to be seen. The new volume is showing up if you exec into the sas-programming-environment pod:   $ df -h Filesystem   Size   Used   Avail   Use%   Mounted on            overlay      124G   18G    106G    15%    / /dev/sdc     125G   13G    113G    11%    /opt/sas/viya/config/var  (output abbreviated) /dev/sdc is the new ephemeral managed disk.   Finally the Kubernetes worker node: the new disk will show up here as well. Run this command from a node shell:   $ df -h Filesystem     Size  Used Avail Use% Mounted on /dev/sda1      124G   18G  106G  15% / /dev/sdb1      464G  156K  440G   1% /mnt /dev/sdc       125G  2.8G  123G   3% /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-dc2...123/globalmount  (output abbreviated). Again, /dev/sdc represents the new ephemeral managed disk. As you can see, on the OS layer the disk is mapped to a folder managed by the Kubernetes daemon (/var/lib/kubelet) which will re-map it for the pod. Just as a side note: /dev/sda1 represents the node’s OS disk – this is where your SASWORK would be located if you had stayed with the default emptyDir configuration. The console output above was taken from an E16ds VM instance which means that there is an additional data disk attached to the machine by default. This is shown as /dev/sdb1. You would want to use this second disk if you configured SASWORK to use a hostPath configuration.   Quiz question: what would happen if you launch a second SAS session? Obivously the only difference will be on the node level. Here you would see yet another disk appearing (/dev/sdd probably) and this is the recurring pattern for every new SAS session – that is: until you reach the maximum number of disks which can be attached to a virtual machine in Azure (which is 32). Don’t worry too much about this limit: keep in mind that these disks are being discarded when the SAS session are closed.   Performance considerations   The example I’ve been using so far uses the managed-csi-premium storage class which uses Premium_SSD disks. How do these disks compare against alternative configurations – most notably against a hostPath configuration? Unfortunately this question is difficult to answer. Remember that this storage driver leverages Azure managed disks and their I/O performance is depending on the size of the disk. The screenshot below is an excerpt taken from the Azure documentation:     The CSI driver tries to match the disk types to the size which was requested for the SASWORK area. In the example above, 128GB were requested and thus the CSI decided to attach a P10 disk to the worker node. Sadly, the P10’s I/O performance (IOPS, throughput) is quite disappointing … What’s the solution for this problem? Requesting tons of gigabytes for SASWORK just to convince the driver to use a more powerful disk? Well, this is one option. Another option is to look for disks with a better performance. Enter Ultra disks.   Quoting the Azure documentation, „Azure ultra disks are the highest-performing storage option for Azure virtual machines”. That sounds appealing, isn’t it? Let’s see what needs to be done.   First of all Ultra disk support is only available for selected Azure VM types and in selected Azure data centers. This page lists the current availability. Luckily, the E*-series of VMs which we prefer to use for SAS Viya compute and CAS nodepools belongs to the machine types which are eligible for Ultra support and the major data centers will be ok to use as well.   Secondly, AKS nodepools do not offer Ultra disk support out of the box. There’s a flag named --enable-ultra-ssd which needs to be set when building the nodepool (or cluster):   # Create a cluster with Ultra_SSD support $ az aks create -g myrg -n myaks  (...) --enable-ultra-ssd # Or: add a new nodepool with Ultra_SSD support to an existing cluster $ az aks nodepool add --cluster-name myaks2 (...) --enable-ultra-ssd   Note that this flag is only available when submitting the commands using the Azure CLI (not the portal).   Finally, you’ll notice that there is no StorageClass available which supports Ultra disks out of the box. So we’ll have to create one on our own:   apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: labels: addonmanager.kubernetes.io/mode: EnsureExists kubernetes.io/cluster-service: "true" name: sas-ephemeral-ultrassd parameters: skuname: "UltraSSD_LRS" cachingMode: "None" DiskIOPSReadWrite: "30000" DiskMBpsReadWrite: "2000" allowVolumeExpansion: true provisioner: disk.csi.azure.com reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer   Note the parameters section. The skuname makes sure that we’ll be requesting Ultra disks while DiskIOPSReadWrite and  DiskMBpsReadWrite are performance settings specifying the requested IOPS and throughput (these values are configurable to some extent, check the documentation here and here for the caps and even more performance flags).   Closing remarks   This blog covered some details about a rather new storage option for SASWORK (and CAS disk cache). Generic ephemeral volumes are available beginning with Kubernetes 1.23 and provide an interesting alternative to the often objected hostPath configuration. Configuring ephemeral volumes is not a very complicated task per se because the SAS developers have prepared most of the groundwork so that it’s easy to “plug in” the right storage configuration. However, keeping an eye on the performance is still of prime importance. In my tests only Ultra SSD disks performed with a reasonable performance when compared to locally attached disks (hostPath). Preparing the AKS infrastructure for Ultra support is a slightly more “involved” task but still a manageable effort which is worth the effort.   I hope that the blog has helped you a bit when planning your Kubernetes infrastructure for SAS Viya on the Azure cloud. Let me know what you think and feel free to ask any question you might have.   ... View more

Introduction   SAS compute sessions have always been heavily depending on good disk I/O performance and this requirement has not changed with the latest SAS platform. Seasoned SAS administrators might not be surprised to hear about this especially when knowing that the SAS compute engine in SAS Viya on Kubernetes is still based on a SAS 9.4 kernel. Choosing a suitable storage configuration for SASWORK usually makes a key difference in terms of compute performance because most SAS programs rely heavily on the WORK library for storing temporary data while executing their data processing steps.   In this blog I’d like to describe some common configuration options for SASWORK when running SAS Viya on the managed Kubernetes service (AKS) on the Azure cloud.   Storage options   In this blog I’d like to talk about 4 alternatives for configuring SASWORK storage:   emptyDir volumes, hostPath mounts, local ephemeral storage mounts and RWX storage (“shared storage”) The following picture tries to summarize these options:   Each option comes with benefits and disadvantages and in some cases the latter actually outweigh the former. Before we discuss these options in more detail, let’s briefly cover some basics about SAS compute sessions in SAS Viya on Kubernetes.   SAS compute sessions are Kubernetes Jobs which are launched on request – for example by users starting a session in SAS Studio, by running a pipeline in SAS Model Studio or by starting a batch job using the SAS Viya CLI. Like any other Kubernetes Job, SAS compute sessions “run to completion” which means that the compute pods will be terminated and removed once the SAS session is completed – for example when a user logs off from SAS Studio of if a batch job completes execution. The runtime configuration of these Jobs is specified by PodTemplates which (among many other things) also describe the storage configuration of the compute pods.   The SAS developers have taken precaution to enable SAS administrators to easily adjust the storage configuration to their needs. Actually, when taking a closer look at this configuration, you’ll notice that the SASWORK location is mounted from a persistent volume which will be mapped to the pod’s filesystem during the start of the session. The mount path of this volume has been carefully chosen to be a parent folder of the default location for SASWORK, which means that data in SASWORK will not be written to the pod’s (virtual) filesystem but to an externalized drive (or folder).   The following command shows some details of how the persistent volume is mounted to the SAS compute pod:   $ kubectl get podtemplates sas-launcher-job-config -n viya4 -o jsonpath="{.template.spec.containers[].volumeMounts}" [ {     "mountPath": "/opt/sas/viya/config/var",     "name": "viya"   } (...) ]   The output above tells us that a persistent volume using the identifier “viya” will be mounted to the pod’s filesystem starting at /opt/sas/viya/config/var. The code snippet below will display the physical location of SASWORK (as seen from the pod’s “point of view”) when submitted from a SAS Studio session:   %let work_path=%sysfunc(pathname(work)); %put "SASWORK: &work_path"; "SASWORK: /opt/sas/viya/config/var/tmp/compsrv/default/932f131e-...-f0bcce9ded4f/SAS_work1C5C000000B9_sas-launcher-9dfdf749-...-r4ltl"   So the SASWORK folder of this (and any other) SAS session is a subfolder of /opt/sas/viya/config/var and hence will be written to the “viya” persistent volume we just saw. Obviously this leads to some questions regarding this mysterious “viya” volume. It turns out that there is a default configuration in place which can (and should) be overwritten during the deployment to ensure that the optimal performance for SAS sessions is available.   This is where the different configuration options mentioned above can be applied. In other words: re-configuring SASWORK storage is done by re-defining the “viya” volume in the PodTemplates used by SAS compute sessions.   emptyDir   This is the default configuration if not changed during the deployment. Assuming you did not follow the instructions given in $deploy/sas-bases/examples/sas-programming-environment/storage/README.md, the SAS compute sessions will use emptyDir volumes for SASWORK.   It’s easy to check if you’re running with the default or not by examining another section of the PodTemplate (abbreviated output):   $ kubectl get podtemplates sas-launcher-job-config -n viya4 -o jsonpath="{.template.spec.volumes}" [ {     "emptyDir": {},     "name": "viya"   } (...) ]   Which tells us that in this case the “viya” volume is of type “emptyDir”, i.e. it’s using the default configuration. It seems that emptyDir is the easiest approach, so why not stick with this option? The major disadvantages are that Kubernetes usually creates the emptyDir on the node’s operating system disk and also limits the capacity of this volume. The limit for emptyDir on AKS is 50GB and the Kubernetes scheduler will evict your session if you create more temporary data than this in your SAS session – simply speaking: your session will be terminated forcefully. Moreover, even other issues can show up because the node’s OS disk might be too small to host more than a few concurrent SAS sessions (even if each of them stays below the 50GB limit) and this will also cause errors.   Just as a side note: emptyDir does not need to be a created on the node’s disk actually. Following the definition of emptyDir on the Kubernetes documentation pages, these volumes can be created as RAM-backed filesystems as well:   Depending on your environment, emptyDir volumes are stored on whatever medium that backs the node such as disk or SSD, or network storage. However, if you set the emptyDir.medium field to "Memory", Kubernetes mounts a tmpfs (RAM-backed filesystem) for you instead. (https://kubernetes.io/docs/concepts/storage/volumes/#emptydir)   While this can speed up the “I/O” of the volume it does not remove all of the disadvantages that come with the use of emptyDir of course. In short: running SAS Viya with the default setting is ok for small test environments, but it’s definitely not recommended for any serious workloads.   hostPath   As of today hostPath is the recommended approach for SASWORK in most cases. Like the name suggests, hostPath allows a pod to directly mount a directory (or drive) from the host’s filesystem. To avoid the issues we just discussed for emptyDir it’s evident that the hostPath directory (or drive) should not be the node’s OS disk. In other words: hostPath usually requires that the node has at least 2 disks attached to it.   Not all VM types on Azure provide this additional temporary storage disk out-of-the-box. The ones that do have a lowercase “d” in their names and are slightly more expensive. Their additional disk is directly mounted to the machine which provides very good I/O throughput rates. SAS often recommends the Edsv5 series of machines especially for the SAS compute and CAS node(s) for this reason. Here’s an excerpt from Microsoft’s documentation showing the specs of some typical machine sizes:   Size vCPU Memory: GiB Temp storage (SSD) GiB Max data disks Max temp storage throughput: IOPS/MBps * Max NICs Max network bandwidth (Mbps) Standard_E8d_v5 8 64 300 16 38000/500 4 12500 Standard_E16d_v5 16 128 600 32 75000/1000 8 12500 Standard_E32d_v5 32 256 1200 32 150000/2000 8 16000     The additional disk is referred to as “temp storage (SSD)” in the table above. Azure VMs by default mount this drive to their OS disk using /mnt and this is the mount point which can be used for defining the hostPath volume for SASWORK. The required kustomize patch is not complicated to create and documented here: $deploy/sas-bases/examples/sas-programming-environment/storage/README.md. Here’s an example of how the patch will look like:   apiVersion: v1 kind: PodTemplate metadata: name: change-viya-volume-storage-class template: spec: volumes: - $patch: delete name: viya - name: viya hostPath: path: /mnt   The I/O throughput of volumes mounted like this is usually very good. Unfortunately there are at least two potential disadvantages which may prevent this configuration from being used.   To begin with, the use of hostPath is often discouraged. It can be seen as an anti-pattern in the Kubernetes world because it weakens the level of isolation of the pod, making it easier for malicious software to escape the container sandbox and take control over the node. The situation is probably not that dangerous given that SAS sessions run in LOCKDOWN mode by default. LOCKDOWN shuts down SAS language feature that provides direct or indirect access to the operating system shell. Nevertheless, Kubernetes administrators in general will not be overly excited when being asked to provide a hostPath configuration for SAS and there might be business requirements which demand that the LOCKDOWN mode gets relaxed to some extent.   Secondly, relying on storage which is directly mounted to the Kubernetes nodes brings some restrictions regarding the available storage volumes. When running on public cloud infrastructure you need to choose between VM instance types of different sizes which means that storage, CPU and memory resources are tightly coupled. This is negatively impacting the overall infrastructure costs, forcing you to select larger (and maybe more) nodes than you would choose if CPU and memory could be sized independently from the storage.   Generic ephemeral volumes   This is a rather new option which was introduced with Kubernetes v1.23. The Kubernetes documentation states that these volumes can be compared to emptyDir but do provide some advantages, most importantly:   The storage can be local or network-attached The volumes can have a fixed size (quota) that pods are not able to exceed Using ephemeral volumes avoids the coupling discussed above – the storage requirements will no longer drive which VM instance types you use for the worker nodes hosting SAS compute workload. This is likely to create much more reasonably sized and more cost-effective cluster topologies. Note that the actual implementation of this API is depending on the infrastructure where your Kubernetes cluster runs. This blog focuses on Azure AKS where ephemeral volumes are implemented using Azure managed disks. Here’s how the kustomize configuration patch for SASWORK could look like:   apiVersion: v1 kind: PodTemplate metadata: name: change-viya-volume-storage-class template: spec: volumes: - $patch: delete name: viya - name: viya ephemeral: volumeClaimTemplate: metadata: labels: type: ephemeral-saswork-volume spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "managed-csi-premium" resources: requests: storage: 100Gi   This example uses a pre-defined storage class on AKS clusters which is configured to provide Premium_SSD disks. Also note the requested “storage” size parameter – this is how a SAS administrator could set up a quota for users. This parameter sets the maximum size for their data in SASWORK. Let’s see how this looks like from the cloud provider’s point of view. Imagine that a SAS session was started and you would run the “df” command on a node shell on the worker machine:   # df -h Filesystem  Size  Used Avail Use% Mounted on /dev/sda1   124G   41G   84G  33% / /dev/sdb1   464G  156K  440G   1% /mnt /dev/sdg    98G   24K   98G    1% /var/lib/kubelet/plugins/                                   kubernetes.io/csi/pv/                                   pvc-4711.../globalmount   The device /dev/sdg represents the managed disk which has been requested for SASWORK by one SAS session (i.e by a volumeClaimTemplate section in the PodTemplate).  This disk is also shown in Azure portal:     As a side note: the “df” output shown above was taken from a Edsv5 instance, so /dev/sdb1 refers to the additional disk mentioned in the previous section and we could have used /dev/sdb1 for SASWORK in a hostPath configuration alternatively. There’s one fundamental difference: if we had used /dev/sdb1 and hostPath for SASWORK then all concurrent SAS sessions would need to fit into the 464 GB of storage which are offered by the local disk available on this VM instance type. While using ephemeral volumes allows us to grant 100 GB of storage for each SAS session, regardless of how many concurrent sessions we have (well, that’s not entirely true as there is a maximum of how many disks you can attach to a single virtual machine at the same time).   Ephemeral volumes behave exactly as their names suggest – once the SAS session terminates, the associated volume will be unmounted and discarded automatically.   Sounds too good to be true? What about disadvantages – are there any at all? It depends … Taking a second look at the screenshot above you might notice that the max IOPS and throughput rates are quite low. In fact, this particular configuration will probably show a pretty bad performance. Why’s that? Well, the volume was requested from the built-in managed-csi-premium StorageClass which provides Premium SSD storage disks. Since we had defined a quota of 100 GB, the CSI storage provisioner decided to provide us a P10 disk with a capacity of 128GB as this was the smallest disk which satisfies our request. And if you check the specifications of the Premium SSD disks you’ll see the same provisioned IOPS (500) and throughput (100) rates as in the screenshot above. These rates are coupled to the disk sizes (P10, P20 etc.) – larger disks provide better performance. This means that we probably have to request larger disks just because we want to see performance improvements.   The good news is that this can be addressed with some extra effort. The configuration details are covered in another blog, but here’s the short summary: switching from the built-in StorageClasses to a custom class leveraging Ultra SSD drives and tuning the IOPS and throughput rates results in a much better performance.   RWX storage   I’d like to cover this option only briefly as it is usually not a recommended approach. Configuring “shared storage” (i.e. storage allowing “RWX” or “ReadWriteMany” access) is easy but often it will not meet the I/O throughput requirements for SASWORK. There are ways to improve the performance (by preferring SAN over NAS devices, by using high-performance filesystem services like NetApp and more), but that again increases complexity and infrastructure costs.   Let’s take the file.csi.azure.com storage provisioner as an example, although I'd not recommend using it for this purpose. This CSI driver supports dynamic and static provisioning, so it can create the required Azure Files file shares “on request” or it can use shares which have been created beforehand. In any case check the SAS Operations Guide for an example of how to create the custom StorageClass so that the required mount options are set correctly. Assuming you want to refer to an existing file share and PersistentVolumeClaim then this is how you would configure the file share to be used for SASWORK:   apiVersion: v1 kind: PodTemplate metadata: name: change-viya-volume-storage-class template: spec: volumes: - $patch: delete name: viya - name: viya persistentVolumeClaim: claimName: pvc-saswork   Conclusion   In this blog I have covered some typical choices for configuring the SASWORK storage for SAS compute sessions. The most important take-away is probably that you really need to change the default configuration: staying with the emptyDir configuration will get you into trouble sooner rather than later. The tests I have run using the different options showed that hostPath seemed to provide the best performance, but ephemeral volumes actually came close (when configured to use Ultra_SSDs). Given that Kubernetes administrators might object the use of hostPath, ephemeral volumes can be a suitable alternative. ... View more

Many years ago – even before I started my professional life as a software engineer – I became fascinated with Perl. It was the first programming language I ever learned and I really loved it because it allowed for nearly unlimited chaotic creativity (take a look at some results of the annual obfuscated Perl contest if you’re not sure what I am talking about).   I admit that over time I forgot how to code in Perl, but what really stayed in my mind was a quote from Perl’s inventor, Larry Wall, which can be found in his foreword of the famous Camel Book:   “laziness is a virtue” (in programming at least)   I must confess that I always believed this statement to be true and for that reason I was excited to see that SAS released a utility called the SAS Deployment Operator earlier this year. Using the Operator has become the recommended approach of deploying SAS Viya as it can help you to fully automate the software lifecycle management of a SAS Viya environment (deploying, re-deploying, updating …). Put simply: it’s a pod taking care of other pods. Here are some of the key benefits of using the Operator (as opposed to using the traditional, manual deployment approach):   It’s an approved standard – the Deployment Operator adheres to the standard Kubernetes Operator pattern for managing workloads on Kubernetes clusters. It’s GitOps – providing automation, version control and configuration management. Clear separation of user privileges – let SAS administrators manage the lifecycle of SAS Viya without the need for elevated permissions and without requiring the Kubernetes administrators to help them out.   In this blog I’d like to provide you with more details on how the Deployment Operator can be integrated into a GitOps workflow on the Red Hat OpenShift Container Platform.     SAS Deployment Operator   In order to save some space I won’t cover the basics on how to deploy the Operator because this is all well documented in the SAS Operations Guide. I'll just mention that I’ll be using the Operator in cluster mode, so the Operator will have its’ own namespace and will be able to handle multiple SAS Viya deployments on the same cluster. And obviously I’m going to use a Git server for storing the deployment manifests which need to be presented to the Operator. There are alternative options for both things – please have a look at the docs to learn more about them. Our starting point looks like this:     Let me briefly explain the major components of this infrastructure before we move on. In the above diagram you see:   The Git server. This is the location where we will store the deployment manifests and the custom resource (CR) which triggers the Deployment Operator The SAS Deployment Operator (in cluster mode). The Operator runs idle until it gets triggered by a SASDeployment CR (more on this later) This is our target namespace, i.e., it’s the namespace where we want SAS Viya to be deployed. For now it’s empty. A deployment of Red Hat GitOps, which is OpenShift’s built-in solution for Continuous Delivery tasks. We’ll talk about this in more detail below.   There are some preliminary steps which are not shown on the diagram. Most importantly the OpenShift administrator should already have created the required custom Security Context Constraints (SCCs) and assigned them to the (not-yet-existing) local SAS Viya service accounts. This is a one-time preparation task when setting up the target namespace. Assigning project users, project quotas and storage volumes are examples for other tasks which also might need to be completed beforehand.   Now let’s focus on the Deployment Operator. Like all operators following the standard pattern it watches for Kubernetes configuration objects of the type CustomResource (CR) which are user-defined extensions to the Kubernetes API based on CustomResourceDefinitions (CRDs). SAS has created a CRD named “SASDeployment” and by submitting an instance manifest of this CRD to a target namespace in the cluster, we can trigger the Operator to start working.   But just what does that exactly mean? How does the Operator know what to do and where to find things? This is all part of the CR we’re submitting. It contains the license information, the URLs of the container image registry (if a mirror is used), the cadence we want to deploy (e.g. STABLE or LTS) and – most importantly – the location where to find the manifests and patches to use for the deployment. Here’s a shortened example of a deployment CR:   apiVersion: orchestration.sas.com/v1alpha1 kind: SASDeployment ... spec:   caCertificate: ...   clientCertificate: ...   license: ...   cadenceName: stable   cadenceVersion: 2021.1.6   imageRegistry: container-registry.org/viya4   userContent:     url: git::https://user:token@gitserver.org/sas/viya-deploy.git   Of course there is no magic involved. You still have to provide the manifests and patches which are required just like you would do for a manual deployment of SAS Viya. However, you do not have to use the kustomize and the kubectl apply commands – this is accomplished by the Operator.   There’s a funny looking URL shown at the end of the abbreviated CR shown above – this is a so-called go-getter URL, which is basically a way to inject information about the source system to a download URL. In our case, the “git::”-prefix tells the Deployment Operator that the userContent resides in a Git repository. The SAS administrator owns the contents of this repository and needs to update it whenever a configuration change is required. The repository is simply a 1:1 copy of the manifests and patches from your site-config folder (note that you do not have to include the sas-bases content – the Operator knows how to get it):     At this point, we nearly have everything we need: there’s the Operator, a target namespace, a Git repository with all manifests and patches and a CR manifest to kick off the Operator. We could actually simply submit the CR file to the cluster with a kubectl command like this:   $ kubectl -n sas-viya apply -f ~/viya-sasdeployment-cr.yaml   This would cause the Operator to deploy a reconcile job to the sas-viya namespace which would start deploying the software. But this is not what we want to do as it would be a manual intervention which we want to avoid if possible. Shouldn’t there be an automated way of managing the lifecycle of a SAS Viya environment? Indeed there is - enter Red Hat OpenShift GitOps.     Red Hat OpenShift GitOps   Red Hat OpenShift GitOps was announced earlier this year as an integrated addon for the OpenShift Container Platform (OCP). Being part of the OCP eco-system, it is very easy to install using the built-in OperatorHub. It provides you with a turnkey-ready tool for automating continuous delivery (CD) tasks. It’s based on the well-known open-source tool ArgoCD, which is often paired with Tekton as a tool for continuous integration (CI) for full CI/CD automation. However for our purposes it’s sufficient to focus on the “CD” part.   In order to eliminate the last manual step mentioned above we will use another Git repository which only contains our CR manifest. We will instruct OpenShift GitOps to monitor this repository and automatically sync its’ contents to the cluster. In other words: pushing the CR manifest to the Git repository will trigger a sync with OpenShift GitOps. The CR will be deployed to Kubernetes, which in turn triggers the Operator and the deployment will start. This is how our Git repository for the CR could look like:     There couldn’t be much less in it, isn’t it? How do we set up the connection between OpenShift GitOps and Git? This is done by creating an Application object:   apiVersion: argoproj.io/v1alpha1 kind: Application metadata:   name: sas-viya-cr-gitops   namespace: openshift-gitops spec:   destination:     name: ''     namespace: sas-viya     server: 'https://kubernetes.default.svc'   source:     path: yaml     repoURL: >-       https://user:token@gitserver.org/sas/viya-deploy-cr.git     targetRevision: HEAD   project: default   There’s one caveat – we need to allow the OpenShift GitOps service account to create the CR resource in our target namespace. A simple way to accomplish this is to add the service account to our target project, for example like this:   $ oc project sas-viya $ oc adm policy add-role-to-user edit system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller # we need to grant additional privileges, see the attached file at the end of the blog for details $ kubectl apply -f ~/grant-privileges-to-openshift-gitops-sa.yaml   And that’s it – at least as far as we are concerned. Get yourself a cup of tea (or coffee), do a “git push” on the CR repository and watch the system do the heavy lifting for you:     The above screenshot shows the OpenShift GitOps web interface shortly after the Operator has kicked in and deployed the reconcile job. In a few minutes this screen will really look busy as more and more SAS objects start to appear.   Before I wrap up this blog let me return to the diagram I showed at the beginning. Here’s the extended version of it, showing the major steps of the GitOps process I’ve described so far:     The SAS administrator pushes the deployment patches and manifests to one Git repository and the deployment CR to a 2 nd Git repository. OpenShift GitOps monitors the CR repository and starts syncing its’ contents to the designated target namespace. This creates the CR object in the cluster. The SAS Deployment Operator monitors the target namespace and creates a reconcile job as soon as it detects the CR object The reconcile job will then start deploying SAS Viya into the target namespace.   Wrapping up   In this blog I have described a way to leverage Red Hat OpenShift GitOps and the SAS Deployment Operator to automate the lifecycle management of SAS Viya on the OpenShift Container Platform. I hope I could whet your appetite to invest some time in getting familiar with these tools as I am fully convinced that it will be worth the effort. Let me close the blog by showing you this fancy view of a SAS Viya deployment on OpenShift. It’s a screenshot taken from the OCP console showing the contents of the target namespace and if you see this it means “mission accomplished” for the Operator.     Leave a comment to let me know what you think about the blog and don’t hesitate to reach out to me if you have any questions.   ... View more

Earlier this year we had the exciting opportunity to contribute to one of SAS’ largest online events which took place this year (and no, we’re not talking about the SAS Global Forum here …).   Unlike in previous years, where Hackathons were regional events, 2021 was the year of the first SAS Global Hackathon where participants from all over the globe would join in teams to tackle “solutions to big business and humanitarian challenges, all while experimenting with the latest analytics and cloud software” as it was stated in the announcement.   To give you a rough idea of the truly impressive scale of this event: we saw more than 1.100 registered participants from almost 80 different countries of which more than 750 estimated themselves as being data scientists. We asked the participants to flock together to a total of 100 teams to work on a dedicated use-case over a 4 weeks period. The teams chose their use-cases from a wide range of ideas and many of them worked on world’s most critical issues like cancer research, sustainable nutrition or energy utilization & CO2 Emission control. For more information we’d suggest that you spend a few minutes to check this page which lists the Hackathon winners and their use-cases.   Not one-size-fits-all   Without doubt, hosting an event of this size can be an organizational and a logistical challenge. All accepted teams were eligible to work on dedicated SAS Viya environments running in Microsoft’s Azure cloud for the duration of the Hackathon and we were tasked with provisioning this infrastructure. Some basic parameters were clear right from the start: since this was a global event, the infrastructure needed to be globally distributed as well in order to be as close to the teams as possible. Hence, running these Viya environments in the cloud was an obvious choice. This diagram shows the Azure data centers we used for deploying the SAS Viya environments.     Secondly, due to the diversity of use-cases which often demanded the latest state-of-the-art analytical capabilities, we knew that we had to deploy the latest and most feature-rich release of SAS Viya, which is a cloud native software platform running on a Kubernetes infrastructure. We decided that each team would have its’ own Kubernetes cluster so that we would not risk to encounter any “noisy neighbour” effects.   Finally it became clear after reviewing the submitted use-case proposals that an one-size-fits-all approach would not be the best choice. The requirements were just too different to cover them all in a single “template”. For example, some teams needed to work on large amounts of data while others didn’t. Moreover, not all teams were of the same size and when looking at the functional requirements we had requests for almost anything you could think of: computer vision capabilities, AI and Machine Learning algorithms, Optimization and Forecasting, Streaming Analytics, Non-SAS programming languages (Python, R) and much more.   For these reasons we decided to offer two different types of SAS Viya environments with quite distinct characteristics which we called “Platform” and “Platform Plus”. The “Platform” environments were designed as self-service offerings for smaller teams with lower requirements in terms of data while the “Platform Plus” environments were more powerful and centrally maintained by us. In the next sections we’re going to describe the technical details for both types of environment, but be sure to also check out this nice blog authored by Frederik about the design factors which were imperative for our work.   The SAS Viya “Platform” environments   The “Platform” types of environments were designed for teams with 2-3 concurrent users which aimed at working on less than 20 GB data (largest table). From a technical perspective they were quite unusual because we deployed the SAS Viya platform stack on a single-node Kubernetes cluster, which in turn allowed us to provision everything on a single virtual machine. Of course, these VMs needed to be powerful enough to host all components of a SAS Viya environment, including the CAS in-memory server. The Standard_E32s_v4 machine types seemed to be a good fit for this, as they come with 32 vCPUs and 256GB of RAM. The machines in Azure’s Esv4 series run on the Intel® Xeon® Platinum 8272CL (Cascade Lake) with a clock speed of 3.4 GHz.     Being cloud-native, SAS Viya requires a Kubernetes infrastructure as it’s “operating system”. We chose to use the open-source upstream release of Kubernetes which is rather easy to deploy even on a single machine where the same node takes over both roles of being a master and a worker. This is certainly not a best-practice as you loose elasticity and fault tolerance capabilities on the infrastructure level, but in this specific case it did fit the bill. After preparing the base image we used the Azure Image Gallery service to replicate the image to multiple Azure data centers in different regions (US, EMEA, APAC), because we wanted the VM instances to be running close to the teams' home location to reduce network latencies.   How did we make the environments available to the teams? Given the large number of teams using the “Platform” type of environments (almost 50), we needed to find an automated approach but at the same time we also wanted to give them full control over their environment’s lifecycle. For that reason we developed a self-service Launcher web application which served 3 purposes:   to create, start, stop and terminate the team’s virtual machine to manage the IP address whitelist (for security reasons each instance was protected by a network security group) and finally we asked teams to consider setting up a schedule when their environment should be stopped automatically in order to save some costs and energy (e.g. during the night or weekend).     One important piece of the puzzle is still missing: how did teams make data available to their environments and where did we store that data? Of course, they could use the built-in capabilities for uploading data through the visual interfaces, but this option has limits in terms of the accepted file sizes. Storing the data on the virtual machine could also be risky because we allowed teams to terminate and re-create their environments (in which case the data could have gone lost).     The solution for this problem was to attach external cloud storage to each machine, so each team had access to its’ own file share which would be available throughout the whole Hackathon. Finally we asked the teams to install the Azure Storage Explorer tool on their local Laptops to access their shares. This provided a convenient way to upload larger amounts data which would then be available for being processed by SAS Viya.   The SAS Viya “Platform Plus” environments   Up to this point, we described how the smaller "Platform" environments were automatically created, stopped and started. But we implemented more automation as we decided to automate the deployment of the larger environments as well, i.e. the "Platform Plus" environments. As said before the large environments were meant to support teams with more users and more data. For that reason we deployed SAS Viya on Microsoft Azure AKS clusters. Those clusters consisted of multiple node-pools, each having enough nodes with sufficient hardware resources.     By making use of Kubernetes taints and labels we applied the recommended SAS workload placement. That way it’s feasible to distribute dedicated SAS workload across the different nodes. The setup of the infrastructure and the deployment of Viya was done by making use of the SAS IaC  & SAS Viya4-deploytment scripts.  To make life easy we automated the run of those scripts with Azure DevOps pipelines. The technical details around how that was done can be found on this SAS communities article. Or check this video if you want to see a deployment in action.   Since the Hackathon we are keeping the installation pipelines up to date so that they make use of the latest IaC and Viya4 scripts. That way we can always install the latest releases and/or cadences of Viya. And most importantly the Azure DevOps pipelines continue to proof their value and are helpful to teams setting up analytical “sprints” or demonstrations, or for conferences and events, or for students or many other uses. Any run of the Azure DevOps pipeline gives us all green checks when the deployment was done successfully:     The Azure DevOps pipeline consists out of stages (initialize, iac_setup, baseline and iac_destroy). Each stage will run different tasks.        During or after the run of the pipeline it’s easy to follow up if the different tasks are executed as expected:     Wrapping up   Providing the technical infrastructure for the SAS Global Hackathon 2021 certainly was a challenging task, but it also was fun to do and a great learning experience of using cloud technologies “at scale” as well. Even more important, the feedback we received from the participants of this event was very positive. In fact, this event has been so successful that it raised the appetite for more - so be on the lookout for more Hackathons coming up in 2022.   Hoping to see you there!     ... View more

With the 2021.1.3 STABLE release of SAS Viya 2 new publishing destinations have been added to SAS Model Manager: you now have the ability to publish SAS models to Azure’s Machine Learning service (AML) as well as to a new container destination called the SAS Container Runtime (SCR). In this blog we will not discuss the AML integration but focus on the configuration steps required to set up the SAS Container Runtime for Viya environments deployed in the Azure cloud. Want to start with an overview first before diving knee-deep into technology? - then this blog is what you need. If you’re interested in the AML integration specifically, you should take a look at this blog describing the new feature in more detail. We also highly recommend that you spend a few minutes reading this blog which gives you a great overview of all available publishing destinations (including the new ones).   To cite the official documentation: the SAS Container Runtime is a “lightweight OCI (Open Container Initiative) compliant container that scores SAS models. This enables you to run models on any OCI compliant compute system, including clusters that are based on Docker and Kubernetes.” (SAS Container Runtime: Programming and Administration Guide).   Simply put: SCR allows you to package your SAS model into a container image which has been specifically designed for executing SAS analytical models (as of now – a future release will bring the support for models written in Python and R as well). Each SCR container image hosts a single SAS model. The image does not have any dependencies to the outside world so it will work right away, no matter where you deploy it. External clients send requests to the SCR container using REST calls by communicating with the built-in REST server in each container. The container approach comes with a lot of benefits: there’s no dependency on the runtime environment (on premise, cloud …), it’s stateless (so it can be deployed using multiple replicas to address scalability and availability requirements), it’s easy to update without downtime (rolling updates) and much more.   This blog will guide you through the configuration steps needed for setting up the SCR destination for the Azure cloud. Apart from publishing to a container registry, SAS Model Manager also supports running a publishing validation check, which allows you to test if publishing your model actually worked. In our case, a Kubernetes cluster is used for running the validation and throughout this blog, we’ll use the terms “Viya cluster” and “validation cluster” rather often. Keep in mind that this is a conceptual thing – nothing is keeping you from using a single Kubernetes cluster on which you deploy SAS Viya and validate your models at the same time (e.g. for testing purposes). However, even then you still need to follow all of the configuration steps written down below.   The conceptual architecture looks like this:     For the rest of this blog, we’re assuming that you have already created the AKS cluster(s) and an instance of the Azure container registry (ACR). We also assume that you can use the Azure CLI for submitting commands to Azure. We’re not going to cover the details of a SAS Viya deployment (this is documented here), but make sure to read the next section before you start deploying Viya.   Here’s a brief overview on the configuration steps we are going to cover: We need to establish a service principal in both the container registry and the validation cluster with the appropriate role assignments to allow the principal to push/pull images to/from the container registry and to deploy the container image on the validation cluster And of course, we also need to register the ACR registry and the validation cluster as a new publishing destination in SAS Model Manager   Apply the kaniko storage patch to your Viya deployment   While we said that we would not cover the deployment of SAS Viya, we’re including this section to remind you that there is one important kustomize patch which you should not forget if you want to publish a SAS model as a container image. This patch (and README) can be found in   $deploy/sas-bases/examples/sas-model-publish/kaniko/   Where $deploy points to the directory where you unpacked the Viya deployment assets.   This patch assigns storage to the model publishing microservice which is needed for the kaniko tool. In case you have not heard about kaniko before: it’s an open-source tool “to build container images from a Dockerfile, inside a container or Kubernetes cluster” (copied from the kaniko homepage). The model publishing microservice uses kaniko to build the SCR image on-the-fly when you publish a model to the SCR destination from Model Manager.   Applying the patch should be pretty straight-forward:   cd $deploy mkdir -p site-config/sas-model-publish/kaniko cp sas-bases/examples/sas-model-publish/kaniko/*.yaml site-config/sas-model-publish/kaniko chmod 644 site-config/sas-model-publish/kaniko/* # set capacity of RWX storage sed -i "s|{{ STORAGE-CAPACITY }}|20Gi|" site-config/sas-model-publish/kaniko/storage.yaml # set storage class providing the RWX storage (NFS, Azure Files ...) MY_RWX_SC=myrwx sed -i "s|{{ STORAGE-CLASS-NAME }}|$MY_RWX_SC|" site-config/sas-model-publish/kaniko/storage.yaml   Don’t forget to include the patch to your kustomization.yaml:   resources: ... - site-config/sas-model-publish/kaniko transformers: ... - sas-bases/overlays/sas-model-publish/kaniko/kaniko-transformer.yaml   Assigning a service principal to the Azure container registry and the validation AKS cluster   First of all create a new service principal and keep it’s ID and password:   # this command will return the tenant and the subscription ID az account list -o table \    --query "[].{name: name, tenantid: tenantId, subscriptionid: id}" SUBSCR_ID=/subscriptions/<your-subscription-id> # create the service principal az ad sp create-for-rbac --scopes $SUBSCR_ID --skip-assignment true -n azure-scr-sp # store AppID SP_APPID=$(az ad sp list --display-name azure-scr-sp --query "[].appId" -o tsv)   Important: Be sure to also write down the password generated by this step. You will need it later when creating the publishing destination in SAS.   Before we can create the role assignments we need to retrieve the IDs of the container registry and the validation AKS cluster. This could be the AKS where you deployed Viya, but it could also be a different cluster (e.g. a “production cluster” while Viya has been deployed on a “development cluster”).   # replace these with meaningful values RESOURCEGRP=myresgrp VALIDATE_AKS_CLUSTER=myaks ACR_NAME=myacr # get the ACR ID ACR_REGISTRY_ID=$(az acr show --name $ACR_NAME --query id --output tsv) # get the AKS ID AKS_ID=$(az aks show --resource-group $RESOURCEGRP \    --name $VALIDATE_AKS_CLUSTER --query id -o tsv) # assign ACR Push and ACR Pull roles to the service principal # (acrpush includes acrpull) az role assignment create --assignee $SP_APPID \    --scope $ACR_REGISTRY_ID --role acrpush # assign the cluster admin role to the service principal az role assignment create --assignee $SP_APPID --scope $AKS_ID \    --role "Azure Kubernetes Service Cluster Admin Role"   Opening the validation cluster’s firewall for accessing NodePort services   If you ask Model Manager to run a publishing validation on your new model after you’ve pushed it to the registry, it will deploy a new SCR instance on the target cluster for the duration of the validation test. Once the new SCR instance is running, Model Manager needs to be able to connect to it in order to send scoring requests to it.   From a technical point of view the Model Manager pod issues REST calls to the SCR pod. By default the SCR pod is deployed to the validation AKS using the NodePort service type, which is a mechanism provided by Kubernetes to make pods accessible to external clients. Pods exposed via NodePort can be reached from any Kubernetes node using the same port address (so a client does not need to know on which particular node a pod has been deployed as long as it knows on which port it has been exposed). NodePort uses a port range of 30000-32767, so we have to allow inbound traffic for this range. Luckily we do not need to open this range to just anyone: because we know that Model Manager will make the request, we can restrict incoming connections from the public IP address of the AKS cluster where Viya is deployed to the public IP address of the Load Balancer serving the cluster where the validation is supposed to happen. This configuration step is required even if you are just using a single cluster.   You can either make these changes manually or by using a few shell commands. Either way, this is how it should look like after applying the changes (this picture shows the network security group used by the AKS nodes):     You can use the following snippet to create the inbound security rule programmatically. Careful: this snippet assumes that there is only one AKS cluster, so you need to make changes to this if you have separate ones.   RESOURCEGRP=myresgrp # find the internal AKS resource group AKS_RG=$(az aks list -g $RESOURCEGRP --query [].nodeResourceGroup -o tsv) echo "AKS resource group: $AKS_RG" # find the LB IP address of the AKS cluster (outbound) OUTBOUND_IP=$(az network public-ip list --query "[].{tags: tags.type, address: ipAddress}" -o tsv -g $AKS_RG | grep aks-slb | cut -f2) echo "AKS outbound IP: $OUTBOUND_IP" # find the IP address of the AKS cluster (inbound) INBOUND_IP=$(az network public-ip list --query "[].{tags: tags.type, address: ipAddress}" -o tsv -g $AKS_RG | grep None | cut -f2) echo "AKS inbound IP: $INBOUND_IP" AKS_NSG=$(az network nsg list -g $AKS_RG --query [].name -o tsv) echo "AKS NSG: $AKS_NSG" az network nsg rule create -g $AKS_RG --nsg-name $AKS_NSG -n AllowNodePortRange \    --priority 100 \    --source-address-prefixes $OUTBOUND_IP/32 \    --source-port-ranges '*' \    --destination-address-prefixes $INBOUND_IP \    --destination-port-ranges '30000-32767' --access Allow \    --protocol Tcp --description "Allow access to pods via nodeport"   Register your Azure Container Registry instance as a publishing destination in Model Manager   With the Azure related preparation steps in place, it’s now time to turn to the SAS configuration. Before we can publish SAS models to Azure (that is: to the instance of the Azure Container Registry (ACR) which you want to use), we first need to define the ACR instance as a publishing destination in Model Manager.   Creating a publishing destination actually consists of two steps as you need to create a Credentials Domain (also referred to as a Base64 Domain) and the Publishing Destination. Luckily both steps can be completed by running just a single command using the SAS Viya CLI. If you have not worked with the SAS Viya CLI before, this would be a good chance to make yourself familiar with it as it greatly simplifies many administration tasks in SAS Viya.   Assuming that you have downloaded the CLI, installed the plugins, created a profile and logged in, creating a new publishing destination is done using this command:   sas-viya models destination createAzure --name Azure \    --baseRepoURL youracregistry.azurecr.io \             # the ACR login server URL   --tenantId a1b1c1d1-... \                             # Azure tenant ID   --subscriptionId 1a3b5c7d-.... \                      # Azure subscription ID   --resourceGroupName yourviyarg \                      # Azure resource group   --kubernetesCluster yourakscluster \                  # name of AKS cluster   --region yourregion \                                 # e.g. eastus   --validationNamespace validation-ns                   # namespace for validation   There is one additional parameter which has not been set in this example: “credDomainID”. The context help of the CLI explains this parameter like this:   --credDomainID, -c          Specifies the credential domain ID for the publishing destination.                             If this option is not specified, you are prompted to create a domain.   So, if you do not specify this parameter you will have to answer a few prompts which are needed to create the Credentials Domain for you. However, if you already have created the domain before (or if you want to re-use the domain for another publishing destination), you can simply include this parameter and refer to the existing domain by specifying the domain ID.   If you want to learn more about this powerful CLI plugin, head over to this Git repository which not only offers a lot of information related to SAS Model Manager but also includes examples of how to create publishing destinations using the CLI.   Wrapping up   That’s it for the configuration steps and you’re now ready for a test run. Why don’t you download some sample models from here, register the SAS models with Model Manager and then test the new publishing destination? We’ve recorded a short video clip showing the new SCR publishing destination “in action”.     Thanks for following us through this blog. We hope you found it helpful and don’t hesitate to reach out in case you have any question! ... View more

The ability to execute analytical models written in the Python language is not new for users of SAS Viya. This capability has been introduced in the 3.x version of the platform and as you would have expected it is still available on the latest release, which is now based on the Kubernetes Container orchestration platform.   However, the shift to a cloud-native architecture has completely changed how you set up the Viya platform and this of course also includes all optional add-ons such as enabling the PyMAS package. PyMAS is SAS’ DataStep2 interface to Python, allowing you to execute Python code using the SAS Micro Analytic Service (MAS) engine. MAS is the realtime engine of SAS and is used by three different SAS products:   SAS Model Manager SAS Intelligent Decisioning and SAS Event Stream Processing   This blog walks you through the steps of deploying and testing the PyMAS module on a SAS Viya 2020.1 release deployed on the Microsoft Azure cloud using Azure’s managed storage service (Azure Files) for storing the Python runtime. We’ll also show you how to use SAS Model Manager to publish a Python model to MAS and validate the this step after setting up the environment.   We recommend that you also take the time to review the PyMAS related documentation found in the MAS Programming and Administration guide. It explains the same steps as this blog but uses a NFS share instead of Azure Files for storing the Python runtime.   Getting started (some background)   In order to enable MAS to execute a model written in the Python language, a Python runtime has to be made available to the engine. On the Viya 3.x platform this simply meant that you had to install a Python environment next to MAS. Obviously this cannot be done in the same manner on the latest release which is fully containerized – installing software into a running container is a true anti-patten since instances should be treated as being ephemeral and it probably would also introduce a lot of security concerns if that option existed. On the other hand it wouldn’t make much sense if SAS would pre-install a Python runtime in the shipped container images – which version should it be, which Python packages should be made available?   The solution for this problem is as simple as clever: you don’t “install” the Python runtime, you simply attach a volume to the MAS container instance, which has the Python environment(s) you want to use. The volume is mounted to the instance’s filesystem like a network drive and SAS can use the Python runtime from there. This brings a lot of advantages: not only can you customize a Python runtime which is tailor cut for your needs but you also can easily swap in and out different versions when you apply updates.   The following picture explains the idea:   Figure 1: Using Azure Files for SAS on Kubernetes   The 3 pods we’re interested in are MAS, CAS and Compute (SPRE). They are deployed in a separate namespace (“viya4”) on a Kubernetes cluster which in our case has been created in the Azure cloud (thus it’s the managed Kubernetes service called “AKS”). “Outside” the Kubernetes cluster, but still part of your Azure subscription, we’re using the Azure Files service, which is a standard offering in the Azure cloud for providing shared storage to your virtual machines (and Kubernetes pods). Using the Files service we have prepared a fileshare which contains the Python runtime.   Basically 2 things are needed to make the magic happen. First, the pods need a way to attach the shared storage. This is done through the standard Kubernetes Container Storage Interface (CSI, see here for more information). Second, the SAS/CAS/MAS sessions need to know where to look for the Python binaries. This is done by setting environment variables which are provided via a Kubernetes ConfigMap object loaded into the pods.   Understanding Kubernetes storage concepts (on Azure)   Not trying to make you an expert in the Kubernetes storage architecture, but we still need to cover a few basics. One of the main characteristics of the Kubernetes design is its’ use of interfaces to shield applications from lower level details. This allows Kubernetes workloads to run on any public (and private) cloud without changes, because they do not need to know anything about the underlying infrastructure. Kubernetes will “translate” any resource requests (for CPU, for memory, for storage …) to the underlying layer.   What does that mean in terms of storage? Look at this picture:   Figure 2: Kubernetes storage concepts   Pods use external volumes by setting a claim (PVC) for a persistent volume (PV) which is the Kubernetes representation of a disk (or fileshare) on the underlying infrastructure. This can either happen dynamically or by referencing a disk / fileshare which is already existing. And this is exactly what we want.   One final word concerning the Azure Files service. This is an easy way to provide shared storage for virtual machines and Kubernetes pods. It comes in different flavours and you can choose the one that matches your requirements in terms of cost and performance. For our purpose you might want to consider the faster premium tier with provisioned capacity (using SSD drives under the hood) over the regular standard tier in order to reduce latency (see here for more information).   Now let’s walk through a sample deployment step by step.   Provisioning Python and customizing the Viya deployment   Please note: all commands and scripts have been shortened in order to improve readability. You can download the resource ZIP file attached to this blog containing all resources in full length.   Here’s a high-level overview of the steps we need to take:   Create a storage account and a fileshare in Azure Files Provision the Python runtime to the fileshare Prepare the kustomize patches needed for the Viya deployment Deploy and perform post-config steps   As said before, we do not rely on dynamic provisioning of storage, so we have to create the fileshare manually. It’s a one-time task and you can do this either using the Azure Portal or by using the Azure CLI (check the attached ZIP file for an example). Make sure that you retrieve the access key and store it as a Kubernetes secret – the pods will need it later when accessing the fileshare.   kubectl create secret generic <storage-account-secret> -n default \   --from-literal=azurestorageaccountname=$AZ_STORAGE_ACCOUNT \   --from-literal=azurestorageaccountkey=$AZ_STORAGE_KEY   Before we can install the Python environment to this fileshare, it has to be made available to Kubernetes (again a one-time task). We need a couple of Kubernetes objects for this: a StorageClass, a PersistentVolume and a PersistentVolumeClaim. All objects can be specified in a single YAML file, basically like this:   kind: StorageClass metadata:   name: python-sc                              <-- provisioner: kubernetes.io/azure-file            | parameters:                                      |        storageAccount: <storage-account>              |      ---                                              |    kind: PersistentVolume                           |      metadata:                                        |        name: pv-python                                |      spec:                                            |        claimRef:                                      |          name: pvc-python                   <--       |         storageClassName: python-sc            |     <--    azureFile:                             |       |          secretName: <storage-account-secret> |       |      shareName: <fileshare>               |       |  ---                                      |       |  kind: PersistentVolumeClaim              |       |  metadata:                                |       |    name: pvc-python                     <--       |  spec:                                            |    accessModes:                                   |      - ReadWriteMany                              |    storageClassName: python-sc                  <--   Note how these objects are connected to each other by their names. Pods can now refer to a PVC (“pvc-python”) in order to access the fileshare on Azure Files.   As a next step, we need to actually install the Python runtime on this fileshare. There are many options how to do this, in our case we’ll simply use a Kubernetes job which runs a shell script for us (this saves us from creating an additional virtual machine). Let’s assume we have a Linux script for installing a Python in version 3.8.6 and a requirements.txt for adding some additional Python packages (check the attached ZIP file for an example). To make them available to the Kubernetes job, we store them as a ConfigMaps object:   kubectl create configmap python-builder-script-3.8.6 \     --from-file=install-python-3.8.6.sh \     --from-file=requirements.txt   And then refer to this map in the job definition:   kind: Job metadata:   name: python-builder-job spec:   template:     spec:       containers:       - image: centos:centos7         name: python-builder-job         command: ["/bin/sh", "-c"]   <-- command to be executed (script takes         args:                            target folder for install as argument)           - /scripts/install-python-3.8.6.sh /python/python-3.8.6         volumeMounts:         - name: host-volume           mountPath: /python         <-- mountpoint of Python volume         - name: install-script           mountPath: /scripts        <-- mountpoint of scripts configmap       volumes:       - name: host-volume         persistentVolumeClaim:           claimName: pvc-python      <-- the claim we created earlier       - name: install-script         <-- the install script and requirements.txt         configMap:           name: python-builder-script-3.8.6                    defaultMode: 0755   Once this job has completed it’s work, we should be able to see our Python runtime in the Azure portal:   Figure 3: Checking fileshare contents in the Azure portal   That’s all what is needed for preparing the Python runtime for Viya.   Moving on to the Viya deployment, you now have to follow the steps explained in the README.md located in the $deploy/sas-bases/examples/sas-open-source-config/python/ folder. To quickly summarize:   Make a copy of these files in your site-config folder Edit the variables in site-config/sas-open-source-config/python/kustomization.yaml, e.g.   configMapGenerator: - name: sas-open-source-config-python   literals:   - MAS_PYPATH=/python/python-3.8.6/bin/python3.8  <-- /python is the default   - DM_PYTHONHOME=/python/python-3.8.6/bin         <-- mount point of the   - DM_PYPATH=/python/python-3.8.6/bin/python3.8   <-- fileshare   Add references to your Python volume claim to all relevant sections in site-config/sas-open-source-config/python/python-transformer.yaml. There are multiple sections in this file which you need to update, but the replacement is always identical. To give you an example:     # Add python volume   - op: add     path: /spec/template/spec/volumes/-     value:        name: python-volume       persistentVolumeClaim:         claimName: pvc-python             <-- the claim we created earlier   Edit your main kustomization.yaml to include your patches to the build:   resources: ... - site-config/sas-open-source-config/python transformers: ... - site-config/sas-open-source-config/python/python-transformer.yaml   We won’t touch this here but you also should to make some persistent storage available for ASTORE files. Luckily that’s a fairly simple step. It’s explained in $deploy/sas-bases/examples/sas-microanalytic-score/astores/README.md and also check this section in the Viya Operations Guide for instructions on how to do this: “Configure Model access”.   Now go ahead, build the main YAML file and deploy Viya as usual. Once your shiny new environment is up and running, don’t forget the final post-configuration step to enable publishing models to MAS. To run Python, one of the following is required on the CAS server for user authentication:   The user who is logged in must be a member of the CASHostAccountRequired group. For information, see The CASHostAccountRequired Custom Group in SAS Viya: Identity Management. (alternatively) the CASALLHOSTACCOUNTS environment variable must be set. For information, see env.CASALLHOSTACCOUNTS in SAS Viya: SAS Cloud Analytic Services.   Validating the deployment in SAS Model Manager   Now that we have completed the deployment steps we’ll want to make sure that everything works as expected. To keep it simple we’ll walk you through the steps of uploading a Python model to SAS Model Manager, publish it to the MAS destination and then validate this step using Model Manager’s built-in test functionality. A Python model and a sample dataset is provided in the attached ZIP file.   First you have to create a new SAS Model Manager project. A project can include different models and different versions of a model. We have collected all model files that belong to the model in a zip file: Metadata information about input and output variables, model property information, the train and score code, model fit statistics and the pickle file to apply the model to new data.   To upload this model to SAS Model Manager click on “Add models” and “Import” and pass the model ZIP file:   Figure 4: Import a Model   Clicking on “policy_claim_XGBOOST” lists all corresponding files of that model. There are separate sections for SCORE CODE, SCORE RESOURCES, VARIABLES and PROPERTIES AND METADATA: Figure 5: Model Files   To execute the Python code through the MAS engine a DS2 wrapper code is needed. Depending on the usecase you can either publish your model to MAS (realtime or transaction scoring) or CAS (batch scoring). In both cases the PyMAS package is used to execute the Python code, but different wrappers are needed:   The DS2 Embedded Process Code Wrapper is used for CAS execution The DS2 Package Code Wrapper is used for MAS execution   Figure 6: DS2 Embedded Process Code Wrappers to execute Python from the CAS and MAS engines   After you published your model to the MAS or CAS publishing destination, a validation test is generated for you automatically. You can find this validation test on the Scoring/Publishing Validation tab.   Figure 7: Publishing Validation   You need some sample data in order to run this test. We’ve provided a small dataset in the attached resource ZIP file for this blog. Upload this dataset to CAS, assign it to the test and then run it. It should return a green status, meaning that the validation test was successful for the Python model. Your model is now ready to “go live”.   Conclusion   This blog has walked you through the steps needed for providing a Python runtime to use the PyMAS package for the CAS and MAS engines when your Viya deployment is located in the MS Azure cloud and you plan to use Azure’s standard Files service for managed shared storage. Remember that there are instructions available in the MAS Programming and Administration guide when you want to use a NFS server instead. The idea to attach the Python runtime as a Kubernetes volume is as simple as clever. It provides you with the flexibility you need to keeping your Python up-to-date and tailor cut for your needs. Using the Azure Files service on the other hand provides you with a storage solution with zero management overhead.   ... View more

I should warn you that this post has a bit of a “nerdy” attitude to it ... I’ll describe how you can run a SAS programming-only Docker container on your local Windows machine and make SAS Enterprise Guide 8.2 connect to it as a client. This certainly sounds like fun (at least to me), but it also has a more serious point.   Over the years, I’ve met many customers with decentralized SAS environments where users run SAS sessions on their Windows desktops, enjoying the maximum control of local SAS workspaces. On the other hand, there’s the IT perspective – the challenging task to keep track of a large user base working with decentralized SAS deployments. How do you:   Maintain a standardized and up-to-date SAS environment? Keep users informed if changes are needed for shared data sources, such as NFS shares or databases? Simplify the annual process of renewing the licenses? Over the years, SAS administrators have found many different ways to ensure governance and support in scenarios like these. In this post, I’d like to describe an additional approach, not claiming that this should replace other solutions. Take it as it’s meant – just “food for thought” – and see if it's reasonable for you or not.   Running a SAS workspace in a container   As you probably are aware, SAS is currently undertaking a major change in the way we build and distribute our software. Listening to customers increasingly demand a shift to cloud-native architectures, SAS is putting great effort into containerizing the SAS platform (mainly SAS Viya, but this also applies to the SAS9.x stack) with the goal of offering container images (“Docker images”) as the units of deployment. For this blog I am focusing on the feature set of a SAS Viya programming-only container image, which means that we will have both the CAS and the SAS runtime engine at our disposal. Usually SAS Studio will be the default user interface for this type of container, hence the label "programming only".   There are multiple ways how to create a SAS Viya Docker image: you might want to download an existing container image prepared by SAS or you might prefer to give it a try and “roll your own.”(Hint: head over to @joeFurbee’s excellent article on Getting started with SAS Containers if you feel the need for a refresher on the topic before continuing.)   Creating a container image from scratch is a bit more involved, but it is certainly the more rewarding approach if you’re interested in learning about SAS in containers and Docker in general. For this approach you will need a Linux-based machine to create the Docker image. This node can also host the Docker registry from which you can “pull” your image to your Windows machine.   For the sake of keeping this short, I’m going to assume you are familiar with the basic steps of how to set up Docker and a Docker registry on Linux. I’ll also refer you to the wiki pages of the public Github project hosting the SAS provided container recipes, which explain how to create a Viya programming-only runtime container image.   If you work with the container recipes, I’d recommend adding the “auth-demo” overlay to the base image so that you later can specify the user account which you want to use for logging in. Note that the “addons” folder in the recipes collection contains a lot of examples of how to include support for various SAS/Access engines as well. Here’s a sample command that will build your container image and push it to your local Docker registry:   ./build.sh \ --type single \ --zip ~/path/to/SAS_Viya_deployment_data.zip \ --docker-registry-url myregistry.myhost.com \ --docker-registry-namespace sas-poac \ --addons "auth-demo" After this has completed, it’s time to switch to the client side of things.   Docker for Desktop   Installing Docker for Desktop is the first thing you need to do. The installation should be straightforward, however, be aware that installing the Docker software will disable other virtualization software on your local machine such as VMWare or Virtualbox.   Now that you have Docker for Desktop running on Windows, open a Powershell prompt to download (“docker pull”) the container image you created before. Depending on the addons you had included, the startup command for your container (“docker run”) might be different. Here’s a typical example of both steps (note the port mappings):     Ok, give it a try! Launch a browser and point to http://localhost:8080 (or whatever port you’ve chosen) – you should be welcomed by SAS Studio. Login using the username and password you specified and you’re good to go.   While this is already pretty cool, you’ll soon discover that it has limitations because everything inside the container is basically ephemeral – meaning that it is lost if the container is stopped. Docker for Desktop allows you to map Windows drives to the container’s filesystem, so this might be a solution for storing your SAS codes and data. However, in my experience, this feature is not fully stable and sometimes corporate security policies will prevent you from using it altogether.   SAS Enterprise Guide 8.2   Oh my, what now? We’ve been so close! But wait, what if we had a client outside the container that could connect to the engines (SAS and CAS) inside? So that all your programs are safely stored locally on your desktop? And the same should be possible for data as well, of course.   Enter SAS Enterprise Guide 8.2. The latest release of Enterprise Guide features one particular capability which makes it well suited for working with containers: it can directly connect to a remote SAS server using hostname and port - in our case: localhost:8591.     Once you have added this connection to the list of SAS Workspace servers (again use the user account that you specified when launching the container), you can use Enterprise Guide to submit your programs to the SAS and CAS engines in the container. And while your code is executed by the SAS or CAS engine inside the container, it is safely stored on your local Windows machine.     And what about data? Nothing special here: You can import local data in the same way you are used to. Simply go to “File/Open” and select the file you want to upload to your container. Just remember to “export” output tables back to your local machine before you shut down your container.       A little bonus at the end   I’d like to end with a little bonus tip: Enterprise Guide is not the only client that can be used to work with the SAS container – Python can do this as well with the help of the SAS SWAT package (the “SAS Scripting Wrapper for Analytics Transfer”). For example if you have installed the Anaconda distribution on your laptop and added SWAT (a simple “pip install swat” might be all that is needed), then the Jupyter notebook application can be your client for running analytics on the CAS server in your container. Note: For this to work you need to set the port mappings when launching your container (the CAS ports 5570 and/or 8777 need to be available from outside the container).       Conclusion   Above, I described a way to connect to a local SAS programming environment running in a Docker container by using the latest SAS Enterprise Guide release (8.2). As I mentioned at the beginning, this can be a mere fun exercise for those of you interested in learning more about SAS and containers, but there might also be some real benefit to it in decentralized environments where supporting a large SAS user base can be challenging.   I hope you liked the post and I’m curious to hear your feedback. Please feel to add your thoughts and comments below! ... View more

I have been reading about ways to access S3 data via NFS, and wondered if it was possible to use Amazon’s Athena as a data extraction interface to S3. Athena supports ODBC/JDBC, so it should be viable. So how would you read data from S3 via Athena in a simple “delta loading” data ingestion scenario?   What is Amazon Athena?   Amazon Athena is a query service specifically designed for accessing data in S3. It is one of the core building blocks for serverless architectures in Amazon Web Services (AWS) and is often used in real-time data ingestion scenarios (e.g. IoT cases).   There are two major benefits to using Athena. First, it allows you to use standard SQL for data retrieval, which greatly simplifies data ingestion for raw data in S3. Second, it’s a serverless service which means that all infrastructure details are hidden. As a customer, you focus on using the service and AWS will manage everything else. Athena use is charged on a “per query” basis and the costs depend on the amount of data scanned.   How can you use Amazon Athena to retrieve data from S3?   There are basically two ways of using Athena from a third party client (SAS in this case): JDBC and ODBC. I chose the JDBC route because it seemed simpler. Athena is easier to understand if you’re familiar with Apache Hive and Hadoop. Like Hive, Athena is a schema-on-read query engine. This means that you store your data without needing to describe its layout (or schema). The schema is provided later, when you need to read the data.   The data is already available in S3 when you want to read it, so your schema definition (the CREATE TABLE statement) needs to point to the storage location. Athena therefore uses the same concept of EXTERNAL tables with a LOCATION attribute as Hive. Like Hive, Athena also treats your data in S3 as read-only, so only SELECT queries are supported.   Finally, Athena treats folders in S3 buckets very like Hive treats folders in HDFS: all data files in a folder or subfolders are considered to belong to the table. If the subfolders fit a certain naming pattern, they are treated as partitions, and this can be leveraged to optimize query performance.   Accessing S3 data via Amazon Athena using Python and SAS   I set up a new EC2 instance which served as a client to Athena. When clicking through the setup wizard, I created and assigned an IAM role, allowing the machine to access both the S3 and Athena services:       I would recommend assigning an IAM role at this point, because it simplifies the process of accessing S3 and Athena by avoiding the need for later client authentication. This is particularly helpful if you are using federated accounts.   Create a bucket and upload some data   The next step is to upload some data to S3. I’ve been using JSON-formatted sample data for my tests. This data has a timestamp attribute which I used for splitting the JSON files (one file per day). You can either use the S3 GUI for uploading or use the AWS CLI, which is an easy way to test if your IAM role is working.   To install the AWS CLI: python -m pip install awscli --upgrade   Now let’s create a bucket and upload some data: aws s3 mb s3://gerhje-test-bucket-123/ --region us-east-1 --endpoint-url https://s3.us-east-1.amazonaws.com make_bucket: gerhje-test-bucket-123 aws s3 cp testfile-0.json s3://gerhje-test-bucket-123/rigdata/rec_dt=2017-06-29-00-00/ upload: ./testfile-0.json to s3://gerhje-test-bucket-123/rigdata/rec_dt=2017-06-29-00-00/testfile-0.json aws s3 cp testfile-1.json s3://gerhje-test-bucket-123/rigdata/rec_dt=2017-06-29-01-00/ upload: ./testfile-1.json to s3://gerhje-test-bucket-123/rigdata/rec_dt=2017-06-29-01-00/testfile-1.json aws s3 ls 2019-03-10 15:24:03 gerhje-test-bucket-123   I stored the data in a folder hierarchy made of a root folder (“rigdata”) with subfolders (“rec_dt=XYZ”). These subfolders will be treated as data partitions in Athena. The reason for this approach is that your (external) source data provider systems is assumed to constantly upload data to S3 and you need an efficient data ingestion strategy using delta loads.   Create Athena metadata for accessing the S3 data   With the data in place, you can now head over to the Athena GUI in the AWS web console . The first step is to run a CREATE DATABASE statement:    create database rigdb   Now define the rigdata table, pointing to the S3 data you have just uploaded: CREATE EXTERNAL TABLE IF NOT EXISTS rigdb.rigdata (   rig            STRING,   well_depth     INT,   bit_depth      FLOAT,   recorded_at    TIMESTAMP,   status_code    INT,   status_message STRING ) PARTITIONED BY (rec_dt STRING) ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' WITH SERDEPROPERTIES ('ignore.malformed.json' = 'true') LOCATION 's3://gerhje-test-bucket-123/rigdata/'   Again, if you’ve worked with Apache Hive, this will look very familiar: PARTITIONED BY: this declares that the table uses partitions. The subfolder names will be treated as a virtual column called “rec_dt” ROW FORMAT SERDE: the source data is held in the JSON format, so you need to specify an appropriate serde (“serializer-deserializer”) class LOCATION: this points to the top folder in S3 where the data is stored     If you submit this statement, the GUI will warn you that you need to update the table metadata to make Athena aware of existing partitions. The easiest way to do this is to run this command: MSCK REPAIR TABLE rigdb.rigdata   This will load all partitions at once. Of course, in real life, a data ingestion strategy using delta loads would use a different approach and continuously append new partitions (using an ALTER TABLE statement), but it’s probably best not to worry about that at this stage. It is then worth checking if this has worked: SELECT count(*) as c, rec_dt FROM rigdb.rigdata GROUP BY rec_dt     Set up a JDBC connection from SAS to upload the S3 data to CAS   As a last preparation step before you can finally switch to SAS, you need to download the Athena JDBC driver to the EC2 instance: yum -y install wget java mkdir -p /opt/sas/athena-jdbc cd /opt/sas/athena-jdbc wget https://s3.amazonaws.com/athena-downloads/drivers/JDBC/SimbaAthenaJDBC_2.0.7/AthenaJDBC41_2.0.7.jar   With this in place, you can now move to SAS Studio and use the test code to read the S3 data: libname athena clear; libname athena jdbc schema="rigdb"    driverclass="com.simba.athena.jdbc.Driver"    URL="jdbc:awsathena://AwsRegion=us-east-1;S3OutputLocation=s3://gerhje-test-bucket-123/athena-results/;AwsCredentialsProviderClass=com.simba.athena.amazonaws.auth.InstanceProfileCredentialsProvider;Schema=rigdb"    classpath="/opt/sas/athena-jdbc/AthenaJDBC41_2.0.7.jar"; cas mySession sessopts=(caslib=casuser timeout=1800 locale="en_US"); caslib _all_ assign; proc casutil;    load data=athena.rigdata outcaslib="public"    casout="rigdata"; run; cas mySession terminate;     The most interesting piece in this code is the JDBC URL= parameter. It contains quite a lot of information: AwsRegion: this points to the AWS region where you set up your Athena database and table (usually “us-east-1” as a default). S3OutputLocation: this specifies a temporary folder which Athena can use for storing query results. AwsCredentialsProviderClass: one of the pluggable credential provider classes. “InstanceProfileCredentialsProvider” is the one to choose if using an IAM role attached to an EC2 instance. Schema: the Athena database to use. Note that you need to repeat the schema a second time in the libname statement.   Moving to delta loading   You should be able to see that the existing S3 data has been successfully loaded to CAS by now. But what happens if you want to add another slice of data?   Upload the data to S3 aws s3 cp testfile-2.json s3://gerhje-test-bucket-123/rigdata/rec_dt=2017-06-29-02-00/   Update Athena’s metadata to recognize the new partition ALTER TABLE rigdb.rigdata ADD PARTITION (rec_dt='2017-06-29-02-00') LOCATION 's3://gerhje-test-bucket-123/rigdata/rec_dt=2017-06-29-02-00/'   Append the new slice of data to your CAS table data public.rigdata(append=yes);    set athena.rigdata(where=(rec_dt='2017-06-29-02-00')); run;   Check the results   Conclusion   There are, of course, many other options available for SAS to read data from AWS S3. Using a simple filename statement or using PROC S3, CAS in particular can even read CSV and SASHDAT files directly from S3. There are also the NFS options that I mentioned at the start of this blog.   The approach I have described simply gives another option. Depending on the task you need to solve, it may be useful. I think the main benefits of using Athena are its flexibility in dealing with raw data formats and its support for filtering and partitioning. This is particularly helpful in delta loading scenarios.   I hope this has provided some useful and interesting information. Please do feel free to send me your feedback! ... View more