Introduction   In January 2024, we held a #DataOpsWeek event with the aims of showcasing our work, learning from our colleagues and building the SAS DataOps community. Each day had a theme, based on Alexey Vodilin's 5 pillars for a successful DataOps strategy blog. Our third theme of the week was " Continuous delivery".     Continuous delivery   Alexey says: "Finally, the speed and quality of content promotion between development and production environments are crucial in accelerating the analytics lifecycle. Implementing an automated continuous integration/continuous delivery (CI/CD) approach (in cooperation with testing automation) reduces the time needed to deliver and embed data pipelines, machine learning models and decisions into production and reduces the probability of errors caused by manual interventions.   Though automation is crucial here it should also be possible to incorporate the functionality of manual approvals and enable additional validation as part of this process. The ability to version continuous delivery pipelines should also be supported."   Top Continuous delivery resources    Webinar: Applying Continuous Delivery to Industrialize Data Pipeline Deployment – Neil Griffin, Kunal Shah, Alexey Vodilin DevOps: Powering disruption: A DevOps journey at SAS - Billy Dickerson, Connie Dunbar Azure DevOps: Configure a SAS Viya Self-Hosted Agent with Azure Pipelines – Bogdan Teleuca How to Automate Model Deployment with SAS Viya and Azure DevOps – Bogdan Teleuca How to Create a SAS Viya Docker Agent for Azure DevOps – Bogdan Teleuca Bitbucket: DevOps Lifecycle with SAS Viya Studio Git Plug-in and Bitbucket – Achal Patel SAS Studio: How To Get the Code From a SAS Studio Flow and Why It Matters – Bogdan Teleuca How to Get the Code From a SAS Studio Flow Using Python – Bogdan Teleuca SAS Viya CLI: The Easiest Way to Run a SAS Studio Flow in Batch – Bogdan Teleuca SAS Visual Analytics: Storing web pages on GitHub for consumption inside the SAS Visual Analytics Data-Driven Content – Xavier Bizoux ModelOps: Get started with ModelOps now – Sophia Rowland SAS Container Runtime: Easily Promote and Use Containerized Model Deployments – Sophia Rowland   Summary   The day brought us many interesting discussions about automation, the reasons why we do it, or why we still choose to apply changes manually (we’d love to hear about your experiences!), and lots of useful content to help you build your own pipelines. But to sum up the day, I’ll borrow Jan Kostrubiec’s advice:     You can find resources from our other DataOpsWeek themes here: Collaboration Orchestration  Continuous delivery (this blog) Testing automation Environmental management ... View more

Introduction   In January 2024, we held a #DataOpsWeek event with the aims of showcasing our work, learning from our colleagues and building the SAS DataOps community. Each day had a theme, based on Alexey Vodilin's 5 pillars for a successful DataOps strategy blog. Our second theme of the week was "Orchestration".     Orchestration   With SAS Viya, we are redefining the boundaries of how you think about Orchestration strategies on SAS products. Aside from the out of the box scheduling capabilities of SAS Viya, we have now support a integration with Apache Airflow. A lot of our talks on the day focussed on how this can be exploited to cater expecially for complex scenarios and multi-integration touch points.   Alexey explains in his blog why it is important to think about a holistic picture when you are defining Orchestration principles: "It's not enough to produce siloed artifacts and share them in a central repository. To solve a specific business use case, artifacts of several types should be seamlessly orchestrated together, starting from a successful data ingestion, proceeding with building or retraining a machine learning model, and ending up with communicating insights back to the business application.   This is where orchestration technologies come into play. Orchestration processes allow integrating assets of multiple types (like data transformation flows, models, API calls, etc.) in a value pipeline that can serve as a single pane of glass for understanding, building and monitoring a specific process. As such, orchestration flows should also be versioned and shareable."   Top Orchestration resources    I have compiled a list of top resources which will support Orchestration enablement for your teams.   Airflow: SAS Viya and Apache Airflow in Kubernetes: A Peaceful Coexistence – Nicolas Robert What’s New in SAS Airflow Provider – Nicolas Robert Waiting for Something to Occur before Triggering SAS Jobs: Airflow Sensors – Nicolas Robert Using Apache Airflow to automate SAS Viya administration tasks – Gerry Nelson Workload Management: Scaling to new heights. Exploring the auto-scaling capabilities of Viya Workload Management - Eoin Byrne Argo Workflows: SAS administration command-line interface in a container: Part 2 Kubernetes and Argo Workflow – Gerry Nelson SAS Studio Custom Steps: Switch on, switch off: run-time control of SAS Studio Custom Steps - Sundaresh Sankaran Python: Python Tools for SAS Viya Reinforcement Learning: How we removed hurdles to adoption - Sundaresh Sankaran   Summary   Finally, if I had to choose just one orchestration tool that we mentioned that day, it would definitely be Airflow. I’m impressed with the constant development of our SAS Airflow Provider and the way we can use Airflow with SAS Viya. Not only can we schedule and run SAS Flows, but we can also use it to automate some administration tasks!    I am also looking forward to further conversation with exploiting Viya Workload Management to know how you manage your costs and infrastructure workloads.    You can find resources from our other DataOpsWeek themes here: Collaboration Orchestration (this blog)  Continuous delivery Testing automation Environmental management ... View more

In my previous SAS Communities article, I explained how to use the Great Expectations - Execute Business Rules custom step to validate your data for min/max values, matching patterns, etc. But what if you need more advanced business rules that are not included in the previous custom step? Or you have a validated dataset and want to generate a set of rules that describe that dataset and reuse those rules on newer data? You might want to try two other Great Expectations custom steps: GE - Generate Expectation Suite and GE - Run Expectation Suite.   I’ve recorded a short demo to show you how you can use the GE - Generate Expectation Suite custom step to automatically generate rules based on a "train" dataset, and the GE - Run Expectation Suite custom step to use the generated rules to validate another dataset. All in two simple SAS Studio flows!   ... View more

One of the important steps in working with data is to ensure that the data is correct and follows the specific business rules that you need. And it’s always good to know how difficult it is to define those rules and to ensure that the data is in line with them. While I’ve been working on some Data Quality related cases, I’ve heard a lot about the Great Expectations package, so I decided to give it a try and just see what it can do for us. And how does this apply to SAS Viya? Let’s see.   I started with our great Custom Steps GitHub repository, just to see what’s already been done with Great Expectations – quite a lot! We’ll start with something nice and simple – do you need to validate your data in terms of min/max values, matching patterns, missing values, etc.? Then Great Expectations - Execute Business Rules custom step is for you. I’ve recorded a really simple video to show you how to use this custom step and what we can get using it:   ... View more