BookmarkSubscribeRSS Feed

DataOpsWeek - Orchestration

Started ‎03-01-2024 by
Modified ‎03-01-2024 by
Views 218

Introduction

 

In January 2024, we held a #DataOpsWeek event with the aims of showcasing our work, learning from our colleagues and building the SAS DataOps community. Each day had a theme, based on Alexey Vodilin's 5 pillars for a successful DataOps strategy blog. Our second theme of the week was "Orchestration".

 

Orchestration.png

 

Orchestration

 

With SAS Viya, we are redefining the boundaries of how you think about Orchestration strategies on SAS products. Aside from the out of the box scheduling capabilities of SAS Viya, we have now support a integration with Apache Airflow. A lot of our talks on the day focussed on how this can be exploited to cater expecially for complex scenarios and multi-integration touch points.

 

Alexey explains in his blog why it is important to think about a holistic picture when you are defining Orchestration principles:

"It's not enough to produce siloed artifacts and share them in a central repository. To solve a specific business use case, artifacts of several types should be seamlessly orchestrated together, starting from a successful data ingestion, proceeding with building or retraining a machine learning model, and ending up with communicating insights back to the business application.

 

This is where orchestration technologies come into play. Orchestration processes allow integrating assets of multiple types (like data transformation flows, models, API calls, etc.) in a value pipeline that can serve as a single pane of glass for understanding, building and monitoring a specific process. As such, orchestration flows should also be versioned and shareable."

 

Top Orchestration resources 

 

I have compiled a list of top resources which will support Orchestration enablement for your teams.

 

Airflow:

Workload Management:

Argo Workflows:

SAS Studio Custom Steps:

Python:

Reinforcement Learning:

 

Summary

 

Finally, if I had to choose just one orchestration tool that we mentioned that day, it would definitely be Airflow. I’m impressed with the constant development of our SAS Airflow Provider and the way we can use Airflow with SAS Viya. Not only can we schedule and run SAS Flows, but we can also use it to automate some administration tasks! 

 

I am also looking forward to further conversation with exploiting Viya Workload Management to know how you manage your costs and infrastructure workloads. 

 

You can find resources from our other DataOpsWeek themes here:

  1. Collaboration
  2. Orchestration (this blog) 
  3. Continuous delivery
  4. Testing automation
  5. Environmental management
Version history
Last update:
‎03-01-2024 09:43 AM
Updated by:

sas-innovate-2024.png

Don't miss out on SAS Innovate - Register now for the FREE Livestream!

Can't make it to Vegas? No problem! Watch our general sessions LIVE or on-demand starting April 17th. Hear from SAS execs, best-selling author Adam Grant, Hot Ones host Sean Evans, top tech journalist Kara Swisher, AI expert Cassie Kozyrkov, and the mind-blowing dance crew iLuminate! Plus, get access to over 20 breakout sessions.

 

Register now!

Free course: Data Literacy Essentials

Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning  and boost your career prospects.

Get Started

Article Tags