My First SAS Studio Flow Custom Step

5 Likes

The SAS Studio Flow “Custom Step” framework has just been released in SAS Viya 2020.1.5. After the advent of SAS Studio Flows, Custom Steps are a big step forward in providing visual and flexible data manipulation/integration jobs with shared reusable components in the new SAS Viya.

What is a Custom Step?

“A custom step enables you to create a user interface for users at your site to complete a specific task.” -- SAS Studio documentation

Custom Steps in SAS Studio Flows are the descendants of the custom “Transformations” in SAS Data Integration Studio (using the “Transformation Generator”), “Custom Tasks” in SAS Enterprise Guide and SAS Studio, and even “Java Plug-Ins” in former SAS ETL Studio! Do you remember?

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

I wanted to try the new framework as soon as possible and I was looking for a practical example to test it. SAS Studio Flows is (as of now) “SAS Compute Server”-centric, meaning the runtime is a SAS Compute Server and objects (tables) that can be dropped on the canvas must be accessible through a SAS Compute Server.

But I love CAS! And following up a discussion about loading database tables directly and efficiently in CAS, without moving them through a SAS Computer Server, I wanted to experiment “Custom Steps” to see:

If I was able to perform an efficient loading in CAS of a database table, driven by a SAS Studio Flow and a Custom Step
How easy/difficult it was to use the Custom Step
How easy/difficult it was to design the Custom Step

I can already say it. The overall experience was very good. This framework is promising.

Let’s review my adventure.

Assumptions

Of course, I had to define some requirements. SAS Studio Flows cannot see natively CAS data source files (the physical files located in a CASLIB’s source path/database). Instead, we can use SAS/ACCESS traditional libraries to allow users to drop database tables on the SAS Studio Flow canvas.

We can just define a naming convention like this one:

The SAS/ACCESS library pointing to the database will be named SAS_XXX
The CASLIB pointing to the exact same database will be named CAS_XXX

We will manage this simple rule in the code and voilà.

For the target table, we needed to make the user define the CAS table. Thus, we had to assign CAS engine libraries (on the SAS Compute Server) pointing to existing CASLIBs so that the user can choose the appropriate target CASLIB. Again, you can make an assumption that the CAS engine library name will be identical to the CASLIB name to simplify your code. Or, better, you can derive the CASLIB name from the CAS engine library name. Keep in mind that CASLIB names can be longer than 8 characters and may not be mapped in a SAS Compute Server session because of that.

We have defined the requirements. Let’s create the Custom Step.

The Prompt UI

The Prompt UI is where you define what the user will see when (s)he will be using the Custom Step. It’s a JSON string, pretty easy to write, especially when you look at the examples provided out-of-the-box:

In my case, I was defining the following inputs (you can preview your UI while designing):

Of course, input and output tables will not show up in the real UI but will rather be using the input and output ports. You can be creative and define parameters that are the most relevant to your case. Here, I wanted to allow the user to use multi-node load, define a slicing column, manage options like replacement or promotion and also the number of copies in CAS for failover. We could have added bulkload or any option of interest as well…

From a JSON perspective, it looks like that:

Each id corresponds to a macro-variable name (or to a set of macro-variables' names).

You have many options to create rich user interfaces (multiple tabs, sections, etc.). Some features are on the roadmap like dynamic lists, cascading prompts and so on.

The Template

The Template is where you define the SAS code that will perform the task your Custom Step is supposed to do. In my case, it’s a direct loading of a database table in CAS taking into account options about multi-node loading, replacement, promotion and failover.

The main piece of code is this:

Yes. PROC CASUTIL to directly load my database table into CAS!

Of course, in our case, we needed to manipulate a little bit the input and output tables’ parameters using SAS code:

Derive the CASLIB name from the SAS/ACCESS library name (using the naming convention SAS_XXX => CAS_XXX)
Derive the CASLIB name from the CAS engine library name (the good old SASHELP.VLIBNAM has helped)

The Custom Step framework provides the designer some helpful information about the input parameters available as SAS macro-variables, especially for table (name, label, engine, etc.) and column (name, type, format, etc.) selectors.

Example:

And that’s it! Very straightforward to design a Custom Step! From a design perspective, I was glad I could develop a Custom Step in such a small amount of time (1-2 hours depending on how robust you want the code to be).

Usage

Let’s see how to use it in practice:

I found it very easy to use the Custom Step. There was nothing special to know or to adapt to make this transform work even in my “mixed” runtime (SAS/CAS) scenario.

Saving and Promoting

Custom Steps are saved in SAS Content and thus can be secured (enabled for some teams, disabled for some others for example). Promoting them from environment to environment is already available! This means we can start to build and make available libraries of Custom Steps!

Conclusion

The Custom Step framework is a nice and cool feature and will get very popular in the next months. It allows customers to build complex business or technical transformations using SAS code and make them available to end users using mini-UIs. This will offer a huge productivity increase to the SAS Viya users’ community.

Thanks for reading.