Building Performance Monitoring Processes in SAS Workflow Manager
SAS Workflow Manager is a tool on SAS Viya that allows users to build out defined processes for their analytical lifecycle. These workflows consist of a defined series of tasks. SAS Workflow Manager can systemically route tasks requiring manual intervention while automatically triggering system tasks and alerts. This allows teams to embed manual checks-and-balances while automating where they can. The processes built into SAS Workflow Manager can be fully customized to fit your business procedures. In this post, we will start by understanding the components of SAS Workflow Manager before moving on to create a performance monitoring workflow.
Understanding the Components of SAS Workflow Manager
Before we can start building a workflow, we need to understand the available components. This allows us to define a workflow to match our current or ideal business process.
Workflow Definition
A workflow definition holds all the pieces necessary to run the workflow. It defines the properties, components, and flow that make up a specified business process.
Data Objects
Data Objects are the variables that are associated with a workflow process. Values are associated with Data Objects within the workflow definition, from users, or from services. The values stored in Data Objects can be used in conditional flows or passed into services.
User Tasks
User Tasks are those performed manually. In the analytical lifecycle, these tasks can include developing models or approving models for production. These tasks can help keep humans in-the-loop to ensure that the process is running smoothly and allows teams to step in if there is a problem. User Tasks can be assigned to specific users or groups and once assigned can be found in the tasks section of SAS Model Manager.
Prompts
Prompts can be added to the User Task or Start Node and allow for a user to enter a value to be stored in a Data Object.
Service Tasks
Service Tasks are those the system performs automatically. There are various types of service task options including invoking a web service via REST or SOAP, sending an email using the SAS Viya Mail Service, sending a notification using the SAS Viya Notification Service, or invoking a job defined in the SAS Job Execution service.
Subprocesses
Subprocesses are workflows within workflows. Subprocesses help modularize and reuse logic to improve readability of the workflow.
Gateways
Gateways allow for conditional logic flow through the workflow. What happens next in the workflow can be controlled by the value of a Data Object. This allows for error handling and looping through tasks that need to be repeated.
Timers
Timers wait a set period before moving forward. This is helpful for waiting for underlying dependencies to become available or for processes that have a cyclical nature. One great use case for timers is for cases when data becomes available once a month. A timer prevents the process from moving forward until the data has arrived.
Creating Your Performance Monitoring Workflow
Now that we understand the various components that make up a workflow, let’s build one. All models decay over time, but knowing when a model has decayed is important. A decayed model needs to be refreshed quickly to ensure an organization can maximize their analytical investments. In a few simple steps, we will create the workflow below to alert a ModelOps team when model performance degrades past a threshold.
This is a simple example using KS statistic, but using the SAS REST API guide, there are a lot of areas where you can expand this process. I’ve included a few suggestions below for additions to harden this process, so feel free to use example as a starting point.
Step 1. Administration
Before getting started, ensure that the default workflow administrator group is configured. Also ensure there is an account configured for the workflow client .
Step 2. Create Definition
Now we will create a new workflow definition. From SAS Workflow Manager, select ‘New Definition’ from the upper right-hand corner. Name the workflow and select ‘OK’.
Step 3. Set the Client Identifier and Create Data Objects
After the workflow is defined, the workflow workspace becomes available. In the upper right-hand corner, select the button for ‘Definition Properties’. Fill out the client identifier and data objects as follows:
The data objects reference information we will need for this workflow including: the project name, the performance threshold, the performance definition ID, the latest performance statistic, and the project ID. We will gather some of this information from the user and others from the system.
Step 4. Add Prompt to Start Node
Next select, the start node. In the options menu, select ‘Prompts’. Add the following two prompts to ask the user for the name of the SAS Model Manager project as well as a performance threshold.
Step 5. Add User Tasks to Define Performance Definition
Drag a User Task onto the canvas and connect the Start Node to it. This tasks it to remind the user to ensure the dependencies for performance monitoring are in place, specifically a defined champion model within the SAS Model Manager project as well as a performance definition. Information about setting a champion model is available here and information about creating a performance definition is here.
Step 6. Use Service Tasks to Grab ID of Model Project and Performance Task
Next, add two service tasks. The first task will get the ID of the model project and the second will get the ID of the performance definition. We will store these values into data objects for use in our future system tasks.
Select the first service task, ‘Get ID of Model Project’, and assign the following values in the ‘Actions’ menu on the left:
|
Action |
Invoke REST web service |
|
|
URL |
/modelRepository/projects?filter=eq(name,'${ProjectName}')&sortBy=name |
|
|
Method |
GET |
|
|
HTTP request headers |
Accept |
*/* |
|
Expected result code |
200 |
|
|
Output type |
JSON |
|
|
Output data objects |
ProjectID |
items[0].id |
Additional information about the API used in this task can be found here.
In the second service task, ‘Get Performance ID’ assign the following values in the ‘Actions’ menu:
|
Action |
Invoke REST web service |
|
|
URL |
/modelManagement/performanceTasks?filter=eq(projectId,'${ProjectID }') |
|
|
Method |
GET |
|
|
HTTP request headers |
Accept |
*/* |
|
Expected result code |
200 |
|
|
Output type |
JSON |
|
|
Output data objects |
PerformanceDefinitionID |
items[0].id |
Additional information about the API used in this task can be found here.
Step 7. Add Service Task to Execute Performance Job
Now that we have all the required information, we can execute the performance job. Next, add a third service task to execute the performance definition.
Set this node up to the following specifications:
|
Action |
Invoke REST web service |
|
|
URL |
/modelManagement/performanceTasks/${PerformanceDefinitionID} |
|
|
Method |
POST |
|
|
HTTP request headers |
Accept |
*/* |
|
Expected result code |
201 |
|
|
Output type |
JSON |
|
Information about this API is available here.
Step 8: Add Service Task for Get Performance Results
Executing a performance job may take a varying amount of time depending on your data sizes and the resources you have available. I have added a timer to wait for the job to execute before my last service task. You may need to experiment with how long this timer should be, but you can also add another service task to check the status of the performance execution before moving forward to harden this process.
Set up the ‘Get Performance Results’ node like the following:
|
Action |
Invoke REST web service |
|
|
URL |
/modelManagement/performanceTasks/${PerformanceDefinitionID}/performanceReportData |
|
|
Method |
GET |
|
|
HTTP request headers |
Accept |
*/* |
|
Accept-Item |
application/vnd.sas.models.performance.task.report.data.ks.statistic+json |
|
|
Expected result code |
200 |
|
|
Output type |
JSON |
|
|
Output data objects |
LatestPerformance |
items[-1].statistic |
I am utilizing the KS statistic, but other options are available in the documentation for this API.
A note for testing this node: if you notice your workflow getting hung up on this step after testing it a few times, you can refresh the definition in SAS Model Manager. This node will return a warning if no new data is available for performance monitoring (if you want to further solidify this process, add a task that checks if new data is available before executing the performance job).
Step 9. Create Loop for Monitoring Performance
Next, add an exclusive gateway that points to a timer. This timer will represent the amount of time it takes for fresh data to be retrieved. Select the arrow pointing to the timer from the gateway and add the following condition:
Step 10. Create User Task to Review Performance Report
From the gateway, add a user task to review the performance report and an end node. Now, we have a completed workflow for performance monitoring!
Step 11. Save and Run Workflow
The last step is to save the workflow, create a new workflow version, and enable the workflow version. These three buttons are available in the upper right-hand corner next to the ‘Definition Properties’ button. Now you should be able to see a workflow tab in your SAS Model Manager projects where you can start your workflow!
Conclusion
Workflows allow users to build out defined processes for their business procedures. Using the available components, a ModelOps team can create a process that automates where possible while ensuring human checks-and-balances.
Interested in learning more about using APIs on SAS Viya? Check out these great posts:
Hello @Dmitry_Alergant!
I personally don't know of a way to add a custom client identifier, but in the recent releases of SAS Viya, Intelligent Decisioning is added to the set of client identifiers. I couldn't find the documentation listing its availability, but I am using SAS Viya 2021.1.1 and this is my view:
Additionally, you can see several of the available SAS Viya APIs here. The APIs specific to Workflow Manager can be found here, which is under Decision Management -> Model Management -> Operations.
