Extract NLP Rule Configuration through a Custom Step.

2 Likes

Rules comprising classification and information extraction (concept) models in SAS Visual Text Analytics (on SAS Viya) live a cloistered life.

Natural Language Processing (NLP) developers interact with these rules through Model Studio, a GUI facilitating easy model development without the need for programming. These rules are persisted inside SAS Viya's solution database as well as the caslib linked with every text analytics project.

At the same time, AI stakeholders demand better understanding and transparency on the logic & machinations of analytics models.

Rule configuration tables provide rich metadata facilitating the same. Even metadata deserves its day out in the sun, sometimes.

Rule configurations help explain model logic to business stakeholders, promote better model governance, assist in customising models, and enable better consumption and understanding through rich visualisation.

Let's look at a recent GitHub contribution which provides a SAS Studio Custom Step to extract NLP rule configurations. SAS Studio Custom Steps are low-code interfaces used either standalone or within an analytics pipeline (a SAS program or a SAS Studio Flow) to perform a task in an extremely convenient manner.

Here's the README for the custom step. In addition, this link takes you to the project folder from where you can download the custom step and start using the same.

Access the Custom Step

Let's first make this step available to a SAS Viya environment. A simple approach is to follow instructions to upload a selected custom step to SAS Viya. Alternatively, make use of Git integration functionality already available in SAS Studio. Clone the SAS Studio Custom Steps GitHub repository and make a copy of required custom steps in your SAS Content folders. The below animated GIF shows you how. In this case, the folder to move / copy is "NLP - Extract Rule Configuration".

Use the Custom Step

There are three primary applications of this custom step, described in the following YouTube video. The custom step can be used to either generate a list of rule configurations within a project, extract all rule configurations as specified in an input list, or extract a specific rule configuration table. Watch the YouTube video to learn more!

What can you do next?

Once extracted, the rule configuration can be applied to a number of downstream activities. Here are three examples.

1. Identify changes in rules

Rule-based systems are usually created in collaboration with multiple team members and are subject to a stewardship process. During model development, the dynamic nature of the current rule configuration necessitates monitoring any incremental changes. This custom step can be part of a solution to capture changes to a model and monitor the same via a Visual Analytics (or other) dashboard. Extracted rule configuration tables at different snapshots of time can be used for this comparison and impact assessment.

2. Analyse rules with low accuracy

Some categories may be more or less accurate than others. This custom step facilitates analysis of rules with low accuracy metrics and identification of avenues for improvement. Decision makers may also like to understand the logic behind the classification before making critical business decisions. Accessible rule configuration facilitates such analysis.

3. Surface configuration within governance applications

Registered NLP models in Model Manager are saved in an astore format, which is a binary format focussed on efficient execution of the model against new data. It's not possible to infer the rules and logic from a binary format. Therefore, it helps to also maintain a version of the rule configuration in human readable format within Model Manager. Once you've extracted the rule configuration table, you would be able to send it via API to Model Manager as a Score Resource and ensure that it's included under the list of governed model assets.

Have fun with the NLP - Extract Rule Configuration custom step. Feel free to email in case of any questions or comments.

Bonus: try it out

An example SAS Studio Flow which makes use of the above custom step (for all three options it offers) is available here. Note, of course, that you would require a Viya 4 environment which contains Visual Text Analytics and SAS Studio Engineer in order to try this out. Also, please create a simple VTA project of your own which contains some models for which you can extract rule configurations.