Scoring Text Using SAS Intelligent Decisioning

1 Like

Tapping into unstructured text data can glean illuminating insights. But discovering the insights buried in text to build better decisions can often be a manual process. With the right tools, this process can be completely automated and even API-enabled. I propose a solution based on SAS Intelligent Decisioning that can take text from an API call, process the text using a model built in SAS Visual Text Analytics, and return a decision based on the results of the text model.

SAS Intelligent Decisioning on Viya creates decision flows to automate decisioning at scale. Decision flows can consist of rules, branches, models, Python code files, DS2 code files, and more. These decision flows offer strong customization and can mimic current business processes well. Following, SAS Visual Text Analytics is a solution on SAS Viya that combines the power of Natural Language Processing (NLP), machine learning, and linguistic rules to reveal insights in text data. SAS Visual Text Analytics builds several types of text models, including contextual models, sentiment models, topic models, and categorization models. These models take unstructured text data and create tabular data and these results can easily be digested in reporting, machine learning, and decisioning.

This approach does present one challenge: SAS Intelligent Decisioning on Viya 3.5 does not support model scoring using models developed in SAS.... While SAS Intelligent Decisioning can use the results of model scoring, it doesn’t perform this scoring out-of-box. Well, SAS Intelligent Decisioning also doesn’t send emails out-of-the-box either, but that doesn’t stop us! Given the flexibility SAS Intelligent Decisioning offers, its available capabilities can be easily augmented through the available DS2 or Python code files nodes. In our solution, we use the Python code node to take in text, apply some processing, score using a text model, and return the results.

The Python Code

When coding in Python for Intelligent Decisioning, one must encapsulate their code in an execute function. This function takes in values from the decisioning pipeline and returns values back to the pipeline at the end of the function. Inside my code block, I start by importing the necessary packages and establishing my connection to CAS. Then I cleaned the text input received from the decisioning pipeline. I added this cleaned text into a table within CAS. Next, I scored my text input using a text model I trained in SAS Visual Text Analytics. Finally, I took the results of the text model and performed some-post processing.

def execute(id_num, text_string):

     "Output: FDCPA, FCRA, Credit_Card"

     # Import necessary packages
     import swat
     import re
     import pandas as pd

     # Connect to CAS
     hostname = your-host-name
     username = your-username 
     password = your-password
     conn = swat.CAS(hostname, 5570, username, password)

     # Process text string
     text_string = str(text_string)
     text_string = re.sub('[^a-zA-Z\d\s]', '', text_string)
     text_string = text_string.strip()

     # Create dataframe
     d = {'CID': [id_num], 'CTEXT': [text_string]}
     df = pd.DataFrame(data=d)
    
     # Create CASTable from dataframe
     sdf = conn.upload_frame(data=df, casout=dict(name='temp_text',caslib='public', replace=True))
    
     # Execute category code
     conn.builtins.loadactionset('textRuleScore')
        
     model_lib = model-lib-from-VTA-score-code
     bin_table = model-table-from-VTA-score-code

     results = conn.textRuleScore.applyCategory(
          model = {'caslib':model_lib, 'name': bin_table},
       	  table = {'caslib':'public', 'name':'temp_text'},
          docID = 'CID', 
          text = 'CTEXT',
          modelOut = {'caslib':'public', 'name':'job_out', 'replace':True} 
     )

     # Pull results table 
     table = conn.CASTable(name="job_out", caslib="public").to_frame()
     conn.table.dropTable(caslib='public', name='temp_text', quiet=True)
     conn.table.promote(name="job_out", caslib="public")
    
     # Pulling specific categories
     FDCPA_Bankruptcy_Collection = table.iloc[0]['category_16'].item()
     FDCPA_Debt_Collection_Time = table.iloc[0]['category_10'].item()
     FCRA_ACCURATE_INFO = table.iloc[0]['category_27'].item()
     FCRA_PERMISSABLE_USE = table.iloc[0]['category_29'].item()
     Credit_Card = table.iloc[0]['category_72'].item()
    
     # Category Data Prep
     FDCPA = FDCPA_Bankruptcy_Collection + FDCPA_Debt_Collection_Time
     FCRA = FCRA_ACCURATE_INFO + FCRA_PERMISSABLE_USE 

     return FDCPA, FCRA, Credit_Card

Adding Business Logic to the Decision Flow

Our Python code will take text data, use a text model to score it, perform some processing, and return the results. With these results, we can build out the rest of the decision pipeline to fit our processing needs. In the example below, I use the results of the Python code to determine which team to send the complaint to. This decision flow is easily customized to fit business processes. With a small addition, you can even send emails alerting staff of complaints they need to investigate!

Publishing the Decision Flow into SAS Micro Analytic Service

Once you are happy with your decision flow, publishing it into the SAS Micro Analytic Service (MAS) requires just a few button clicks. SAS MAS is a high-performance program execution service that exposes a REST API end-point into your decision. To use the API, you must first get an access token. With the token and domain specified, you can test the decision flow using the requests package in python (or your preferred way of testing APIs).

from requests import request
url = domain + '/microanalyticScore/modules/textscore/steps/execute'
payload = '{"inputs":[{"name":"complaint_id_", "value":1}, 
    {"name":"consumer_complaint_narrative_", "value":"My credit card payment is being auto-drafted from my bank account and I never asked for that! "}]}'
    headers = {
         'Content-Type': 'application/vnd.sas.microanalytic.module.step.input+json', 
         'Authorization': 'Bearer ' + token
     }

request = request('POST', url, data=payload, headers=headers, verify=False)
request.json()

In conclusion, this solution can be used to score text models created in SAS Visual Text Analytics using SAS Intelligent Decisioning and is deployable via for decisioning within a few seconds.

Want to learn more about text analytics? Then check out these other great posts:

Want to learn more about intelligent decisioning? Then check out these lovely posts:

For more information about MAS and REST APIs, see these informative posts: