Creating Custom Fraud Monitoring KPIs in SAS Model Manager

[#1]

Introduction

One of the key features of SAS Model Manager is the ability to monitor a model's performance over time. Out-of-the-box, SAS Model Manager examines various statistical measures of input drift and model performance.  These measures include Variable Distribution, Characteristic, Stability, Lift, Gini, ROC, Kolmogorov-Smirnov (KS), Average Squared Error (ASE), and Feature Contribution Index (FCI). With the release of SAS Viya 2021.1.1, users now have the ability to define their own Key Performance Indicators (KPIs) for model monitoring. The MMKPI Macro and Action Set allow ModelOps teams to define KPIs that are important to their specific use cases.

There are numerous options for domain-specific KPIs, but today I want to focus on utilizing this new feature for fraud operations. When combatting fraud, organizations typically utilize a two-pronged approach: a fraud model and a series of Boolean rules. I already have a model in place that predicts the probability that a transaction is fraudulent. I want to translate this probability into a score that can be utilized in fraud rules. To look at the effectiveness of this score, I want to bucket the score and explore KPIs across each of these score buckets.

In today's article, we will create the custom KPI table based on our score buckets, add the customer KPI table using the MMKPI Action set, use the custom KPIs to create visualizations, and add those visualizations into Model Manager's performance tab.

Create Custom KPI Table

Since we are looking at performance monitoring over time, we will have multiple tables compiled at different time points. For our example, we have four quarters of fraud-tagged transaction data. We need to use our model to score each transaction before applying our function to transform the fraud probability into a score. Next, we need to bucket those scores and for each bucket determine the number of total transactions, the number of fraudulent transactions, and the number of genuine transactions. Finally, we need to mark each the data with its quarter.  In this example, our table will take the following form:

Bucket	Total_Count	Fraud_Count	Genuine_Count	TimeSK	TimeLabel
S1 - S2	T1	F1	G1 = T1 - F1	1	Q1
S2 - S3	T2	F2	G2 = T2- F2	1	Q1
…	…	…	…	…	…
SN-1 - SN	TN	FN	GN = TN - FN	1	Q1
…	…	…	…	…	…
SN-1 - SN	TN	FN	GN = TN - FN	4	Q4

A copy of the completed code is available here. Before we can get started, we must first define and run our performance job in SAS Model Manager. For each table, I used a looping macro program to perform the following steps:

1. Score the table using the model. To find the scoring information specific to my chosen model, I ran a scoring test in SAS Model Manager. Upon completion, the score code used for the test is made available in the results. I modified that code to run within my looping macro program.
2. Create a score from the model probability.  I utilized a formula common to credit scoring where the score is calculated using a chosen factor and offset. The formula is represented as score = offset + factor * ln(odds). Instead of using a score, an organization can also utilize the probabilities by themselves or implement their own calculation.
3. Generate score buckets using if-then-else statements and Boolean logic.
4. Use PROC FEDSQL to generate a query grouping score buckets and aggregating to get the total transaction count and fraudulent transaction count for those score buckets.
5. Add timestamped variables to the data and calculating the genuine transaction count by subtracting the fraud transaction count from the total transaction count.
6. Append all data quarters together using data step.

Add Custom KPI Table

Now that we have our custom KPI table, we can easily add it into the KPIs for this project using the Add Custom KPI action. This is implemented at the bottom of my completed code.  This article provides more information about this action and additional examples are available on GitHub. The SAS log shows the name of your new KPI table that typically exists in the Model Performance library and ends in MM_STD_KPI. Remember or save that table name for the next step.

Build Visualizations

Our goal is to understand the customer and business impact of manually investigating or even stopping transactions within each of our model's score value buckets. Manually investigating every transaction requires a  herculean workforce that is cost-prohibitive. Stopping each transactions angers or annoys customers. On the other hand, doing nothing leads to heavy fraud losses. Thus, we need to understand which buckets to post, which to investigate, and which to stop.

To better understand the utility of each bucket, we will approach the problem from three angles. First, we examine a bucket's ability to differentiate between fraudulent and non-fraudulent transactions. Ideally, we would want a bucket to be entirely genuine or entirely fraudulent. In some organizations, this metric is referred to as the False Positive metric (not to be confused with False Positives of confusion matrix fame) and is calculated by taking the total number of transactions in a bucket and dividing by the number of fraudulent transaction in a bucket plus 1 (to ensure we don't divide by 0 for buckets with no fraud). A low value means that bucket consists of more fraudulent transactions and a high value means that a bucket consists of more genuine transactions. 

Second, we will look at the percentage of all fraudulent transactions that are found within a bucket. This is referred to as Detection Rate.  Third, we estimate the effort to investigate a particular bucket by examining the percent of all transactions within a bucket. This is called Outsort Rate.  Metrics and definitions may vary across organizations, but this will give us a good initial understanding.

From SAS Visual Analytics open your newly customized MM_STD_KPI table. From the Data pane, click + New data item, and select Calculated item.

The resulting pop-up displays all of the variables in your data and numerous Operators to utilize. The operations we need are under Numeric (Simple) and Aggregated (Simple).  Our False Positive metric can be calculated by Total_Count / (Fraud_Count + 1). Detection Rate is calculated as Sum {By Group} Fraud_Count / Sum {For All} Fraud_Count. Finally, Outsort Rate is calculated as  Sum {By Group} Total_Count / Sum {For All} Total_Count.

Next, let's put together our visualizations. For each measure, I dragged a Line chart onto the canvas. I specified my Category as my bucketed score, my Measure as my calculation and my Group as my Time Label.

I added a few tweaks using the Options pane to make the chart more visually appealing. Repeat this process for each calculation. The resulting Detection Rate Chart looked like the following:

Ideally, we would want higher detection rates in the higher score ranges. We see spikes in detection rates in a lower range in Q4 and to a lesser extent in Q3 alluding to the necessity of a model refresh. You can also use the charts to explore the bucketed distribution of all transactions, genuine transactions, and fraudulent transactions.

Once we are happy with our charts, we can embed them all into our SAS Model Manager Performance tab.

Embedding Visualizations in Model Manager Performance Tab

Now, embedding your custom visualizations into SAS Model Manager is a piece of cake! First, from Visual Analytics, open the option menu for a chart you want to embed. Pro-tip: if you want to embed several components together such as filters and charts, place everything within a container and use the options for the container object. From the options menu, select Copy Link … From the resulting pop-up, ensure that Embeddable web component is checked before selecting Copy Link. Now the URL is copied to your clipboard.

Next, head over to the Performance tab of your SAS Model Manager project. You should have defined and ran your performance monitoring job in previous step. From the options menu in the right-hand corner of the Performance tab, select Edit Layout. In the pop-up window, select + Add Custom Chart. From the Add Custom Chart Window, name your chart and add a description before pasting the URL from Visual Analytics. After you add the chart, you can re-arrange the charts or remove charts to further customize your performance monitoring report.

Conclusion

In this example, we walked through how to add custom KPIs to the performance monitoring tab in SAS Model Manager. This article focused specifically on fraud operations, but I hope you have found pieces useful for other domains and use cases.

To keep up to date on Model Manager updates, bookmark the Model Manager Communities tag. To learn more about the using the new KPI Action Set, see Diana Shaw's article and the documentation. Additional KPI examples are available on GitHub.