I was part of a team that recently helped a bank’s Enterprise Risk department address the quality of their risk and non-compliance issues.
Rather than talk about the methods and details of risk management, for which great references exist, we wish to focus more on the approach, as this has good scope for replication in other situations.
Enterprise risk analysts expect to provide brief, clear and concise descriptions of the issues they need to log into their systems.
Easier said than done!
There tends to be disparity in these analysts’ skills, experience and understanding of risk issues entered. A gap in any of these traits usually leads to poor quality issues being entered. This impedes further and effective risk management activities.
Wouldn’t we agree the more time we spend in understanding an issue, the longer we take to react to it, and the greater the chance of an (usually negative) impact?
We (a team from SAS) followed an approach we termed "prime the pump".
The popular understanding of this idiom is rooted in economics (simply put, money spent leads to more money generated).
However, in our case, we follow the broader definition – to improve a process through encouragement in the form of initial inputs and answers. To illustrate the point further, we’ll outline in more detail the process we followed with the bank.
Using a sample of the bank’s enterprise risk issue datasets and SAS Visual Text Analytics®, we built an initial natural language processing (NLP)-based information extraction and classification model. That is a mouthful, I know, but we did do a lot of work!
This model attempted to capture important elements of a risk issue, such as:
You can guess that a model built by a team of non-domain experts (though, mind you, very smart people) would be riddled with imperfections. If you guessed so, you are right!
One of the more dynamic features of the SAS Viya® platform is its flexibility. Data and results need not reside within the confines of an obscure project where only statisticians dare venture. The SAS team and the bank’s risk team collaborated to examine a sample of results. The bank carried out this investigation through a custom-built web application that used SAS Viya REST APIs to not only fetch results for each issue, but also enter the bank’s annotations and validation back into the system.
Therefore, if the initial model stated a particular issue related to the impact/loss, but the bank felt that this was not relevant, a click on a button would mean that the correct classification is now entered into the system (under a different column).
As the bank’s team went through and labelled enough issues, an interesting process started behind the scenes. A new SAS Visual Text Analytics (VTA) project used the re-labelled data to create a new model, using automatic rule generation for the required categories. This meant based on what the bank (and not the SAS team) entered as a classification, new rules were generated which tended to be more accurate in classifying that text extract.
Below is a short demonstration of the web application used in this project.
In case you are interested, a similar concept of using feedback to improve models is baked into some specialized SAS solutions such as the Adaptive Learning and Intelligent Agent System (ALIAS).
The SAS team and the bank carried out three rounds of priming the pump, and more importantly, measured the improvement at the end of each round. This indicated the accuracy of the SAS model improved by 11 percentage points in two small-sample iterations. Not bad when you take the relative inexperience of both sides on each other’s areas of expertise.
Anecdotal evidence provides a truer indication of how useful this exercise was. The bank obtained a better understanding of VTA features and the ease with which rules could be written for detecting such subjective elements.
As an approach, "prime the pump" is also applicable for:
SAS team: Tim Hutchinson, Mike Ferraro, Erin Davis, and Sundaresh Sankaran
Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.
If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website.
Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning and boost your career prospects.