Published on behalf of Sundaresh Sankaran and his team.
I was part of a team that recently helped a bank’s Enterprise Risk department address the quality of their risk and non-compliance issues.
Rather than talk about the methods and details of risk management, for which great references exist, we wish to focus more on the approach, as this has good scope for replication in other situations.
What did we deal with?
Enterprise risk analysts expect to provide brief, clear and concise descriptions of the issues they need to log into their systems.
Easier said than done!
There tends to be disparity in these analysts’ skills, experience and understanding of risk issues entered. A gap in any of these traits usually leads to poor quality issues being entered. This impedes further and effective risk management activities.
Wouldn’t we agree the more time we spend in understanding an issue, the longer we take to react to it, and the greater the chance of an (usually negative) impact?
How did we solve this?
We (a team from SAS) followed an approach we termed "prime the pump".
The popular understanding of this idiom is rooted in economics (simply put, money spent leads to more money generated).
However, in our case, we follow the broader definition – to improve a process through encouragement in the form of initial inputs and answers. To illustrate the point further, we’ll outline in more detail the process we followed with the bank.
We prime the pump.
Using a sample of the bank’s enterprise risk issue datasets and SAS Visual Text Analytics®, we built an initial natural language processing (NLP)-based information extraction and classification model. That is a mouthful, I know, but we did do a lot of work!
This model attempted to capture important elements of a risk issue, such as:
- What should happen: the usual process followed
- What actually happened: events that are of interest to the bank
- What are the consequences: possible negative impact/loss
- What should we do: steps or recommendations that the bank could take to mitigate the risk
We pump further
You can guess that a model built by a team of non-domain experts (though, mind you, very smart people) would be riddled with imperfections. If you guessed so, you are right!
One of the more dynamic features of the SAS Viya® platform is its flexibility. Data and results need not reside within the confines of an obscure project where only statisticians dare venture. The SAS team and the bank’s risk team collaborated to examine a sample of results. The bank carried out this investigation through a custom-built web application that used SAS Viya REST APIs to not only fetch results for each issue, but also enter the bank’s annotations and validation back into the system.
Therefore, if the initial model stated a particular issue related to the impact/loss, but the bank felt that this was not relevant, a click on a button would mean that the correct classification is now entered into the system (under a different column).
As the bank’s team went through and labelled enough issues, an interesting process started behind the scenes. A new SAS Visual Text Analytics (VTA) project used the re-labelled data to create a new model, using automatic rule generation for the required categories. This meant based on what the bank (and not the SAS team) entered as a classification, new rules were generated which tended to be more accurate in classifying that text extract.
Below is a short demonstration of the web application used in this project.
.
In case you are interested, a similar concept of using feedback to improve models is baked into some specialized SAS solutions such as the Adaptive Learning and Intelligent Agent System (ALIAS).
We strike oil, and a lot of wisdom.
The SAS team and the bank carried out three rounds of priming the pump, and more importantly, measured the improvement at the end of each round. This indicated the accuracy of the SAS model improved by 11 percentage points in two small-sample iterations. Not bad when you take the relative inexperience of both sides on each other’s areas of expertise.
Anecdotal evidence provides a truer indication of how useful this exercise was. The bank obtained a better understanding of VTA features and the ease with which rules could be written for detecting such subjective elements.
In conclusion
As an approach, "prime the pump" is also applicable for:
- Other text analytics use cases (auditor’s notes, customer complaints, legal documents)
- A labeling and annotation tool
- A rule discovery mechanism
- An educational aid for training and coaching of risk analysts
SAS team: Tim Hutchinson, Mike Ferraro, Erin Davis, and Sundaresh Sankaran
