ABSA Bank
Contact: Dewald Fourie
Country: South Africa
Award Category: Innovative Problem Solver
Tell us about the business problem you were trying to solve.
Monitoring model performance is an essential part of the model lifecycle to ensure models remain fit for purpose and regulatory compliant. Historically, this essential process relied heavily on highly trained analysts performing repetitive tasks, which included running scripts manually in a disparate, non-standardised manner. A complete overhaul of the process and its interaction with Business Stakeholders was required to create a more sustainable process that was ready for a seamless migration to SAS Viya, Analytical Platform End of Life (On Premise) - Migration to SAS VIYA(Cloud-AWS), Monitoring of Credit Risk Models is an essential Regulatory and Risk function, >500 risk models developed over a long period of time, time-consuming to create, maintain and run, with corresponding effort and cost, Business desired a complete overhaul in terms of platform, automation, speed of delivery, data visualization, standardization and simplification.
At Absa, various credit risk models are monitored frequently to ensure that the bank’s capital reserves, estimated credit losses and impairments (as predicted by the models) are at the right levels. Previously, model monitoring was a lengthy exercise that required highly skilled resources to execute large batches of SAS code on a on-premise SAS Grid platform using extremely large (millions+ rows) datasets from different sources. Once the data was extracted/aggregated, additional SAS programs used to perform statistical tests (e.g., Gini, accuracy, point-in-time and through-the-cycle metrics, stability) and produced several graphs, which enabled stakeholders/regulators to visually assess the model’s performance. This process was executed on the bank’s on-premises SAS Grid platform and could take multiple hours to a few days for a single model. Considering that the bank has 500+ models in its inventory, it was evident that model monitoring is a resource-intensive process. Once the results were produced by the SAS programs, they were copied over to Microsoft PowerPoint presentation packs by analysts. As part of the final step, the analysts were then responsible for providing commentary on the metrics/visuals. It could take an analyst 2 to 4 weeks to produce a report for a single model.
Data is growing exponentially, and with the ease at which machine learning and AI models can be created, Absa realized that the previous monitoring process was costly to sustain. A Centre of Excellence (CoE) was established and tasked with developing a model monitoring framework that adheres to the following:
•Standardization: Because monitoring reports are compiled manually in the current process, it leads to inconsistencies between the metrics supplied in reports, the data provided in reports and the overall aesthetic of the reports. The CoE sought a standardized framework.
•Ease of understanding: The manual way monitoring reports are compiled by technical resources coupled with the lack of standardization would often result in complex monitoring metrics being provided to business stakeholders who do not have technical expertise. The CoE created monitoring reports that are visually pleasing and easy to understand.
•Automation: The most important aspect – a monitoring framework is required which allows for data to be extracted and aggregated, performance metrics to be calculated and visuals to be generated in an automated fashion with limited involvement of analysts. This leads to the following benefits: 1) monitoring reports are produced faster, 2) analysts can spend more time on commentary and assessing the model, 3) monitoring reports can be produced more frequently, and 4) whenever model changes are implemented, monitoring reports can be automatically reproduced.
•Scalability: When a new model is developed, we should be able to “plug” the new model into the automated framework. This reduces the amount of time required to code up new SAS monitoring artefacts, which frees up capacity.
What SAS products did you use and how did you use them?
Absa Bank has been a longstanding SAS customer. In 2021 Absa successfully piloted SAS VIYA and the decision was made to transition from Absa’s on-premises SAS Grid platform to SAS VIYA in AWS cloud. Over a period of 2.5 years, a diverse team involving individuals from Model Risk, Model Monitoring, Model Implementation, Model Development and SAS, successfully implemented an automated monitoring framework for the bank’s Retail Credit Risk models. The following offerings from SAS were used extensively:
•SAS Content Assessment Tool (Essential assistance with migration)
•SAS VIYA (Complete Cloud ready platform)
•SAS CAS (In-memory provisioning of results to VA)
•SAS VA (Model Monitoring Visualizations, Batch Visualization)
•SAS ESM (Monitoring of platform loads/processes/consumption in the cloud)
SAS Content Assessment Tool
First and foremost, due to large existing investments in a SAS Grid platform, the ability to migrate existing models/code across to VIYA meant that a minimal code change was required. We used the SAS Content Assessment Tool to identify gaps and rectify. This is essential as during the migration process, models still must be maintained/run on both Grid and VIYA platforms.
SAS VIYA (Model Results Visualization, SAS VA)
The following steps were followed:
1. Each model suite to be automated was identified (e.g. Basel, IFRS9, etc.).
2. Existing Model development docs and reports for the model suite were identified. This was done to obtain a view on the model methodologies.
3. Model performance metrics and visuals that were common across all the models were selected for standardization.
4. Prototype dashboards containing common metrics and visuals were created and signed off, and shared with the project resources and SAS consultants who created the dashboards in SAS VA.
5. SAS macros in VIYA were used to extract/aggregate the data needed to create the visuals/metrics and were designed to be re-usable across multiple models. This delivered on the CoE objective of standardization.
SAS VIYA (Batch Automation incl. Visualization, SAS Studio)
Absa has a team within the Model Risk Processing Centre (MRPC) responsible for automation/optimization.
AUTOMATION
MRPC created an automation framework of SAS developed utilities which allow the same SAS program code to be scheduled/executed on conventional SAS 9.4, SAS EG, SAS Grid OR on SAS VIYA (within an existing container OR to start a new container).
The event driven (not time only) scheduler with dependencies/error-handling/notifications is designed in SAS and therefore can be developed on the existing Grid platform and ported AS-IS across to VIYA on AWS. With hundreds of users still on Grid and migration being an “over a period of time” event, a modeler can develop code in his/her preferred environment (SAS Enterprise Guide is particularly popular) and then hand it over for automation in VIYA with relative ease.
OPTIMIZATION
Utilizing SAS VIYA (AWS) means that we are no longer restricted to the “sunk cost” of on-premise infrastructure. Along with the benefits of being able to scale resources in the cloud, is the fact that cost can scale correspondingly. The standardized monitoring code meant that time could be spent optimizing the “single versions” instead of the previous manual, labor-intensive code. There were huge resource savings (expert time, cloud disk and CPU) across the board. In one instance, a model which had inefficient code, and many “stop-start” manual interventions was reduced from an 18 days of effort to 2 hours! This model was also automated in terms of running “just in time” when the monthly source data was detected as being available. A “smart restart” process within the framework also monitors the models. If an error is produced for any temporary, restartable reason (non-logic error, a short-term machine/memory/disk issue), the scheduler automatically restarts the process. This has been referred to as a “ghost analyst” who fixes things during the night.
What were the results or outcomes?
Ultimately, the project achieved the objectives that were specified by the CoE at the start. The team automated the entire credit risk model landscape for the bank’s retail portfolio, and the success of the project showcased the power of SAS VIYA to the bank’s quantitative community. It was a game changer for a sizeable team of skilled, quantitative resources in terms of transitioning them from “running models” to managing model outcomes and their effect on business performance. This, in turn, served as motivation for analytical leaders to transition from an on-premises SAS Grid platform to SAS VIYA.
The end-result can be associated with the following benefits:
• Migration to an on-demand, scalable cloud platform. Optimization and automation of processes allows for resizing of a dynamic cloud environment. With a traditional, on-premise architecture, optimization “frees-up” capacity, not costs. Cloud capacity is sized dynamically.
• Instead of Model monitoring analysts spending 2 to 4 weeks creating the model monitoring reports, these are automatically created.
• Before the automated framework was implemented, a large constituency of model monitoring analysts dedicated 100% of their capacity to the creation of monitoring reports. After the automated framework was implemented, many of these analysts were reassigned to other quantitative projects in the bank. This allows the bank to make effective use of its quantitative talent pool, which, as many would know, consists of scarce skills.
• Whereas many model developments previously experienced a “bottleneck” at the monitoring stage (after a new model is developed and approved), the automation framework now enables new models to simply be “plugged” into the dashboards. This allows for many more models to be developed in a shorter time.
• The CoE determined that before the automation framework was established, it could take between 6 and 12 months to develop a monitoring framework and produce the first monitoring report for a new model. In some instances, the 12-month deadline was exceeded. With the automated framework now in place, model monitoring analysts can realistically complete the entire monitoring step for a new model (including associated approvals) in less than 6 months.
A Final practical parting gift: DID YOU KNOW?
SAS VA on VIYA, out of the box, contains a button on the top right which displays “Insights were found”. Clicking on it means you can analyze run times and obtain suggestions for where to concentrate optimisation efforts (often for very large and complex batch schedules of SAS code).
Why is this approach innovative?
Those who work in the banking/financial industry would appreciate that model monitoring is often seen as a mundane, reluctant exercise. In some cases, model monitoring may be neglected, where models are not monitored correctly or at incorrect frequencies simply because of a lack of interest by many stakeholders. The general negative perception around model monitoring is further worsened by the fact that monitoring is not seen as an innovative exercise.
With the model monitoring automation project, the CoE demonstrated that with the right team and an innovative platform that delivers cutting edge capabilities (SAS VIYA), a seemingly uninteresting and laborious process can be transformed into an impressive end-product that delivers exceptional value to business. One should also consider the fact that modern platforms such as SAS VIYA enable an organization to quickly and easily automate many manual, resource-intensive processes – this frees up resources and overcomes significant capacity constraints, thereby allowing individuals within the organization who would previously be preoccupied with these processes to dedicate much more time to innovative endeavors.
The development of a SAS-based, smart scheduler which works exactly the same across SAS Grid and SAS VIYA was critical to the conversion and migration phase with:
• Event-driven architecture (Just in time, not time based)
• Error handling (incl. Smart restart)
• Status notifications
• Parallel processing
• Restartability (minimizes Cloud resources)
• Amazing Batch Visualization from SAS VA (incl. Optmization suggestions)
• New container/Existing container flexibility (Cloud Resources)
See attached document that includes further details and accompanying images.
