How Quantum Bancorp Strengthened Its Fraud Detection Model Using SAS Model Risk Management
For years, Quantum balanced this tightrope well. Losses stayed low, customer friction stayed manageable, and the fraud detection model performed exactly as expected. But in early Q2, something shifted. Within just fourteen days, the bank saw a 32% spike in disputed card transactions—one of the largest jumps in its recent history.
Customer complaints climbed. Social media chatter increased. Fraud losses followed. It was clear that something underneath the surface had changed.
Although Quantum Bancorp is a fictional institution, the scenario in this post reflects real model governance logic and a process that many financial organizations follow. This story walks through how the bank went from a simple business goal—reduce fraud while protecting customers—to conducting a structured, evidence‑driven model performance review using SAS Model Risk Management (MRM). If you’ve ever wondered what it looks like to apply strong model governance during a real‑world‑style performance breakdown, this journey offers a practical view from inside the walls of a typical financial institution.
(Note: The screen captures in this post are from the Long-Term Stable Release 2025.09 of SAS Model Risk Management.)
A Business Goal Turns Into a Model Governance Problem
At the start of every quarter, the Fraud Strategy team sets one straightforward goal:
Quarterly Goal:
Reduce fraud losses while avoiding unnecessary declines for legitimate customers.
It sounds simple, but it’s actually a careful balancing act.
- If the model is too strict, good customers get blocked or flagged.
- If it’s too lenient, fraud slips through and losses rise.
Finding the right balance is critical.
So when a sudden spike in customer disputes appeared, Fraud Strategy Lead Christine Chen reframed the goal with urgency:
Validate how the fraud model is performing, figure out what changed, and take action before the issue gets worse.
Instead of guessing at the root cause, Christine relied on her governance foundation—SAS Model Risk Management—to guide the investigation.
Monitoring Starts Long Before the Reports
Monitoring doesn’t begin when you upload backtesting results or click Run Performance Review. It doesn’t even start with the Model Reviews object. Effective monitoring begins much earlier—at the point the model is created—and with the framework the organization builds around that model.
Once a model exists, the organization must define how it will measure success. That means identifying the right performance indicators (KPIs), setting meaningful thresholds, and documenting everything in a Model Evaluation Plan.
For Quantum’s fraud detection model, the KPIs were a blend of SAS‑provided metrics and a few custom indicators tailored to fraud behavior. These included Recall, Precision, False Positive Rate, AUC, and PSI for the features most likely to drift. The KPIs answered one question: What should we measure?
But KPIs alone can’t tell you whether performance is good enough. For that, you need thresholds—the standards that define acceptable, warning, and critical performance levels.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Thresholds: Defining What “Good Performance” Actually Means
Thresholds give each KPI meaning by setting the acceptable boundaries. For Quantum’s fraud model, these limits reflected the bank’s risk appetite:
- Recall must stay above 80% so the model continues to catch most fraudulent transactions.
- False Positive Rate must stay at or below 1.5% to avoid frustrating legitimate customers.
- AUC must remain 0.85 or higher to show the model is still separating fraud from non‑fraud effectively.
- PSI for key variables must stay below 0.10 to confirm the data feeding the model hasn’t drifted too far from expected patterns.
Together, these thresholds define what “acceptable performance” looks like—and they must exist before any model review can start.
KPI Thresholds for Recall
Once KPIs and thresholds were set, the next step was building the Model Evaluation Plan. This plan acts as a reusable monitoring blueprint, combining:
- all the KPIs
- their formulas
- their threshold definitions
- the evaluation logic that determines when a finding should be triggered
For the fraud detection model, Christine’s evaluation plan functioned like a grading rubric. It told SAS MRM how to interpret the backtesting results that would later be attached during the review.
Assigning the Evaluation Plan to the Model
After creating the plan, Christine needed to formally link it to the model. She returned to the Models object—Quantum’s central registry for all modeling assets—and selected the fraud detection model. Its metadata gave a complete snapshot:
- Purpose: Detect anomalous card activity
- Status: In Production
- Monitoring Cadence: Quarterly
- Owner: Fraud Analytics team
- Validators: Model Risk Management Groups (MRMG)
These details confirmed that the model was cataloged correctly and ready for governance activities.
To start a new performance review, Christine opened the model’s Reviews and Findings tab. This tab stores all current and historical review entries.
From the Model Reviews widget, she clicked the Create New icon to add a new review record—kicking off the formal monitoring cycle.
Why a Model Performance Review?
- It provides a structured, auditable process.
- It enforces the use of validated evidence, not gut instinct.
- It ties investigation, documentation, validation, and remediation together.
- It ensures findings are tracked and acted upon.
With the review created, Christine opened the entry, clicked Claim Task, and officially kicked off the workflow.
On the Details tab she verifies:
- Review type
- Monitoring frequency
- Model owners
- Planned dates
- Workflow template
Nothing seems out of place. Everything is complete except for one crucial element: evidence.
SAS Model Risk Management Linked Model Reviews
On the new Model Evaluation Plan tab, Christine added information about the model review object as shown below. Including changing the review type to Model performance review and toggling the threshold breach to create findings for critical breaches and warnings.
Now that the KPIs, thresholds, Model Evaluation Plan, and main details for the model performance review are defined, Christine can start the workflow.
Christine opened the entry and clicked Claim Task to take ownership. She reviewed the Details tab to ensure accuracy, then moved to the Performance Assessment tab.
Here, the system asked for the most critical piece of evidence: the latest backtesting results.
Christine selected the date of the last review, clicked Add attachment, uploaded the fraud model’s backtesting spreadsheet, described the file, and saved it.
Now SAS MRM has the full picture: the model, its KPIs, threshold rules, the evaluation plan, and the backtesting data needed for assessment.
Backtesting Results Uploaded to Model Performance Review
When Christine advanced the workflow, she chose to preview the monitoring results first. This showed her 2 findings would be created from the backtesting file: 10001 (Recall) and 10002 (PSI).
Once she ran the official script, SAS MRM triggered the automated evaluation behind the scenes. The system retrieved the attachment, read the KPI values from the backtesting file, compared each KPI value to its threshold range, and automatically created a finding for every breach.
By the time Christine finished uploading the backtesting results and previewing the monitoring script, the picture became much clearer. The KPIs confirmed what the business had been feeling for weeks: the fraud model wasn’t just drifting — it was slipping outside Quantum’s risk appetite. Recall had dropped, the False Positive Rate had risen, and customers were caught in the middle. When the monitoring script ran and SAS MRM automatically generated two findings, those issues moved from “suspicions” to documented governance evidence.
This is where performance monitoring ends and the next phase of model governance begins. Christine now knows something in the model has changed — but she doesn’t yet know why, how serious it is, or what needs to happen next. Those answers come from the validation stage, where the findings are reviewed, the model’s overall health is scored, and the remediation path is defined.
In Part Two of this series, we’ll follow Christine as she works with the Model Risk Management Group to dig into the findings, complete the Validation Assessment, understand the deeper root causes behind the model’s issues, and plan the steps needed to bring the fraud model back into alignment.
Performance monitoring has revealed the problem.
Next, we’ll uncover what it means — and what Quantum decides to do about it.
Stay tuned for Part Two.
For more information on SAS Model Risk Management, click here.
For more articles on SAS Model Risk Management, click here.
Find more articles from SAS Global Enablement and Learning here.
