BookmarkSubscribeRSS Feed
Ronein
Meteorite | Level 14
Hello
Let's say that someone else developed a model that the response varaible is binary indicator of failure ( default) and the predictor varaibles are credit card usage behavior. I don't know exactly how the model was built and what was checked regarding credit card usage but in the bottom line for each customer I received calculated pd ( probability of default).
I received pd for customers in different times.
I was asked to check the model in different data ( data of customers from different times).
I thought to do following-
Calculate Gini coefficient for each out of time data .
Calculate 20% capture rate for each out of time data.
My question- what other checks do you recommend me to do that test the model and its stability?
2 REPLIES 2
Ksharp
Super User
"I was asked to check the model in different data"
Check for what ? Check/Predict pd of customer at different times ? It would lead you to build Mixed Model for repeated measure. Check PROC GLIMMIX .

"what other checks do you recommend me to do that test the model and its stability?"
If your 'stability' means goodness of fitness test of model, then check GOF option of MODEL statement of PROC LOGISTIC.

If you are building a Credit Score Card , you could check PSI index to see whether the population of customer is change or not .
https://communities.sas.com/t5/SAS-Programming/PSI-stability-index/m-p/802144
https://communities.sas.com/t5/SAS-Programming/Is-my-PSI-Population-Stability-Index-calculation-code...

Check @Rick_SAS blogs:
https://blogs.sas.com/content/iml/2019/02/20/easier-calibration-plot-sas.html
https://blogs.sas.com/content/iml/2018/05/14/calibration-plots-in-sas.html
https://blogs.sas.com/content/iml/2016/11/28/goodness-of-fit-large-small-samples.html
Season
Barite | Level 11

It seems like you wish to validate a model on your data. Given that the dependent variable is binary, you can employ the concordance statistic (statistic) to validate the model.

hackathon24-white-horiz.png

The 2025 SAS Hackathon Kicks Off on June 11!

Watch the live Hackathon Kickoff to get all the essential information about the SAS Hackathon—including how to join, how to participate, and expert tips for success.

YouTube LinkedIn

How to choose a machine learning algorithm

Use this tutorial as a handy guide to weigh the pros and cons of these commonly used machine learning algorithms.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 2 replies
  • 4688 views
  • 0 likes
  • 3 in conversation