turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Find a Community

- Home
- /
- Analytics
- /
- Stat Procs
- /
- how to compare two models

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

09-13-2012 09:27 AM

Hello everyone,

I have one dataset and I want to model the probability of default from a loan. I used the logistic regression for it but with the data in different structures. I do use the same (original) data, but one model I used the row data and for another model I used the categorized data (I built all the categories) and now I want to be able to compare the two models in order to say if the structure of the data has some influence on the results.

I tought I could maybe compare their misclassification rate, but how could I build such a test to make this comparison, since here I have only one value (the misclassification rate) for each model?

Do you have some idea?

Thanks

Accepted Solutions

Solution

10-11-2017
11:33 AM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to danirecchia

09-13-2012 01:03 PM

Rick, I guess I am dating myself....

Dan, Rick's example will work for your situation as long as the dependent variable (LoanDefault, say) is the same. Just specify multiple predictor variables in the ROC statement.

ods graphics on;

proc logistic data=_last_ plots=roc(id=prob);

CLASS d e f;

model LoanDefault = a b c d e f / nofit;

roc 'Model1' a b c;

roc 'Model2' d e f;

roccontrast reference('Model1') / estimate e;

run;

ods graphics off;

All Replies

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to danirecchia

09-13-2012 10:04 AM

One measure of quality of a logistic regression is the ROC curve (often summarized by the c-index). This article provides a general method for comparing two ROC curves that were derived from the same underlying data.

DeLong ER, DeLong DM, Clarke-Pearson DL: Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837–845, 1988

At one point, there was a SAS macro around to implement the comparison, but I haven't looked for it in a while.

Doc Muhlbaier

Duke

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Doc_Duke

09-13-2012 10:22 AM

You don't need the macro anymore. This is now built into PROC LOGISTIC via the ROCCONTRAST statement: http://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#statug_logistic_sec...

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Rick_SAS

09-13-2012 10:34 AM

Hello Rick,

as I saw in the example that you sent, the ROCCONTRAST compares the ROC Curves from different the 3 different models with the 3 variables. But you see, for me originally the data is the same, but to adjust the model with categorized data I had to do some modifications and I can't see how should I use this option to compare the curves, since not only the data (structural - variables in classes) but also the chosen variables, I would like to compare both final models from row - and categorized data.

Solution

10-11-2017
11:33 AM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to danirecchia

09-13-2012 01:03 PM

Rick, I guess I am dating myself....

Dan, Rick's example will work for your situation as long as the dependent variable (LoanDefault, say) is the same. Just specify multiple predictor variables in the ROC statement.

ods graphics on;

proc logistic data=_last_ plots=roc(id=prob);

CLASS d e f;

model LoanDefault = a b c d e f / nofit;

roc 'Model1' a b c;

roc 'Model2' d e f;

roccontrast reference('Model1') / estimate e;

run;

ods graphics off;

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Doc_Duke

09-13-2012 02:15 PM

Hello Doc@Duke,

thank you very much for the explanation! I see now, what should I do.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to danirecchia

09-13-2012 01:04 PM

The fastest and easiest way is to compare the c-stat values between the 2 models. c-stat ranges between 0.5 and 1. The higher the c-stat, the better the performance and accuracy of the model. And PROC LOGISTIC returns c-stat. c-stat is nothing but the area between the 45 degree line and the curve, which is the ROC.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to tarek03

09-13-2012 02:20 PM

Hey tarek03,

yep, the c-statistic is a very used measure and I tought on this comparison but you see, if I just look for the values of the c-statistic I am not able to compare them. But now, whith the ROCCONTRAST I know what to do.

But I also think that a very interesting test would be the one, which I could compare the misclassification rate from the models, maybe this could be more specific on the discrimination power of the model. But for this case I don't know yet what to do.