topic Re: Comparison between AUC-ROC Curves - Hypothesis test in SAS Procedures

Comparison between AUC-ROC Curves - Hypothesis test

claudiopcjr — Fri, 14 May 2021 20:50:07 GMT

Hi everyone, I'm running tests to generate comparisions between ROC curves using the PROC LOGISTIC procedure as below:

ods graphics on;
proc logistic data=DATA_CASE_STUDY_ING;
model intodefault = mrating / outroc=troc;
roc; roccontrast;
run;
ods graphics off;

The date 'have' has the column 'defaulted' identifying the customers that presented default (with value 1) or didn't (with value 0) and the column 'rating' that ranks the customers from worst to the better ones.

The main result for me on this procedure is:

By default the ROC1 curve, generated for comparision, has a ROC value of 0.5.

Does anyone know how to change it ? It would be the same as changing the null hypothesis for this test.

I'd like to test it for 0.6, 0.7 and 0.8.

Thanks in advance.

Re: Comparison between AUC-ROC Curves - Hypothesis test

PaigeMiller — Fri, 14 May 2021 21:25:01 GMT

Does not the confidence interval give you that test?

0.6 is not in the confidence interval, so the area under the curve is statistically different than 0.6.

Re: Comparison between AUC-ROC Curves - Hypothesis test

claudiopcjr — Sat, 15 May 2021 03:36:18 GMT

For the used data yes, but I'd like to test it in generical terms and also to check the p-value.

Thank you.

Re: Comparison between AUC-ROC Curves - Hypothesis test

PaigeMiller — Sat, 15 May 2021 10:37:27 GMT

@claudiopcjr wrote:

For the used data yes, but I'd like to test it in generical terms and also to check the p-value.

Give an example. It seems to me that the solution to use the confidence interval reported by PROC LOGISTIC is perfectly generic, it applies to all situations and all data sets and any value (0.5, 0.6, 0.7, ...) that you wish to test it against.

Re: Comparison between AUC-ROC Curves - Hypothesis test

SteveDenham — Thu, 20 May 2021 19:23:13 GMT

And to follow up on what @PaigeMiller was saying about an interval, it doesn't depend on the unrealistic assumption that the underlying population values are identical, which is what the hypothesis test p value assumes. You know a priori that they are not identical, and so whatever p value comes out, it isn't "correct", and you don't know how far away from correct the p value is (or else you could apply some non-central correction). For some things (say a binomial underlying variable), it isn't even a smooth curve as you move away from identical.

(Thanks and a tip of the hat to my Math Stat prof Dr. Norm Matloff)

SteveDenham