Programming the statistical procedures from SAS

K fold CV - how to draw "average" ROC

Accepted Solution Solved
Reply
Frequent Contributor
Posts: 78
Accepted Solution

K fold CV - how to draw "average" ROC

I am running a 5 fold cross validation on my logistic model. The predicted probability of the outcome for each test group was generated using data from the training groups. My question is - how can I draw a "mean" ROC curve to summarize my results?


Accepted Solutions
Solution
‎07-10-2017 03:44 PM
SAS Employee
Posts: 282

Re: K fold CV - how to draw "average" ROC

If you have a data set of actual binary responses and predicted probabilities (from whatever source), you can produce an ROC plot and analysis using the PRED= option in the ROC statement of PROC LOGISTIC as shown in this note.  Notice that the predicted probabilities in each of the examples come from other modeling methods than PROC LOGISTIC.

View solution in original post


All Replies
Super User
Posts: 19,815

Re: K fold CV - how to draw "average" ROC

Are you asking how to do the calculation or how to create the graph?

Frequent Contributor
Posts: 78

Re: K fold CV - how to draw "average" ROC

How to create the graph, but it would also be useful to calculate the AUC. I essentially have output predicted probabilities from fitting my training data on each of my test data, I just dont know how to use this to create the ROC curve, ultimately get the AUC.

Super User
Posts: 10,035

Re: K fold CV - how to draw "average" ROC

What do you mean by 'average' ROC .

You can use ROC= option to get those K-fold  roc dataset,

but you might not get the average roc, x could be different, you could try.

if you want plot all these 5 roc in the same graph. here is an example .

 

 

proc logistic data=test9 ;
model good_bad(event='good')= &varlist 
/outroc=_roc lackfit scale=none aggregate rsquare firth;
output out=_output h=h c=c cbar=cbar;
run;

data plot_roc;
 set roc(in=ina) _roc;
 if ina then dsn='训练集';
  else dsn='测试集';
run;

title ' ';
proc sgplot data=plot_roc  aspect=1;
 series x=_1MSPEC_ y=_SENSIT_ /group=dsn smoothconnect name='x';
 lineparm x=0 y=0 slope=1/lineattrs=(color=verylightgray);
 xaxis grid;
 yaxis grid;
 keylegend 'x' /title=' ' location=inside position=nw across=1;
run;
Frequent Contributor
Posts: 78

Re: K fold CV - how to draw "average" ROC

Well, I thought that since the training data was used to fit the test data, which is a different portion of the main dataset each time, those predicted probabilities could be used to get an "average" ROC (by basically combining each of the test datasets, each with predicted probabilities of the outcome, together). So, I basically have a dataset with my outcome, predictors, and predicted probabilities from my k fold CV and dont know how to construct the ROC curve.

Solution
‎07-10-2017 03:44 PM
SAS Employee
Posts: 282

Re: K fold CV - how to draw "average" ROC

If you have a data set of actual binary responses and predicted probabilities (from whatever source), you can produce an ROC plot and analysis using the PRED= option in the ROC statement of PROC LOGISTIC as shown in this note.  Notice that the predicted probabilities in each of the examples come from other modeling methods than PROC LOGISTIC.

Frequent Contributor
Posts: 78

Re: K fold CV - how to draw "average" ROC

Posted in reply to StatDave_sas

Thanks so much!

☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
  • 6 replies
  • 163 views
  • 2 likes
  • 4 in conversation