K fold CV - how to draw "average" ROC

[Melk]

I am running a 5 fold cross validation on my logistic model. The predicted probability of the outcome for each test group was generated using data from the training groups. My question is - how can I draw a "mean" ROC curve to summarize my results?

[Reeza]

Are you asking how to do the calculation or how to create the graph?

[Melk]

How to create the graph, but it would also be useful to calculate the AUC. I essentially have output predicted probabilities from fitting my training data on each of my test data, I just dont know how to use this to create the ROC curve, ultimately get the AUC.

[Ksharp]

What do you mean by 'average' ROC .

You can use ROC= option to get those K-fold  roc dataset,

but you might not get the average roc, x could be different, you could try.

if you want plot all these 5 roc in the same graph. here is an example .

proc logistic data=test9 ;
model good_bad(event='good')= &varlist 
/outroc=_roc lackfit scale=none aggregate rsquare firth;
output out=_output h=h c=c cbar=cbar;
run;

data plot_roc;
 set roc(in=ina) _roc;
 if ina then dsn='训练集';
  else dsn='测试集';
run;

title ' ';
proc sgplot data=plot_roc  aspect=1;
 series x=_1MSPEC_ y=_SENSIT_ /group=dsn smoothconnect name='x';
 lineparm x=0 y=0 slope=1/lineattrs=(color=verylightgray);
 xaxis grid;
 yaxis grid;
 keylegend 'x' /title=' ' location=inside position=nw across=1;
run;

[Melk]

Well, I thought that since the training data was used to fit the test data, which is a different portion of the main dataset each time, those predicted probabilities could be used to get an "average" ROC (by basically combining each of the test datasets, each with predicted probabilities of the outcome, together). So, I basically have a dataset with my outcome, predictors, and predicted probabilities from my k fold CV and dont know how to construct the ROC curve.

[StatDave · Accepted solution]

If you have a data set of actual binary responses and predicted probabilities (from whatever source), you can produce an ROC plot and analysis using the PRED= option in the ROC statement of PROC LOGISTIC as shown in this note.  Notice that the predicted probabilities in each of the examples come from other modeling methods than PROC LOGISTIC.

[Melk]

Thanks so much!