topic Re: K fold CV - how to draw "average" ROC in Statistical Procedures

K fold CV - how to draw "average" ROC

Melk — Mon, 10 Jul 2017 04:37:50 GMT

I am running a 5 fold cross validation on my logistic model. The predicted probability of the outcome for each test group was generated using data from the training groups. My question is - how can I draw a "mean" ROC curve to summarize my results?

Re: K fold CV - how to draw "average" ROC

Reeza — Mon, 10 Jul 2017 05:26:38 GMT

Are you asking how to do the calculation or how to create the graph?

Re: K fold CV - how to draw "average" ROC

Melk — Mon, 10 Jul 2017 05:29:03 GMT

How to create the graph, but it would also be useful to calculate the AUC. I essentially have output predicted probabilities from fitting my training data on each of my test data, I just dont know how to use this to create the ROC curve, ultimately get the AUC.

Re: K fold CV - how to draw "average" ROC

Ksharp — Mon, 10 Jul 2017 13:50:28 GMT

What do you mean by 'average' ROC .

You can use ROC= option to get those K-fold roc dataset,

but you might not get the average roc, x could be different, you could try.

if you want plot all these 5 roc in the same graph. here is an example .

proc logistic data=test9 ;
model good_bad(event='good')= &varlist 
/outroc=_roc lackfit scale=none aggregate rsquare firth;
output out=_output h=h c=c cbar=cbar;
run;

data plot_roc;
 set roc(in=ina) _roc;
 if ina then dsn='训练集';
  else dsn='测试集';
run;

title ' ';
proc sgplot data=plot_roc  aspect=1;
 series x=_1MSPEC_ y=_SENSIT_ /group=dsn smoothconnect name='x';
 lineparm x=0 y=0 slope=1/lineattrs=(color=verylightgray);
 xaxis grid;
 yaxis grid;
 keylegend 'x' /title=' ' location=inside position=nw across=1;
run;

Re: K fold CV - how to draw "average" ROC

Melk — Mon, 10 Jul 2017 16:29:02 GMT

Well, I thought that since the training data was used to fit the test data, which is a different portion of the main dataset each time, those predicted probabilities could be used to get an "average" ROC (by basically combining each of the test datasets, each with predicted probabilities of the outcome, together). So, I basically have a dataset with my outcome, predictors, and predicted probabilities from my k fold CV and dont know how to construct the ROC curve.

Re: K fold CV - how to draw "average" ROC

StatDave — Mon, 10 Jul 2017 19:12:19 GMT

If you have a data set of actual binary responses and predicted probabilities (from whatever source), you can produce an ROC plot and analysis using the PRED= option in the ROC statement of PROC LOGISTIC as shown in this note. Notice that the predicted probabilities in each of the examples come from other modeling methods than PROC LOGISTIC.

Re: K fold CV - how to draw "average" ROC

Melk — Mon, 10 Jul 2017 19:48:19 GMT

Thanks so much!