How to calculate the ROC Index Confidence Interval in Enterprise Miner 14.1

[danielsen]

Dear Community

I have constructed a workflow as below. My data contain categorical predictors, natural language and a target. The text miner nodes creates predictors from the text. Three models were trained: a Neural Network, Lasso with cross validation (LARS node) and a Random Forest. The first Random Forest node makes predictor selection. I used the ROC Index in the Model Comparison Node for model selection. The chosen model (Random Forest) was then validated using an independent test set. I would like to publish the results (ROC Index for the test set) but in my field (biomedical research) 95% confidence intervals (CI) are expected/mandatory.

As far as I understand, confidence intervals can not be produced by the Model Comparison node.

Does anyone know how to calculate a ROC Index 95% CI (for the test set) for for a Random Forest model (or a Lasso model) using SAS Enterprise Miner 14.1 implemented in the workflow above?

Thanks.

-Andreas

[RalphAbbey]

Unfortunately to my knowledge, Enterprise Miner does not compute confidence intervals for the ROC index.

I hadn't heard of confidence intervals on ROC curves before, so I decided to look it up. I found several different results, so unfortunately I'm not sure what method you need for your work.

Ultimately, if you know the equations behind the confidence interval calculation, then you can use a SAS code node to do the computations for the ROC index and the confidence interval after the modeling node.

I hope this helps guide your work. Let me know if there's anything I can help with here.