topic Re: Determine a cutpoint (from ROC curve) after multiple imputation in Statistical Procedures

Determine a cutpoint (from ROC curve) after multiple imputation

ZoeUH — Fri, 18 Feb 2022 10:07:17 GMT

Dear,

I would like to determine a cutpoint for a diagnositic test. Similar to what is shown in the following document: https://www.pharmasug.org/proceedings/2016/SP/PharmaSUG-2016-SP11.pdf

However, the difficulty in my situation is that I do have some missing data. Therefore, I proceeded in the following way:

Performed multiple imputation using proc MI.
Performed a logistic regression model for each of the imputed datasets and outputed the results using the outroc option in the model statement.
Combined the results from using proc MIanalyze.
Used proc logistic with the options inest = (combined parameter estimates from proc mianalyze) and maxiter=0.

These were the steps suggested by this thread (https://communities.sas.com/t5/Statistical-Procedures/How-to-obtain-an-average-ROC-curve-using-multiple-imuputation/td-p/540376) to obtain a single ROC curve.

My question is, how do I get standard errors around my cutpoints obtained from this ROC curve to correctly display the variance from the different imputed datasets? Or do I just assume that on average my cutpoint will be X from the single ROC curve? Could someone please provide some insight on how to correctly account for the variation from the different imputed datasets when calculating a cutpoint?

Thanks in advance,

Zoë

Re: Determine a cutpoint (from ROC curve) after multiple imputation

StatDave — Fri, 18 Feb 2022 16:03:47 GMT

You don't need standard errors on the cutpoints (predicted probabilities) to compute statistics for finding the optimal cutpoint if that is the goal. With the OUTROC= data set from your step 4 analysis, and adding the data set of predicted probabilities (from adding the OUTPUT OUT= P= statement) in that step, you can then use the ROCPLOT macro to find an optimal cutpoint based on various statistics such as Youden's index or the minimum distance. This only requires the point estimates of the probabilities.

Re: Determine a cutpoint (from ROC curve) after multiple imputation

ZoeUH — Mon, 21 Feb 2022 08:51:39 GMT

Hi StatDave_sas,

Thanks for your reponse. If I understand you correctly: when the aim is determining the optimal cutpoint, it is not necessary to have standard errors (from the imputated datasets). Then I have a follow-up question: How would you interpret the obtained cutpoints (for instance based on the Youden index)? As an average cutpoint over all imputed datasets?

Re: Determine a cutpoint (from ROC curve) after multiple imputation

StatDave — Mon, 21 Feb 2022 15:00:14 GMT

They are simply the cutpoints from the final model fit to imputed data. Recall that the cutpoints are just predicted probabilities which are computed from the final parameter estimates and the data values (imputed) in the observations.

Re: Determine a cutpoint (from ROC curve) after multiple imputation

ZoeUH — Tue, 22 Feb 2022 08:32:32 GMT

Thanks a lot for your explanation. It is clear to me.