topic Re: How to compare predictive validity of two variables in multivariable model?? in Statistical Procedures

How to compare predictive validity of two variables in multivariable model??

iressa131 — Sun, 17 Jun 2018 21:18:04 GMT

Hello,

I am looking to assess the predictive validity of two measures (M1 and M2) on a binary outcome. After constructing my model, do I just calculate odds ratios and compare the odds ratios for the two measures? I am not looking to assess the predictive value of the whole model just 2 specific predictors. Thanks for any clarification!

Re: How to compare predictive validity of two variables in multivariable model??

PGStats — Sun, 17 Jun 2018 22:11:44 GMT

Compare the AIC for the two fits in the model fit statistics table. The model with the lower AIC is better.

Re: How to compare predictive validity of two variables in multivariable model??

Reeza — Sun, 17 Jun 2018 23:53:35 GMT

For logistic regression make sure to look at the confusion matrix and the AUC as well.

@iressa131 wrote:

Hello,

I am looking to assess the predictive validity of two measures (M1 and M2) on a binary outcome. After constructing my model, do I just calculate odds ratios and compare the odds ratios for the two measures? I am not looking to assess the predictive value of the whole model just 2 specific predictors. Thanks for any clarification!

Re: How to compare predictive validity of two variables in multivariable model??

pau13rown — Mon, 18 Jun 2018 01:33:54 GMT

a sas macro is available for pencina's net reclassification index: https://analytics.ncsu.edu/sesug/2010/SDA07.Kennedy.pdf

pencina's method would be relevant for the scenario you describe ie 'not the whole model' just the added variables: "Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond" https://www.ncbi.nlm.nih.gov/pubmed/17569110.

I'm going to write a brief blog post about about it's implementation when i find time....

Re: How to compare predictive validity of two variables in multivariable model??

iressa131 — Mon, 18 Jun 2018 11:59:50 GMT

Thanks for the help! I didn't realize I needed two separate models. Now if I then wanted to determine if M1 and M2 are independently associated with the outcome, would I construct a new model containing both in order to test for independence?

Re: How to compare predictive validity of two variables in multivariable model??

PaigeMiller — Mon, 18 Jun 2018 12:41:05 GMT

@iressa131 wrote:
Thanks for the help! I didn't realize I needed two separate models. Now if I then wanted to determine if M1 and M2 are independently associated with the outcome, would I construct a new model containing both in order to test for independence?

The way you have worded the question, I don't think this is a question that can be answered by fitting a model that has both variables.

If M1 and M2 have a correlation of zero, they have independent effects on the outcome. If they have a correlation that is not zero, then the effect of M1 and M2 will be correlated and not independent.

Re: How to compare predictive validity of two variables in multivariable model??

iressa131 — Mon, 18 Jun 2018 15:08:26 GMT

Okay I think I understand that thank you! If its not too much trouble, could you clarify why 2 separate models is better for my initial aim of comparing predictive validity of M1 and M2 on the outcome? Would fitting a model with both M1 and M2 allow for potential bias?