Determining Predictive Validity

[SAS_BK]

I am running a logistic regression that includes several covariates, and has a binary outcome.  I have 2 variables that I am interested in that I want to determine and comment on their predictive validity.  I have run models with just these two predictors, and have also run models included all the covariates.  I compared ROC curves for each, and commented on p-values.

Are there other good tests that SAS can perform to check predictive validity?

[PaigeMiller]

"predictive validity" isn't really a term used in statistics; however you may mean something like "is the term statistically significant" in the model. This is the Pr>ChiSq output. A common cutoff is 0.05, if Pr>ChiSq is less than 0.05, then the term is statistically significant.

[SAS_BK]

Thanks for the reply! That makes sense, but I am hoping to further discuss the impact of these variables on the ability of the model to predict the outcome.  For instance, adding V2 gives me a significant effect with a parameter estimate of about 0.059, but only increases the AUC by 0.0029, and only has an AUC of ~0.6 with no other covariates.  This seems to me like a significant, but not very good predictor of the outcome...?

[PaigeMiller]

---

@SAS_BK wrote:

Thanks for the reply! That makes sense, but I am hoping to further discuss the impact of these variables on the ability of the model to predict the outcome.  For instance, adding V2 gives me a significant effect with a parameter estimate of about 0.059, but only increases the AUC by 0.0029, and only has an AUC of ~0.6 with no other covariates.  This seems to me like a significant, but not very good predictor of the outcome...?

---

You seem to be running into difficulties that happen when your input variables are correlated with one another. Then, there is no unique way to determine the true effect of that variable, and in fact, depending on the terms in the model, the effect of the variable might be large or might be small, and depending on the amount of correlation, a variable could have the wrong sign.

[SAS_BK]

What would you recommend to check collinearity issues between my variables?

[PaigeMiller]

The simple correlation between two variables is what you need to check.

[pau13rown]

seems to be the same Q from a different username: https://communities.sas.com/t5/SAS-Statistical-Procedures/How-to-compare-predictive-validity-of-two-...

[Ksharp]

PROC PLS is a good friend with you.

You can check the importance of variables in model by it.

The more important a variable is ,and it has more predictive power.

Check PROC PLS 's documentation, there is already an example about it .

[StatDave]

You can use the ROCCONTRAST statement to compare pairs of your one-variable models to get a formal test of the difference in AUCs. This will tell you if the difference in AUC is significantly different for two variables when considered alone. However, a test of the changes in AUCs due to each of two variables in a single, multi-variable model isn't possible. And even if it were, there is always the issue of statistical vs. practical significance - what is a "big" change in AUC for your purposes might not be an amount that a statistical test would detect. That ultimately requires having the right sample size.