BookmarkSubscribeRSS Feed
SAS_BK
Calcite | Level 5

I am running a logistic regression that includes several covariates, and has a binary outcome.  I have 2 variables that I am interested in that I want to determine and comment on their predictive validity.  I have run models with just these two predictors, and have also run models included all the covariates.  I compared ROC curves for each, and commented on p-values.

 

Are there other good tests that SAS can perform to check predictive validity?

8 REPLIES 8
PaigeMiller
Diamond | Level 26

"predictive validity" isn't really a term used in statistics; however you may mean something like "is the term statistically significant" in the model. This is the Pr>ChiSq output. A common cutoff is 0.05, if Pr>ChiSq is less than 0.05, then the term is statistically significant.

--
Paige Miller
SAS_BK
Calcite | Level 5

Thanks for the reply! That makes sense, but I am hoping to further discuss the impact of these variables on the ability of the model to predict the outcome.  For instance, adding V2 gives me a significant effect with a parameter estimate of about 0.059, but only increases the AUC by 0.0029, and only has an AUC of ~0.6 with no other covariates.  This seems to me like a significant, but not very good predictor of the outcome...?

PaigeMiller
Diamond | Level 26

@SAS_BK wrote:

Thanks for the reply! That makes sense, but I am hoping to further discuss the impact of these variables on the ability of the model to predict the outcome.  For instance, adding V2 gives me a significant effect with a parameter estimate of about 0.059, but only increases the AUC by 0.0029, and only has an AUC of ~0.6 with no other covariates.  This seems to me like a significant, but not very good predictor of the outcome...?


You seem to be running into difficulties that happen when your input variables are correlated with one another. Then, there is no unique way to determine the true effect of that variable, and in fact, depending on the terms in the model, the effect of the variable might be large or might be small, and depending on the amount of correlation, a variable could have the wrong sign.

--
Paige Miller
SAS_BK
Calcite | Level 5

What would you recommend to check collinearity issues between my variables?

PaigeMiller
Diamond | Level 26

The simple correlation between two variables is what you need to check.

--
Paige Miller
Ksharp
Super User

PROC PLS is a good friend with you.

You can check the importance of variables in model by it.

The more important a variable is ,and it has more predictive power.

Check PROC PLS 's documentation, there is already an example about it .

StatDave
SAS Super FREQ

You can use the ROCCONTRAST statement to compare pairs of your one-variable models to get a formal test of the difference in AUCs. This will tell you if the difference in AUC is significantly different for two variables when considered alone. However, a test of the changes in AUCs due to each of two variables in a single, multi-variable model isn't possible. And even if it were, there is always the issue of statistical vs. practical significance - what is a "big" change in AUC for your purposes might not be an amount that a statistical test would detect. That ultimately requires having the right sample size.

sas-innovate-2024.png

Available on demand!

Missed SAS Innovate Las Vegas? Watch all the action for free! View the keynotes, general sessions and 22 breakouts on demand.

 

Register now!

What is ANOVA?

ANOVA, or Analysis Of Variance, is used to compare the averages or means of two or more populations to better understand how they differ. Watch this tutorial for more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 8 replies
  • 1545 views
  • 0 likes
  • 5 in conversation