Programming the statistical procedures from SAS

Determining Predictive Validity

Reply
Occasional Contributor
Posts: 5

Determining Predictive Validity

I am running a logistic regression that includes several covariates, and has a binary outcome.  I have 2 variables that I am interested in that I want to determine and comment on their predictive validity.  I have run models with just these two predictors, and have also run models included all the covariates.  I compared ROC curves for each, and commented on p-values.

 

Are there other good tests that SAS can perform to check predictive validity?

Respected Advisor
Posts: 3,059

Re: Determining Predictive Validity

"predictive validity" isn't really a term used in statistics; however you may mean something like "is the term statistically significant" in the model. This is the Pr>ChiSq output. A common cutoff is 0.05, if Pr>ChiSq is less than 0.05, then the term is statistically significant.

--
Paige Miller
Occasional Contributor
Posts: 5

Re: Determining Predictive Validity

Posted in reply to PaigeMiller

Thanks for the reply! That makes sense, but I am hoping to further discuss the impact of these variables on the ability of the model to predict the outcome.  For instance, adding V2 gives me a significant effect with a parameter estimate of about 0.059, but only increases the AUC by 0.0029, and only has an AUC of ~0.6 with no other covariates.  This seems to me like a significant, but not very good predictor of the outcome...?

Respected Advisor
Posts: 3,059

Re: Determining Predictive Validity


@SAS_BK wrote:

Thanks for the reply! That makes sense, but I am hoping to further discuss the impact of these variables on the ability of the model to predict the outcome.  For instance, adding V2 gives me a significant effect with a parameter estimate of about 0.059, but only increases the AUC by 0.0029, and only has an AUC of ~0.6 with no other covariates.  This seems to me like a significant, but not very good predictor of the outcome...?


You seem to be running into difficulties that happen when your input variables are correlated with one another. Then, there is no unique way to determine the true effect of that variable, and in fact, depending on the terms in the model, the effect of the variable might be large or might be small, and depending on the amount of correlation, a variable could have the wrong sign.

--
Paige Miller
Occasional Contributor
Posts: 5

Re: Determining Predictive Validity

Posted in reply to PaigeMiller

What would you recommend to check collinearity issues between my variables?

Respected Advisor
Posts: 3,059

Re: Determining Predictive Validity

The simple correlation between two variables is what you need to check.

--
Paige Miller
Regular Contributor
Posts: 164

Re: Determining Predictive Validity

Super User
Posts: 10,787

Re: Determining Predictive Validity

PROC PLS is a good friend with you.

You can check the importance of variables in model by it.

The more important a variable is ,and it has more predictive power.

Check PROC PLS 's documentation, there is already an example about it .

SAS Employee
Posts: 386

Re: Determining Predictive Validity

You can use the ROCCONTRAST statement to compare pairs of your one-variable models to get a formal test of the difference in AUCs. This will tell you if the difference in AUC is significantly different for two variables when considered alone. However, a test of the changes in AUCs due to each of two variables in a single, multi-variable model isn't possible. And even if it were, there is always the issue of statistical vs. practical significance - what is a "big" change in AUC for your purposes might not be an amount that a statistical test would detect. That ultimately requires having the right sample size.

Ask a Question
Discussion stats
  • 8 replies
  • 107 views
  • 0 likes
  • 5 in conversation