Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Find a Community

- Home
- /
- Analytics
- /
- Stat Procs
- /
- Determining Predictive Validity

- RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

06-18-2018 12:44 PM

I am running a logistic regression that includes several covariates, and has a binary outcome. I have 2 variables that I am interested in that I want to determine and comment on their predictive validity. I have run models with just these two predictors, and have also run models included all the covariates. I compared ROC curves for each, and commented on p-values.

Are there other good tests that SAS can perform to check predictive validity?

- Mark as New
- Bookmark
- Subscribe
- RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to SAS_BK

06-18-2018 12:49 PM

"predictive validity" isn't really a term used in statistics; however you may mean something like "is the term statistically significant" in the model. This is the Pr>ChiSq output. A common cutoff is 0.05, if Pr>ChiSq is less than 0.05, then the term is statistically significant.

--

Paige Miller

Paige Miller

- Mark as New
- Bookmark
- Subscribe
- RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to PaigeMiller

06-18-2018 01:15 PM

Thanks for the reply! That makes sense, but I am hoping to further discuss the impact of these variables on the ability of the model to predict the outcome. For instance, adding V2 gives me a significant effect with a parameter estimate of about 0.059, but only increases the AUC by 0.0029, and only has an AUC of ~0.6 with no other covariates. This seems to me like a significant, but not very good predictor of the outcome...?

- Mark as New
- Bookmark
- Subscribe
- RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to SAS_BK

06-18-2018 01:28 PM

@SAS_BK wrote:

You seem to be running into difficulties that happen when your input variables are correlated with one another. Then, there is no unique way to determine the true effect of that variable, and in fact, depending on the terms in the model, the effect of the variable might be large or might be small, and depending on the amount of correlation, a variable could have the wrong sign.

--

Paige Miller

Paige Miller

- Mark as New
- Bookmark
- Subscribe
- RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to PaigeMiller

06-18-2018 01:29 PM

What would you recommend to check collinearity issues between my variables?

- Mark as New
- Bookmark
- Subscribe
- RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to SAS_BK

06-18-2018 01:31 PM

The simple correlation between two variables is what you need to check.

--

Paige Miller

Paige Miller

- Mark as New
- Bookmark
- Subscribe
- RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to SAS_BK

06-18-2018 04:08 PM

seems to be the same Q from a different username: https://communities.sas.com/t5/SAS-Statistical-Procedures/How-to-compare-predictive-validity-of-two-...

--------------

blog: papersandprograms.com

blog: papersandprograms.com

- Mark as New
- Bookmark
- Subscribe
- RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to SAS_BK

06-19-2018 10:36 AM

PROC PLS is a good friend with you.

You can check the importance of variables in model by it.

The more important a variable is ,and it has more predictive power.

Check PROC PLS 's documentation, there is already an example about it .

- Mark as New
- Bookmark
- Subscribe
- RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to SAS_BK

06-20-2018 09:06 AM

You can use the ROCCONTRAST statement to compare pairs of your one-variable models to get a formal test of the difference in AUCs. This will tell you if the difference in AUC is significantly different for two variables when considered alone. However, a test of the changes in AUCs due to each of two variables in a single, multi-variable model isn't possible. And even if it were, there is always the issue of statistical vs. practical significance - what is a "big" change in AUC for your purposes might not be an amount that a statistical test would detect. That ultimately requires having the right sample size.