turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Find a Community

- Home
- /
- Analytics
- /
- Stat Procs
- /
- Questions on PROC LOGISTIC, k-fold cross validati...

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

07-15-2014 09:50 PM

Good day,

I am recent SAS user and kind of overwhelmed by various options.

Please, suggest right approach for (probably) standard modelling task below.

1. In PROC LOGISTIC, is there way to specify AUC (or c-statistics?) as

criteria for attribute removal/inclusion in forward/backward/step-wise

model optimization?

2. In PROC LOGISTIC, is there way to specify internal k-fold self-split

cross-validation in dataset

3. If these options are not available directly in PROC LOGISTIC,

are there frameworks / macros / code examples for

the following (e.g. backward) attribute selection process:

-*- for given dataset

-*-*- loop until no improvement to AUC

-*-*-*- loop all "current_attribute_set"

-*-*-*-*- remove attribute

-*-*-*-*- loop (e.g.) 10 times with stratified 10-fold train-test split

-*-*-*-*-*- run LOGISTIC(no_selection) in train set and Score test set

-*-*-*-*- combine all 10 test sets and get AUC (or c-statistics?)

-*-*-*- select best AUC and adjust "current_attribute_set" (IF there is an improvement!)

I found several relevant articles, but not a direct example:

SAS Programming for Data Mining: AUC calculation using Wilcoxon Rank Sum Test

41364 - ROC analysis for binary response models fit in the GLIMMIX, NLMIXED, GAM or other procedures

39724 - ROC analysis using validation data and crossvalidation

http://www2.sas.com/proceedings/sugi27/p248-27.pdf

Thanks in advance,

Alex.

Accepted Solutions

Solution

07-06-2017
10:08 AM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to AlexV

07-16-2014 03:14 PM

PROC LOGISTIC does not offer effect selection based on AUC, nor k-fold crossvalidation. However, LOGISTIC uses a 'leave one out' crossvalidation approximation when the CTABLE option is specified. See the "Details: Classification Table" section of the LOGISTIC documentation. Predicted probabilities based on this approximation are available using the PREDPROBS=X option in the OUTPUT statement. Note that k-fold crossvalidation is available in GLIMMIX and ADAPTIVEREG as mentioned in this note:

22220 - Procedures with bootstrapping, crossvalidation, or jackknifing capabilities

Note that ADAPTIVEREG can also fit a logistic model (using multivariate adaptive regression splines) and offers effect selection and k-fold crossvalidation. See its documentation for details and examples.

All Replies

Solution

07-06-2017
10:08 AM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to AlexV

07-16-2014 03:14 PM

PROC LOGISTIC does not offer effect selection based on AUC, nor k-fold crossvalidation. However, LOGISTIC uses a 'leave one out' crossvalidation approximation when the CTABLE option is specified. See the "Details: Classification Table" section of the LOGISTIC documentation. Predicted probabilities based on this approximation are available using the PREDPROBS=X option in the OUTPUT statement. Note that k-fold crossvalidation is available in GLIMMIX and ADAPTIVEREG as mentioned in this note:

22220 - Procedures with bootstrapping, crossvalidation, or jackknifing capabilities

Note that ADAPTIVEREG can also fit a logistic model (using multivariate adaptive regression splines) and offers effect selection and k-fold crossvalidation. See its documentation for details and examples.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to StatDave_sas

07-16-2014 04:49 PM

Thanks a lot , StatDev.

Your links will definitely help me

to use solid existing code and not to invent bicycle :-)

With best regards,

Alex.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to AlexV

07-01-2016 12:17 PM

Dear,

I am using SAS 9.4.

Using Proc Logistic, predprobs=crossvalidate, is it possible to output or estimate a single prediction error?

How could I generate such output?

Kind regards,