turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Find a Community

- Home
- /
- Analytics
- /
- Data Mining
- /
- Hosmer-Lemeshaw Test in Enterprise Miner

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

11-17-2010 11:28 AM

Prior to using EM, I often undertook classification problems using PROC LOGISTIC. With the correct options, I could call for a Hosmer-Lemeshaw test which I believe gave me a good idea of how well my model could predict overall on a 'decile by decile' basis (as opposed to misclassification rate that is just an overall measure of fit).

Is there any way to call for this test in EM - like in the model assessment node or otherwise?

Correct me if I am wrong about the interpretation and use of the HL test. ( I know there are criticisms of it having low power for n < 400, but I typically deal in 1000's of obs)

Thanks.

Is there any way to call for this test in EM - like in the model assessment node or otherwise?

Correct me if I am wrong about the interpretation and use of the HL test. ( I know there are criticisms of it having low power for n < 400, but I typically deal in 1000's of obs)

Thanks.

Accepted Solutions

Solution

Monday

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Monday

Statistical measures of model performance are based on both model error and degrees of freedom. Some SAS Enterprise Miner models, such as Decision Tree models, do not output degrees of freedom and are not suitable for benchmarking using the statistical measures listed here. The information that follows pertains to Mallows’ Cq, Akaike’s Information Criterion, Bayesian Information Criterion, and Kolmogorov-Smirnov Statistic and is suitable only for specific models.

**Mallow’s Cq**

Mallows' Cq (Hosmer and Lemeshow, 2000) is a variant of Mallows Cp measure (1973), which can be used to analyze linear regression models for assessment. Hosmer and Lemeshow derived the corresponding Cq statistic to evaluate logistic regression models, using the following equation

where

*q*is the number of selected variables,*p*is the number of candidate variables-
is the Pearson chi-square statistic for the model with p variables

- λ is the multivariable Wald statistic that measures significance of p-q coefficients that are not in the model

The expected value of C_{q} is q + 1. Models with C_{q} values near q + 1 are candidates for final models.

In general, data mining problems have massive numbers of variables which leads to a high likelihood of missing values. Given the typical data size, several things are often true of these problems:

* missing values must be imputed

* the imputed data will have a large number of observations

* the number of usable observations will be inflated by the imputation

* the presence of imputed data makes many of the classical estimates of error more questionable.

* holdout data is present to validate/test the fitted model (empirical validation, statistical validation less critical)

For these reasons, classical statistical scenarios such as those appropriate for treatment by Hosmer-Lemeshow are not routinely calculated in SAS Enterprise Miner.

I hope this helps!

Doug

All Replies

Solution

Monday

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Monday

Statistical measures of model performance are based on both model error and degrees of freedom. Some SAS Enterprise Miner models, such as Decision Tree models, do not output degrees of freedom and are not suitable for benchmarking using the statistical measures listed here. The information that follows pertains to Mallows’ Cq, Akaike’s Information Criterion, Bayesian Information Criterion, and Kolmogorov-Smirnov Statistic and is suitable only for specific models.

**Mallow’s Cq**

Mallows' Cq (Hosmer and Lemeshow, 2000) is a variant of Mallows Cp measure (1973), which can be used to analyze linear regression models for assessment. Hosmer and Lemeshow derived the corresponding Cq statistic to evaluate logistic regression models, using the following equation

where

*q*is the number of selected variables,*p*is the number of candidate variables-
is the Pearson chi-square statistic for the model with p variables

- λ is the multivariable Wald statistic that measures significance of p-q coefficients that are not in the model

The expected value of C_{q} is q + 1. Models with C_{q} values near q + 1 are candidates for final models.

In general, data mining problems have massive numbers of variables which leads to a high likelihood of missing values. Given the typical data size, several things are often true of these problems:

* missing values must be imputed

* the imputed data will have a large number of observations

* the number of usable observations will be inflated by the imputation

* the presence of imputed data makes many of the classical estimates of error more questionable.

* holdout data is present to validate/test the fitted model (empirical validation, statistical validation less critical)

For these reasons, classical statistical scenarios such as those appropriate for treatment by Hosmer-Lemeshow are not routinely calculated in SAS Enterprise Miner.

I hope this helps!

Doug