Solved
Contributor
Posts: 47

# Hosmer-Lemeshaw Test in Enterprise Miner

Prior to using EM, I often undertook classification problems using PROC LOGISTIC. With the correct options, I could call for a Hosmer-Lemeshaw test which I believe gave me a good idea of how well my model could predict overall on a 'decile by decile' basis (as opposed to misclassification rate that is just an overall measure of fit).

Is there any way to call for this test in EM - like in the model assessment node or otherwise?

Correct me if I am wrong about the interpretation and use of the HL test. ( I know there are criticisms of it having low power for n < 400, but I typically deal in 1000's of obs)

Thanks.

Accepted Solutions
Solution
‎08-14-2017 03:07 PM
SAS Employee
Posts: 179

## Re: Hosmer-Lemeshaw Test in Enterprise Miner

Posted in reply to SlutskyFan

### Statistical Measures

Statistical measures of model performance are based on both model error and degrees of freedom. Some SAS Enterprise Miner models, such as Decision Tree models, do not output degrees of freedom and are not suitable for benchmarking using the statistical measures listed here. The information that follows pertains to Mallows’ Cq, Akaike’s Information Criterion, Bayesian Information Criterion, and Kolmogorov-Smirnov Statistic and is suitable only for specific models.

Mallow’s Cq

Mallows' Cq (Hosmer and Lemeshow, 2000) is a variant of Mallows Cp measure (1973), which can be used to analyze linear regression models for assessment. Hosmer and Lemeshow derived the corresponding Cq statistic to evaluate logistic regression models, using the following equation

where

• q is the number of selected variables,
• p is the number of candidate variables
• is the Pearson chi-square statistic for the model with p variables

• λ is the multivariable Wald statistic that measures significance of p-q coefficients that are not in the model

The expected value of Cq is q + 1. Models with Cq values near q + 1 are candidates for final models.

In general, data mining problems have massive numbers of variables which leads to a high likelihood of missing values. Given the typical data size, several things are often true of these problems:

* missing values must be imputed

* the imputed data will have a large number of observations

* the number of usable observations will be inflated by the imputation

* the presence of imputed data makes many of the classical estimates of error more questionable.

* holdout data is present to validate/test the fitted model (empirical validation, statistical validation less critical)

For these reasons, classical statistical scenarios such as those appropriate for treatment by Hosmer-Lemeshow are not routinely calculated in SAS Enterprise Miner.

I hope this helps!

Doug

All Replies
Solution
‎08-14-2017 03:07 PM
SAS Employee
Posts: 179

## Re: Hosmer-Lemeshaw Test in Enterprise Miner

Posted in reply to SlutskyFan

### Statistical Measures

Statistical measures of model performance are based on both model error and degrees of freedom. Some SAS Enterprise Miner models, such as Decision Tree models, do not output degrees of freedom and are not suitable for benchmarking using the statistical measures listed here. The information that follows pertains to Mallows’ Cq, Akaike’s Information Criterion, Bayesian Information Criterion, and Kolmogorov-Smirnov Statistic and is suitable only for specific models.

Mallow’s Cq

Mallows' Cq (Hosmer and Lemeshow, 2000) is a variant of Mallows Cp measure (1973), which can be used to analyze linear regression models for assessment. Hosmer and Lemeshow derived the corresponding Cq statistic to evaluate logistic regression models, using the following equation

where

• q is the number of selected variables,
• p is the number of candidate variables
• is the Pearson chi-square statistic for the model with p variables

• λ is the multivariable Wald statistic that measures significance of p-q coefficients that are not in the model

The expected value of Cq is q + 1. Models with Cq values near q + 1 are candidates for final models.

In general, data mining problems have massive numbers of variables which leads to a high likelihood of missing values. Given the typical data size, several things are often true of these problems:

* missing values must be imputed

* the imputed data will have a large number of observations

* the number of usable observations will be inflated by the imputation

* the presence of imputed data makes many of the classical estimates of error more questionable.

* holdout data is present to validate/test the fitted model (empirical validation, statistical validation less critical)

For these reasons, classical statistical scenarios such as those appropriate for treatment by Hosmer-Lemeshow are not routinely calculated in SAS Enterprise Miner.

I hope this helps!

Doug

☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
• 1 reply
• 179 views
• 0 likes
• 2 in conversation