Hosmer-Lemeshaw Test in Enterprise Miner

[SlutskyFan]

Prior to using EM, I often undertook classification problems using PROC LOGISTIC. With the correct options, I could call for a Hosmer-Lemeshaw test which I believe gave me a good idea of how well my model could predict overall on a 'decile by decile' basis (as opposed to misclassification rate that is just an overall measure of fit).

Is there any way to call for this test in EM - like in the model assessment node or otherwise?

Correct me if I am wrong about the interpretation and use of the HL test. ( I know there are criticisms of it having low power for n < 400, but I typically deal in 1000's of obs)

Thanks.

[DougWielenga · Accepted solution]

The short answer is that other than Mallow's Cq, SAS Enterprise Miner does not provide a Hosmer-Lemeshow test.  You could write code to run the LOGISTIC procedure in a SAS Code node but any results would be contained in that node so it would not likely benefit the user more than just running it in SAS.

 

Looking further, a quick search of the SAS Enterprise Miner help utility (available by clicking on Help --> Contents and then clicking on the magnifying glass to reveal the search window) only shows one place where the term "Hosmer-Lemeshow" even appears and it is related to Mallow's Cq.   Here is an extract from the help: 

 

Statistical Measures

Statistical measures of model performance are based on both model error and degrees of freedom. Some SAS Enterprise Miner models, such as Decision Tree models, do not output degrees of freedom and are not suitable for benchmarking using the statistical measures listed here. The information that follows pertains to Mallows’ Cq, Akaike’s Information Criterion, Bayesian Information Criterion, and Kolmogorov-Smirnov Statistic and is suitable only for specific models.

Mallow’s Cq

Mallows' Cq (Hosmer and Lemeshow, 2000) is a variant of Mallows Cp measure (1973), which can be used to analyze linear regression models for assessment. Hosmer and Lemeshow derived the corresponding Cq statistic to evaluate logistic regression models, using the following equation

where

• q is the number of selected variables,
• p is the number of candidate variables
• 

  is the Pearson chi-square statistic for the model with p variables

• λ is the multivariable Wald statistic that measures significance of p-q coefficients that are not in the model

The expected value of C_q is q + 1. Models with C_q values near q + 1 are candidates for final models.

 In general, data mining problems have massive numbers of variables which leads to a high likelihood of missing values. Given the typical data size, several things are often true of these problems:

* missing values must be imputed

* the imputed data will have a large number of observations

* the number of usable observations will be inflated by the imputation

* the presence of imputed data makes many of the classical estimates of error more questionable.

* holdout data is present to validate/test the fitted model (empirical validation, statistical validation less critical)

For these reasons, classical statistical scenarios such as those appropriate for treatment by Hosmer-Lemeshow are not routinely calculated in SAS Enterprise Miner. 

I hope this helps!

Doug