I have developed several predictive (logit) models, all with sample sizes ~ 10-20K or more. Typically my model results are postive- rejecting B's = 0, no multicollinearity, significant coefficients, high AROC, high percentage of correct predictions for binary groups (0's and 1's). However, when I conduct the Hosmer Lemeshaw test (in base SAS), most of the time it is significant, indicating lack of fit.  I've read several places- (see links and refs below- but few 'hard' references) that  "As the sample size gets large, the H-L statistic can find smaller and smaller differences between observed and model-predicted values to be significant." leading to an erroneous conclusion about model fit. Do you agree? Do you have any stronger references than I have on this? If you agree, then how large is too large when it comes to sample size? And, since SAS Enterprise Miner doesn't include the HL test (I typically use a code node to do this in EM) I'm curious if people care much about it. In fact, SAS tech support told me it was not included  as an option in EM because the HL test is a holdover from the past, and basically doesn't fit the data mining paradigm (i.e. large data sets?)  that SAS EM is built upon.  Thanks in advance. NCSU Faculty Web Site , the STATA stat listserv,  from quality forum p. 5, another academic web page, and  JOURNAL OF PALLIATIVE MEDICINE Volume 12, Number 2, 2009 Prediction of Pediatric Death in the Yearafter Hospitalization: A Population-Level  Retrospective Cohort Study Chris Feudtner, M.D., Ph.D., M.P.H.,1,5 Kari R. Hexem, M.P.H.,1 Mayadah Shabbout, M.S.,3James A. Feinstein, M.D.,1 Julie Sochalski, Ph.D., R.N.,4,5 and Jeffery H. Silber, M.D., Ph.D.2,5 http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2656437/  "‘The Hosmer-Lemeshow test detected astatistically significant degree of miscalibration in both models, due to the extremely large sample size of the models, as the differences between the observed and expected values within each group are relatively small." ... View more

I have 2 questions: 1) Cross validated Decision trees: Under the panel for cross validation, if you select 'yes' and for number of subsets ='10' and number of repeats ='10' , are these results equivalent to 10-fold cross validation? Cross validated regression: When you choose 'cross validation misclassification' as your selection criteria for the logistic regression node, it seems that this is similar to an n-fold cross validation where n = the total # of observations in your data set? Is that correct? 2) With cross validation techniques, do you still partition your data into training and validation subsets? I'm thinking, based on sas help documentation, since it is primarily used when small data sets are not large enough for partitioning, you wouldn't generally use a cross validation technique with partitioned data. ... View more

The documentation for output in EM is not thorough. Basically I'd like to know 1)how the column Mean Posterier Probability is computed. I'd also like to verify 2) that the values under Cumulative % Response are based on *OBSERVED* class values, not predicted. Below is my output and the base SAS code I used to try to verify my conception of this output using the scored validation data from EM, but I'm not able to do so. Could someone help me with interpretation? For more details, say I have the following Output From EM: Mean Cumulative % Cumulative Number of Posterior Percentile Gain Lift Lift Response % Response Observations Probabil 5 143.210 2.43210 2.43210 100.000 100.000 10 0.81433 10 106.728 1.70247 2.06728 70.000 85.000 10 0.69886 15 78.354 1.21605 1.78354 50.000 73.333 10 0.64560 20 64.167 1.21605 1.64167 50.000 67.500 10 0.59028 25 50.790 0.97284 1.50790 40.000 62.000 10 0.53578 30 45.926 1.21605 1.45926 50.000 60.000 10 0.51555 35 44.516 1.35117 1.44516 55.556 59.420 9 0.47502 40 35.459 0.72963 1.35459 30.000 55.696 10 0.44177 45 28.437 0.72963 1.28437 30.000 52.809 10 0.41554 50 20.377 0.48642 1.20377 20.000 49.495 10 0.39732 55 18.258 0.97284 1.18258 40.000 48.624 10 0.36956 60 16.495 0.97284 1.16495 40.000 47.899 10 0.33354 65 18.777 1.45926 1.18777 60.000 48.837 10 0.31359 70 18.080 1.08093 1.18080 44.444 48.551 9 0.30182 75 13.388 0.48642 1.13388 20.000 46.622 10 0.27938 80 9.291 0.48642 1.09291 20.000 44.937 10 0.25124 85 4.233 0.24321 1.04233 10.000 42.857 10 0.20888 90 2.476 0.72963 1.02476 30.000 42.135 10 0.15998 95 2.200 0.97284 1.02200 40.000 42.021 10 0.08486 100 0.000 0.54047 1.00000 22.222 41.117 9 0.00652 By exporting the scored validation data from EM to csv and reading into SAS,sorting by descending predicted probabilty and using PROC SQL to ad a row number (call this data 'VALIDATE' for reference) I can recreate the values under 'Cumulative % Response' by /* CUM % RESP FOR 10TH PERCENTILE */ PROC FREQ DATA = ; TABLES OBS_TARGET; WHERE ROW_NUM =20; RUN; This tells me that the value in 'Cumulative % Response' is based on the *OBSERVED* response not the predicted response. ( I can write code to get predicted response outside EM in base SAS if necessary) I am trying to interpret the values under Mean Posterior Probability. I just assumed it was the mean of the predicted probability for the target class level in that slice of the sorted data. However, if I invoke the following in base SAS code: PROC MEANS DATA = VALIDATE; VAR Predicted_Term_GPA_Less_than_1_; /* CREATED IN EM */ WHERE ROW_NUM =20; I do not get the same number reported in the column for Mean Posterior Probabilty for the 10th percentile. I do match exactly for the 5th percentile, but it must be a coincidence. Can someone tell me how this value is computed? ... View more

Via the settings in the decision tree node, is it possible to mimic random forests? I've read SAS help, Applied Analytics Using SAS Enterprise Miner, and done a google search, but I'm not getting very far. ( I think I can do k-fold cross validation using the cross validation settings for the decision tree, but I'm not sure I'm doing it correctly) Any suggestions or references? I feel like I've got a good basic feel for enterprise miner, and a decent theoretical background in various machine learning techniques (I've read a lot of Elements of Statistical Learning: Data Mining, Inference and Prediction (http://www-stat.stanford.edu/~tibs/ElemStatLearn/) and wathced Andrew Ng's Machine learning lectures (http://www.youtube.com/view_play_list?p=A89DCFA6ADACE599). I need some more advanced references using Enterprise Miner. Thanks. ... View more