I agree on the three points you make, just one addition on point 1, I believe based on what i have encountered before that you can get the sense that variable clustering offers by checking the IV or Gini ratio of the variables, usually variables with similar values on those metrics tend to be conflicting, the beauty though of the logistic regression is that it will eliminate one of them during the backward procedure. ... View more

From what i have seen using EM the binning or the grouping is done in accordance to maximize the gini ratio or IV , problem is that although automated it is still necessary to handle the binning manually in order to merge or split bins cause the event rate doesnt make sense otherwise and will for sure cause problems when checking the estimates of your logistic regression (they will have wrong signs). Ofcourse the more bins the higher the gini ratio but believe me the regression signs will go bananas so its quite difficult to handle a large number of bins. I will try to create some code keeping into mind a monotonically increasing or decreasing event rate (or WOE if prefered) and play interactively with it just like in EM cause i dont have it now. The gini calculation after that is solved by a sas code. I was reading this book Mamdouh Refaat - Data Preparation for Data Mining Using SAS - 2006 which has a optimal binning macro code but believe me its too complicated to read through and always gets me errors when trying to apply, long story short i believe there are smaller and more efficient ways for it and that was what i was looking for with my initial enquiry ... View more

yeas you are right, in EM and the interactive binning node i have spent a lot of times creating own splits casue the ones you get are not always so good , in most cases it doesnt create bins even there are opportunities for it, i guess i will create something in a macro form to change cut offs every time , cause i am afraid that something automated that you usually encounter means more number of bins then possible to support later on the regression cause its just looking to maximize the gini ratio. Thank you for your time ... View more

a lot you find on the net that are supposively creating optimal bins, also in data mining preparation tools their macros do not work as they should , anyways i will look into the paper you sent me , thank you for the reply ... View more

any suggestions or example  on the format ? i am not sure i follow thnx ... View more

yes i guess thats what optimal binning would do so yes i am looking for the cut off values for creating bins that would give me the highest gini ratio. I was just wondering if there was any tested way in code so i could try out instead of depending on some descriptives and if then else clauses. Basically thats what i need cause i can calcultate the gini myself once i have the groups formed. ... View more

unfortunately i am not in EM otherwise i would use interactive binning to define myself based on event rate and gini, in general binning is preferable to handle rare levels and missing as well as outliers. Besides the relationship between the continuous and target is not always linear so it would lead to instable model cause the effects wouldnt be captured ... View more

well i have lets say two continuous variables against a binary response, number of bins optimal in accordance with what i have, although i prefer lets say max 4 or 5 so i able to control the estimate signs in the regression later on ... View more