Hi Miguel! Thank you for continuing the conversation.  I created this test data set to compare against different implementations of "optimal binning."  I designed it specifically to be tricky to find the best four bins.  In practice, if you're looking in detail at only a few predictors, you wouldn't restrict yourself so severely, and if there were six perfect bins, as in this case, you'd find them.  But when you're dealing with hundreds or thousands of predictors as you might in a typical data mining exercise, then it's not at all uncommon to go with something like the best four or five bins, and you'd like to feel confident that your algorithm can locate them.  Right now I feel a bit queasy about Enterprise Miner.  If I allowed the Transform Variables node to search for 25 bins with my example data set, it found the six perfect ones.  But if I limited it to 17 bins, it only found four of the uniform bins, and it split the two smallest ones into three pieces, one of which had only 92 points, less than 1% of the data.  A bin that small could be significant, or it could be a blip.  In my opinion, that's another weakness of the Transform Variables node -- you can't specify a minimum bin size.  The R package, smbinning, has the opposite constraint, you can specify a minimum bin size, but not a maximum bin number; by the way, the smbinning function has a default minimum bin size of 5%, which is why it finds five bins for this data set with its default setting.  If you submit: bt3_xg.t3<-smbinning(bt3_xg.data,y="bt3_xg",x="ipxg",p=0.04) it will find the six uniform bins (because the two smallest bins each have 4% of the data). About three years ago Ivan Oliveira in the Enterprise Miner development team described to me some improvements in optimal binning that were being added to EM 7.1.  It strikes me now that he was talking only about the Interactive Grouping node in the Credit Scoring application, but at the time I thought he was referring to the Transform Variables node.  I wish SAS would get the optimal binning in the Transform Variables node up to speed with the Interactive Grouping node.  Okay, I promise to stop ranting now (or at least in the not too distant future). Chi-square is a useful measure of dependence / association that goes back to Karl Pearson and the early days of the science of Statistics in the late nineteenth / early twentieth centuries.  One of the best things about it is that its distribution is well-understood, so you can generate p-values and perform significance tests based on your results.  I don't know whether anyone has ever bothered to do that for Gini or entropy value distributions.  The original automated decision tree algorithms eventually coalesced (about forty years ago?) into CHAID, which uses chi-square as its splitting rule measure of association.  One useful property of chi-square is the way it scales fractally -- if you break up a big group into k identical subgroups, then the sum of the subgroup chi-squares equals the big group chi-square.  But that's actually a bit of a drawback for optimal binning, because if a bin can split into two identical sub-bins, you'd rather just keep the original large bin, and this gives you no incentive to do so.  Most of the association measures I've seen: Gini, entropy, information value, weight of evidence, within group sum of squares,... all have the property that they improve with increased granularity, so that if you removed all restrictions on the number or size of bins, they'd give you as many bins as data points; even with the drawback described above, chi-square will favor some lumpiness over complete granularity.  And if you remove the chi-square denominators, so that you take the sum of the squared differences between the actual and expected number of hits in each group, you naturally seek out larger bins. Thanks! ... View more

Hi Miguel! Thank you for running the Interactive Grouping node and reporting back the results.  For your results you got five bins.  I had been restricting my search to four bins.  If I let the Enterprise Miner Transform Variables node try for five bins, it comes up with: 1 - 2000 2001 - 5117 5118 - 6800 6801 - 7600 7601 - 10000 Interestingly enough, it's not a refinement of its four bin solution, but at least it has a higher Gini: 86.829 > 78.115.  But it's not nearly as good as the Interactive Grouping solution -- are you paying attention, Enterprise Miner development team? But you know what?  Here's the "proc eyeball" five bin solution: 1 - 2000 2001 - 4000 4001 - 6800 6801 - 7600 7601 - 10000 and the Gini is... 99.358 So I'm not ready to give up on "proc eyeball" just yet. By the way, if you run smbinning as in the code I posted previously, but drop the p=0.2 restriction, you'll get the same five bin solution as "proc eyeball." Thanks! ... View more

Hi Miguel! Thank you for responding.  Gergely's code might be slightly more efficient (it definitely would be if he'd put the target assignment before the start of the inner loop), but my code to create the SAS data set is similar: %let pattern  = %str(0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 1 0 1 1 1 1 1 1) ; %let pattern_length = %length(&pattern.) ; %let pattern_count = %sysfunc(countw(&pattern.)) ; %let expansion = 400 ; %* number of data points per pattern point, data size = pattern_count*expansion ; %let number_of_points = %sysevalf(&pattern_count. * &expansion.) ; data staging.gini_test_input ; keep interval_predictor binary_target ; length pattern $&pattern_length.. ; pattern = "&pattern." ; put pattern= ; do paix = 1 to &pattern_count. ;   binary_target = input(scan(pattern, paix), best32.) ;   do expix = 1 to &expansion. ;    interval_predictor + 1 ;    output ;   end ; end ; run ; I'm also curious to see how the Interactive Grouping Node will partition this example.  We don't license the Credit Scoring Application, so I can't test it directly. The R code is for using smbinning is: library(smbinning) bt3p<-as.integer(c(rep(0,5),rep(1,5),rep(0,7),1,0,rep(1,6))) bt3_xg<-rep(bt3p,each=400) ipxg<-as.double(c(1:length(bt3_xg))) bt3_xg.data<-data.frame(bt3_xg,ipxg) bt3_xg.t1<-smbinning(bt3_xg.data,y="bt3_xg",x="ipxg",p=0.2) Thanks! ... View more