02-03-2013 08:41 PM
The following products are analyzed in this benchmark study:
Hadoop); SAS Enterprise Miner 12.1 client
SAS Enterprise Miner 12.1, SAS 9.3
The SAS applications are commercial products. The R2.15.1 and the Mahout 7.0 are open source. There were three methods that were used across all of the applications to build the models: logistic regression, decision tree, and random forest.
One thing that was not clear from this study was how the results were derived. Although under the section called "Model Quality" the study states that “Standardized training and validation data sets stratified by the target were used across the model test suite,” it’s not clear as to what standards were applied. Were the standards based on how a human would classify the events or by some other method? If so, what was the method?
02-08-2013 09:50 AM
Perhaps knowing that standardized training and validation data sets stratified by the target were used across the model test suite mean that all of the models for each package were evaluated using the same data.
And as cited: model quality was assessed through common model quality measures (Han & Kamber 2006), ie.
• cumulative lift in the first decile,
• percentage of correctly classified events (often called event precision), and
• overall percentage of correct classification
Since the analysis used historical data, the event value for the target is known. You need historical data with known values to do predictive modeling. The predictions from each model were then compared on the KNOWN common validation data to evaluate model quality using the statistics above.
As a side note - oftentimes there is an improvement in predictive model performance with the inclusion of variables derived from text data. At last years Analytics 2012 event, United Health Group - indicated that they generally found that predictive models improved significantly when variables from text data were added to the algorithms - citing for example that the missclassification rate (from 30% to 10%).