I have been working on one Decision Tree on SAS EM. I created Validation / Train dataset for hold-out validation and run decision tree model. I will have two questions, appreciate if you could help.
1. How can I make sure train and validation data look reasonable besides checking GINI, ROC? Any other thoughts I have to look at?
2. Do you think, does it make sense to do out-of-time validation in Decision Tree Model ? i know we should do this type of validation in credit scoring, but not sure if it is worth or appropriate for Decision Tree Models.
I'm not sure what you mean that train and validation data look reasonable. In most modeling scenarios, both the TRAIN and VALIDATION data sets are representative of the same population. Candidate models are typically built on the TRAIN data and then compared on the VALIDATION data. There is no clear answer as to when your results appear to be reasonable, but if the model performs far better on one of these data sets than on the other, you should consider checking some things such as
* were important stratification variables taken into account when the observations were divided between TRAIN and VALIDATE?
* are there sufficient observations for each target level outcome (if building a classification tree) and/or for both the TRAIN and VALIDATE data sets? (too few observations can cause wildly varying results)
Note: These questions are not limited to decision tree models but should be considered anytime partitioning is done.
Regarding out-of-time validation, you are blending together two different ideas -- the performance of the initial model (which will likely be better than when the model is deployed) and the performance on future data. Using future data too validate might yield a model that performs better in more recent data but it also requires you to either build the model using older data (because your training data comes from further in the past) or forces you to wait to decide which model is best. Neither is typically desirable, particularly in situations where the relationships in the data might be constantly changing. In the end, you can evaluate using different approaches and will likely identify best practices around the business problem you are answering if you compare the performance of these strategies. In general, they both have good things and bad things about them, but it is more common to use training and validation data drawn from the same time period in my own experience, but there are certainly others who might prefer the out-of-time approach.
Either way, SAS Enterprise Miner allows you to use either with approach with ease.
Hope this helps!
Doug
