Hi Jade, Forecast Studio supports utilizing the validation and test data set to evaluate the model performance. You can set HOLDOUT (number of periods) or HOLDOUT_PCT (percent of total periods) to use validation data set to diagnose and select models. You can use BACK (number of periods) to use test data set to evaluate true model performance as the test data is not used in any way during the modeling process. In data mining world, the training data is partitioned randomly (or in a fashion that is not time related) to form the training, validation, and test data. Models will be built based on the training data and guarded by the validation data. That's the end of modeling process. However, in forecasting, the data has to be partitioned by time sequences since you are not building a model to forecast any random period in the past. The model is built to forecast into the future periods in sequence. The test data has to be the most recent observations, and then the validation data, and then the training data. You might lose the recency effect (most recent data are typically important for forecasting) if you are holdout out data for validation purpose (or the test data). In Forecast Studio, the validation data set (via the HOLDOUT option) will be used for diagnose to create model candidate list. Then the training + validation data will be used to select the best model from the model candidate list, and generate forecasts. This is the most common practice. However, If the data has long enough of history, you do have the luxury to compare models with the test data (via the BACK option). But i would use the BACK for reporting the expected model performance, and then set the BACK to 0 and generate forecasts using the selected model in order to utilize the latest data.
sorry for the long reply... have a nice weekend
alex
