I've read this thread: https://communities.sas.com/t5/Forecasting-and-Econometrics/Out-of-sample-range-vs-holdout-sample/m-...
and am wondering, what happens if you select both:
Model Select -> Use holdout sample for model selection
Forecast -> Calculate statistics of fit over an out-of-sample range
for the same range?
Would it ever make sense to do so?
Hello Bonnie -
First of all thanks for using SAS Forecast Server.
The purpose of using out-of-sample data is to assess the predictive power of your champion model for a specific series _before_ the actual values arrive. The purpose of using holdout data is to pick the champion model on data which was not used for initial estimation of parameters.
Here is an example - when using both out-of-sample and holdouts:
I'm hoping I'm addressing your question. Otherwise you may have to provide an example of what you are asking for.
Thanks,
Udo
Hello -
Yes, it can make sense to select both out-of-sample and holdout samples, but you cannot define them for the same range (at least this statement is true for SAS Forecast Server).
I'm attaching some slides which I presented at SAS Global Forum 2009 - they may be useful understanding the 2 concepts and why you cannot define them for the same range.
Thanks,
Udo
Thanks for your response Udo. To clarify, I am using SAS forecast server exclusively. I do understand the difference between the 2 concepts - using the holdout sample affects the model generation process, while the out of sample process does not. When using only the holdout sample, the statistics reported are for the holdout sample and the entire sample. When using only out of sample process, the statistics reported are for the in-sample and out of sample. But, when you use both the holdout sample and out-of-sample for the same range, the statistics reported are for the holdout sample and out-of-sample I'm not sure what this means - I would think the holdout sample and out-of-sample would be the same if they are defined for the same range, as they are in this case. But the value of the statistics is different, which leads me to believe that you cannot specify both the holdout sample and out-of-sample for the same range. However, I still do not understand why they cannot be defined over the same range in SAS forecast server.
I really appreciate any clarification you can provide!
Thanks,
Bonnie
Hello Bonnie -
First of all thanks for using SAS Forecast Server.
The purpose of using out-of-sample data is to assess the predictive power of your champion model for a specific series _before_ the actual values arrive. The purpose of using holdout data is to pick the champion model on data which was not used for initial estimation of parameters.
Here is an example - when using both out-of-sample and holdouts:
I'm hoping I'm addressing your question. Otherwise you may have to provide an example of what you are asking for.
Thanks,
Udo
Thank you Udo! That answers my question exactly and was a very lucid example. Thanks again!
Registration is now open for SAS Innovate 2025 , our biggest and most exciting global event of the year! Join us in Orlando, FL, May 6-9.
Sign up by Dec. 31 to get the 2024 rate of just $495.
Register now!
Learn how to run multiple linear regression models with and without interactions, presented by SAS user Alex Chaplin.
Find more tutorials on the SAS Users YouTube channel.