BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
Jade_SAS
Pyrite | Level 9

Hi All,

 

    I have a general question about forecasting, how can we know the forecasting model is good enough for your data? Usually what parameter in forecast studio result we should look at? Thank you!

 

Thanks,

Jade

1 ACCEPTED SOLUTION

Accepted Solutions
Puwang
Obsidian | Level 7

I always like to set a holdout period to validate the model. In forecast studio, you can set a holdout period in Project->Forecasting Settings->Forecast->Calculate statistics of fit over an out-of-sample range. See attached picture, where I set the length of holdout period to be 6.

 

In Modeling view, there are several useful tables and plots that you can view to help determine whether it is a good model or not.

For example, you can compare summary statistics of fit between in-sample period and holdout period. If holdout MAPE is much worse(bigger) than in-sample MAPE, you may want to check if there's any overfit in your model.

You can also check the Unbiasedness Test table, or view Prediction Error Histogram to see if there's any systematic bias in error.

You may also want to check data points that have extreme errors, to check for outliers, data error, or uncaptured features.

Hope this helps.

Set holdoutSet holdoutModeling ViewModeling View

View solution in original post

1 REPLY 1
Puwang
Obsidian | Level 7

I always like to set a holdout period to validate the model. In forecast studio, you can set a holdout period in Project->Forecasting Settings->Forecast->Calculate statistics of fit over an out-of-sample range. See attached picture, where I set the length of holdout period to be 6.

 

In Modeling view, there are several useful tables and plots that you can view to help determine whether it is a good model or not.

For example, you can compare summary statistics of fit between in-sample period and holdout period. If holdout MAPE is much worse(bigger) than in-sample MAPE, you may want to check if there's any overfit in your model.

You can also check the Unbiasedness Test table, or view Prediction Error Histogram to see if there's any systematic bias in error.

You may also want to check data points that have extreme errors, to check for outliers, data error, or uncaptured features.

Hope this helps.

Set holdoutSet holdoutModeling ViewModeling View

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

Multiple Linear Regression in SAS

Learn how to run multiple linear regression models with and without interactions, presented by SAS user Alex Chaplin.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 1 reply
  • 750 views
  • 1 like
  • 2 in conversation