Turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- Home
- /
- Analytics
- /
- Forecasting
- /
- How does forecast server calculate error statistics for Intermittent f...

Options

- RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page

🔒 This topic is **solved** and **locked**.
Need further help from the community? Please
sign in and ask a **new** question.

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

Posted 03-29-2016 01:04 PM
(1051 views)

Hi,

I'm fairly new to SAS. I'm using Forecast Studio 12.1 for a Forecast Value Add analysis of another system's forecasts. I'd like to know how SAS forecasts the MAE for intermittent demand forecasts. For example, the Holdout MAE for one of my forecasts was 0, the forecast was 1 for all four cycles, and the holdout values were 0 2 0 2. By my calculation this means the MAE is 4, but the 0 is a better representation of how well the forecast is doing. What is the exact calculation for MAE that SAS is using? I'd like to use the same calculation for my other system. I think it is grouping together multiple weeks of demand into one "demand qty" for the forecast accuracy, but I'm not sure how it decides which weeks to group. Is it based on a forecast "demand frequency" or on the actual next demand?

I've attached the SAS forecast studio guide definition of MAE.

1 ACCEPTED SOLUTION

Accepted Solutions

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

The forecasts showing in MAE2 image are not the forecasts used to compute holdout MAE. Holdout samples are used for model selection. To be specific, all candidate models are first built based on the data excluding the holdout samples.Forecasts are then be generated for the holdout samples to compute the holdout MAE. The model with the smallest MAE will be selected as the winning model. The winning model is then re-estimated using the entire dataset (assuming BACK option is disabled). Of course you can pick a different forecast error measurement for model selection. Hope this helps.

Alex

4 REPLIES 4

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

Hello -

From SAS Forecast Server Procedures 14.1: User's Guide ("Forecasting Process Details"):

"Time-Indexed versus Demand-Indexed Holdout Samples

Holdout samples are typically specified based on the time index. But for intermittent demand model selection, demand-indexed-based holdouts are used for model selection. For example, consider “holdout the last six months data.” For a demand series, the demand indexed holdout refers to the “demands that have occurred in the last six months.” If there are four demands in the last six months, the demand indexed holdout is four for a time-indexed holdout of six. If there are no demands in the time-indexed holdout, the demand-indexed holdout is zero and in-sample analysis is used."

You will find additional details in the "Intermittent Models" chapter.

Thanks,

Udo

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

Hi Udo. Thank you for your answer. Unfortunately, I'm still not getting the same numbers when I calculate it myself. Here is an example.

Say I have a 4 week holdout for my non-intermittent demand. So I'm looking at the demand in that period.

During these 4 weeks I have 1 nonzero demand (2 units) on the third week and my forecast is 0.9226 for the previous 3 weeks (see MAE2). So I would say abs(3*.9226-2)/1=0.76 is the MAE. However SAS says that the Holdout MAE is 0.006299.

I'm attaching screenshots of the MAE (MAE1), the forecast/history chart (MAE2), and the Holdout Size (MAE3). Can you tell me where I'm going wrong here? Thank you.

Edit: I tried to calculate the MAE for a few of the non-intermittent models as well and came up with different numbers. So I'd appreciate any help you can give me understanding what I'm doing wrong. I also tried searching the forecast studio guide for the word intermittent, but there was no information about how error is calculated. There is also no information on MAE beyond the formula: 1/n sum(abs(dmd-fcst))

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

The forecasts showing in MAE2 image are not the forecasts used to compute holdout MAE. Holdout samples are used for model selection. To be specific, all candidate models are first built based on the data excluding the holdout samples.Forecasts are then be generated for the holdout samples to compute the holdout MAE. The model with the smallest MAE will be selected as the winning model. The winning model is then re-estimated using the entire dataset (assuming BACK option is disabled). Of course you can pick a different forecast error measurement for model selection. Hope this helps.

Alex

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

Yes. That makes much more sense now. Thank you!

Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.

**If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website. **

Multiple Linear Regression in SAS

Learn how to run multiple linear regression models with and without interactions, presented by SAS user Alex Chaplin.

Find more tutorials on the SAS Users YouTube channel.