Understanding SAS Model Studio Forecast Task Options in SAS Visual Forecasting
In SAS Visual Forecasting using SAS Model Studio you have the opportunity to choose different forecasting task options for hierarchical forecasting. You might do this if you regularly add recent data to your historic data and you want to:
• Reduce processing time
• Avoid dramatic swings in results due to short-term changes
This post explains what SAS Visual Forecasting does when you choose:
1. DIAGNOSE
2. SELECT
3. FIT
4. UPDATE
5. FORECAST
Background
For each individual time series in the hierarchy, SAS Visual Forecasting:
• Diagnoses the time series using time series analysis techniques
• Creates a list of candidate model specifications based on the diagnostics
• Fits each candidate model specification to the time series
• Generates forecasts for each candidate fitted model
• Holds a model tournament and selects the most appropriate model specification based on either in-sample or holdout sample evaluation using model selection criterion specified by the user
• Estimates parameters to fit the selected model specification to the entire historical range of the time series
• Generates forecasts from the fitted model
• Evaluates the forecast using in-sample analysis and provides for out-of-sample analysis of forecast performance
But here’s a little secret. You do not need Visual Forecasting to repeat all of this every time you update your data! SAS Model Studio lets you save processing time by only running a subset. For example, if you have already run the DIAGNOSE forecasting task option, you could run just the FIT forecasting task option. The following table indicates what SAS Visual Forecasting will do based on the forecast task option that you choose.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Remember that you must first run the full DIAGNOSE forecasting task option. That is set by default and will be chosen if you don’t make any changes in SAS Model Studio. If, however, you do not want to use DIAGNOSE, select your Hierarchical Forecasting node. In the options pane, use the dropdown to choose from one of the other four forecasting task option settings, as shown in the screen capture below.
So what exactly does each of these forecasting task options do?
1. DIAGNOSE
DIAGNOSE will complete everything, A to Z, including the initial diagnostics.
2. SELECT
SELECT will reselect a champion model from your candidate models. It will then estimate parameters of this newly selected model, and produce new forecasts.
Model selection can be in-sample or can use a holdout sample (honest assessment). In the options, you can select the number of data points to use in the holdout sample and the model selection criteria, as shown in the screen sample below.
You get to choose from a over 50 model selection criteria (shown in the screen capture below), including the Akaike Information Criterion, Mean Absolute Percent Error, and Root Mean Square Error.
You cannot run the SELECT task option unless you have already run the DIAGNOSE task option successfully.
3. FIT
FIT will reestimate the champion model parameters from scratch and then reforecasts. It does not evaluate nor select new models. You would use this if you believe that your champion model is the best model, but you want to update the parameters to fine tune it.
You must run DIAGNOSE successfully before running FIT.
4. UPDATE
UPDATE estimates parameters for the individual models that you choose and then reforecasts them. Again, no new models will be evaluated nor selected. UPDATE is very similar to FIT. It differs in that for UPDATE the estimated parameters are used as starting values in the estimation
And again, you cannot run UPDATE until you have successfully run DIAGNOSE.
5. FORECAST FORECAST forecasts the time series using the model parameters estimates that were already created. It uses the existing champion model to create the forecast and does NOT update the model parameters.
Again, you need to run DIAGNOSE successfully before running FORECAST.
6. Which SAS Model Studio Forecasting Nodes Let Users Specify the Forecast Task
The Hierachical Forecasting node is not the only node that let’s you take a short cut past diagnose. The three nodes that let you set the forecast task options in SAS Model Studio are:
• Hierarchical Forecasting
o all forecast task options
• Hierachical Forecasting (Pluggable)
o all forecast task options
• Panel Series Neural Network
o all forecast task options except Select
The following forecasting nodes in SAS Model Studio do not let the user set forecast task options to avoid rediagnosing.
• Naïve Model
• Non-seasonal Model
• Multistage Model
• Regression for Time Series
• Retired Series
• Seasonal Model
• Stacked Model (NN + TS)
• Temporal Aggregation Model
7. When To Rediagnose?
When do you need to rediagnose? The answer is, of course, it depends.
If you have a long and relatively stable time series data, you may not need to rediagnose very often to get useful results. If you have an erratic and volatile time series, you MAY want to diagnose more often. However, be careful not to rediagnose too often with volatile data. From a pragmatic business perspective, you don’t want to rediagnose too frequently if it results in rapidly changing results and unfeasably inconsistent business decisions. Credibility of the results and process and software can also suffer in the eyes of many business practitioners when they see rapid changes in results.
But when your model degrades and is no longer accurate and useful, you need to rediagnose. But how to determine if your model is degrading? First of all you can rely on your longtime expertise and that of your fellow statisticians, data scientists, forecasters and domain expert. But what if you have hundreds or thousands of models and limited time to lay human eyes on each one?
There is a clever trick that you can use! SAS’s own Joe Katz developed a patented innovative way to use control charts to monitor forecast model errors to let you know if they are degrading. See his article in Foresight or view his patent number 9,208,209 to learn this method.
8. For More Information
• SAS Visual Forecasting Documentation
• Katz, J. H. (2020), Monitoring Forecast Models Using Control Charts, Foresight (Winter 2020), 20-25.
• Katz, J. H. (2015). United States of America Patent No. 9,208,209.
Find more articles from SAS Global Enablement and Learning here.
