To properly evaluate (and improve) forecasting performance, we recommend our customers use a methodology called Forecast Value Added (FVA) analysis. FVA lets you identify forecasting process waste (activities that are failing to improve the forecast, or are even making it worse). The objective is to help the organization generate forecasts that are as accurate as can reasonably be expected (given the nature of what they are forecasting), and do this as efficiently as possible (using the fewest resources).
In its simplest form, FVA compares the accuracy of a Naive forecast to the organization’s current forecasting method (usually some form of statistical forecasting with manual adjustments, or even an entirely manual process). FVA can also compare performance to alternative methods, like an automated statistical forecast (which can be generated from software such as SAS® Forecast Server or SAS® Forecasting for Desktop).
In the latter two cases, there is often the option to make manual adjustments to the computer generated forecast.
The accuracy of the Naïve forecast serves as a basis for comparison against all other forecasting activities. The Naïve forecast can be created automatically at virtually no cost, so it is important for the organization to understand what accuracy the Naïve forecast can be expected to achieve. If the Naïve’s accuracy is “good enough” for the organization’s planning and decision making purposes, then it makes sense to just use it, and stop doing any manual or statistical forecasting efforts. (Why spend time and money on a costly forecasting process if the Naïve can generate “good enough” forecasts for free?)
In most situations, however, the organization seeks forecasts that are more accurate than what the Naïve can achieve. But a typical forecasting process, even when aided by statistical forecasting software, can consume a lot of management time, at considerable cost.
The FVA approach lets you focus your improvement efforts in areas most in need of improvement – such as those where your accuracy is worse than the Naïve forecast.
Many companies find that overall, their forecasting process is performing better than just using the Naïve forecast. However, there are usually specific areas (products, locations, or other points in the organization’s forecasting hierarchy) where the Naïve forecast performs better. These should be investigated, to see if there are explainable reasons why the forecast is worse than the Naïve, and whether it can be improved. (Often the cause of such poor forecasting is political pressures within the organization. The forecast represents what management wants to happen, rather than being an unbiased best guess of what really will happen.)
Upon doing FVA analysis, a surprisingly large number of companies find that overall they are forecasting worse than doing nothing and just using the Naïve forecast! If the forecasting process cannot be improved in these areas, then it is simply wasted effort, and should be eliminated in favor of using the Naïve.
Organizations often find that, overall, Automated forecasts are more accurate than what their current forecasting process produces. However, there will likely be some areas where the Automated forecast is less accurate than the existing process, or even less accurate than the Naïve forecast. Once these areas are identified, they can be investigated. (It is important to note that for some sales patterns, the naïve “no change” model is the most appropriate forecasting model, and cannot be meaningfully improved upon.)
To make it easy to identify non-value adding areas, you can build a simple application using SAS® Visual Analytics software. Such an application lets you point and click your way through the organization’s forecasting hierarchy, and at each point view performance of the Naïve, Manual, Statistical, and Automated forecasts (or whatever different types of forecasting methods you are using). This makes it very quick and easy to identify where forecasting methods are not adding value, so these areas can be investigated. SAS Visual Analytics provides a much better environment for doing this kind of analysis than trying to do it in Excel.
The value of an automated (or largely automated) forecasting process is threefold:
In large-scale forecasting situations, automated forecasting is not just an option, it may be a necessity. It is not uncommon for a retailer, having thousands of items sold at hundreds of stores, to have over a million store/item forecasts they want to create. But no company, in any industry, can afford the army of analysts needed to manually build a million+ forecasting models.
When you combine FVA analysis with the large-scale automated forecasting capabilities in SAS Forecast Server (and larger-scale capabilities when you also utilize SAS® Grid Manager), even the largest enterprise can efficiently generate quality forecasts, and focus analyst efforts where they are most needed.