How to incorporate Recurrent Neural Networks in your SAS Visual Forecasting pipelines
Recurrent neural networks (RNNs) belong in the family of deep learning algorithms and are widely used in forecasting. They became popular due to their ability to deal with complex patterns, which results to high levels of accuracy that many times outperform conventional forecasting techniques like ARIMA and Exponential Smoothing.
RNNs have been around for a while but integrating them into forecasting processes has always been a challenging task for various reasons.
- They require different data pre-processing from traditional forecasting methods
- They require expert data science knowledge to configure and fine-tune
- Integrating NNs into a forecasting process often requires some intermediate code to ‘glue’ models together and properly analyse.. This is not efficient and could bring inconsistencies in the final forecasts.
Fortunately the Time Series Neural Network Forecasting (TNF) package which was introduced in SAS Visual Forecasting in Viya 4 comes to tackle these challenges. Like all other packages in SAS VF, you can get access to the package functionalities by using ‘proc TSmodel’ that executes user-defined programs on time series data. And just a reminder of what we mean by package: a package is a set of related specialized objects and functions that tackles a unique facet of the time series analysis problem.
The TNF package has made available in a user-friendly and simple manner the three most common types of RNNs which include:
- The original RNN
- The long short-term memory unit network (LSTM)
- The gated recurrent unit network (GRU)
You could implement RNNs using the deep learning cas actions in VDMML in the in previous versions of SAS Viya but the data scientist would need to perform many manual tasks, which are now automated in SAS VF, and of course a VDMML license would be required. Additionally, integrating and comparing your RNN forecasts with traditional forecasting techniques was challenging. Now, in a simple way, we can run RNN methods alongside our traditional forecasting techniques and select the winning model per time series to use as our final forecast. And we can even use the SAS Model Studio UI to explore our winning forecasts even further! We’ll see all this in detail later but let’s take things from the start and let’s discuss data pre-processing.
One significant automation in TNF package is the simplicity of configuration and the automatic data arrangement for time series. Let’s give an example following SAS documentation to understand this a bit better. Suppose that you have a time series with 10 points and you want to forecast 3 points in the future while keeping aside 2 points of your time series for validation purposes. Then your data would look like the image below, where y1 - y10 are your training and validation data and y11 - y13 are the data you want to forecast:
In order to train your RNN on this series the data has to be restructured of course. This will depend on a parameter called ‘window size’. Fortunately by just setting a number to this parameter your data in VF is automatically structured like the image below (example using ‘window size = 3’)
What is important to see from the image above is that for calculating the multistep-ahead forecasts we use the previous forecast values if the actual values are not available; in our case this applies for y12 and y13. Fortunately, VF will automatically do that for you.
And that’s not all. As TNF is a package of ‘proc TSModel’ in SAS VF, you can easily perform data aggregation and accumulation of transactional data inside the same procedure, while also applying ‘by’ variables to perform all these transformations by the hierarchy level of choice. In other words, all the pre-processing steps needed for the RNN methods runs for all series that you would like to forecast in a simple and consistent way.
As mentioned above, one of the great benefits of the TNF package is that you can run your deep learning models inside Model Studio, which is the UI of Visual Forecasting. In that way you can:
- Directly integrate RNNs in your main forecasting process
- Instantly compare the performance of auto-forecasting methods from the ATSM package (ARIMA, ESM, UCM, IDM)
- Automatically select the best forecast per series in a consistent way
- Take advantage of the ‘Forecasting View’ in SAS VF to further explore and analyse your results
- Package your custom RNN-configured node and make it available to be re-used by other data science and forecasting teams to tackle different business problems
How you’ll get there though? Let’s break it in some simple steps.
- The most important thing you should focus is to understand how to structure your code in VF. Start with simple examples of the automatic time series modeling (ATSM) package as we’ll need it later. The TNF package, which includes RNNs, uses the same logic plus a simple way to configure your Neural Networks. If you haven’t written code in VF before, a great article to help you with the logic can be found here.
- The next step is to have a good understanding of the VF pipelines. This should be the easiest step as VF UI is simple and self-explanatory.
- Finally, you will need to understand how to modify an auto-forecasting VF node by combining the TNF package with the ATSM package, using the ‘external model specification’ (EXMSPEC) object.
The EXMSPEC is a key element of our process. It generates an external model (EXM) specification for use with the automatic time series modeling (ATSM) package. So the sequence of steps in our code is prety simple: we develop the RNNs, we use EXMSPEC to treat them as external forecasts and we finally use ATSM which will run a models’ competition between traditional forecasting techniques and RNNs for all our time series and select the best one to use based on a metric of choice.
So let’s say we have already developed a simple pipeline like the one below:
We use 2 auto-forecasting nodes and we amend the code of one of them via the ‘Code Editor’ button to include RNNs via the TNF package. In this way we will be able to compare our custom node which includes a combination of an auto-forecasting techniques with neural networks included, to the output of an original auto-forecasting node.
So let’s see some parts of the code that you need to define for this to work:
1. First let’s see the output array. This will include information about forecasts, errors and confidence limits to be passed on as external forecasts via EXMSPEC to the ATSM to be included in the models’ competition.
2. Secondly, we have to define the RNN model, here we developed an LSTM network and used Adam as the optimizer. Then we see how we collect the results into an array and pass them on to EXMSPEC to be treated as external forecasts later on in our code.
4. If we run our custom node and click on the ‘Results’, we can see that the LSTM forecasts are included as external forecasts and compared directly with the automatic models which could include ARIMA, ESM, UCM, IDM. The models that perform best per time series are selected as champion models.
5. By examining the ‘Model Comparison’ node, we can compare the results we got from our custom node to the results of an original auto-forecasting node. As we see in the image below, the manual node we developed outperformed the original node, decreasing the original WMAPE by ~10%.
Another interesting thing we could do is to explore our LSTM forecasts in a graphical way using the Forecast Viewer of SAS VF. On the graph below we see that we can select the category rExmSpec which includes our LSTM forecasts. We see that 196 series have been selected where LSTMs have outperformed the traditional methods.
By selecting any of the time series from the pane on the right we can examine and analyse individual time series forecasts as below.
Staying in the ‘Forecast Viewer’, we can see that SAS automatically segments our series based on demand patterns so we can take these 196 LSTMs forecasts and analyse them even further. From the image below we see that in most of the cases that LSTMs outperformed traditional techniques, we were dealing with low volume series. This confirms our hypothesis that RNNs can add value in our forecasting process especially when we are dealing with complex patterns.
Finally, to make this process reproducable for different datasets, we used VF macros for the different parameters (e.g. &vf_depVar. for referencing the target variable) so the node can be saved to ‘The Exchange’, which is a repository for custom nodes, and get reused for different use-cases.
In case you want to take this project a step further, you could create your own manual interface of the RNN node where users could select parameters via a point-and-click interface. Details on how to do that can be found here where it’s this process is showcased for developed a custom gradient boosting model for forecasting purposes.
In case we want to further optimise our RNN in our example, we should go back to the ‘Code Editor’ and change the settings of our Neural Network from there. Nonetheless, for the default setting we used and without any fine-tuning of the LSTMs we got some impressive results and added significant benefit in our forecasting process.
So… Happy forecasting!
The full code that has been used can be found at the end of this article.
References
External Models - Forecasting Process Details
SAS Visual Forecasting User Guide
Code used to incorporate RNNs in Auto-forecasting node
/*-------------------------------------------------------------------
* Forecast at a particular level
*-----------------------------------------------------------------*/
*request ODS output outInfo and name it as sforecast_outInformation;
ods output OutInfo = sforecast_outInformation;
*run TSMODEL to generate forecasts;
proc tsmodel data = &vf_libIn.."&vf_inData"n
logcontrol = (error = keep warning = keep)
/*outSum = &outsum
outlog = &outlog*/
%if "&vf_inEventObj" ne "" %then %do;
inobj = (&vf_inEventObj)
%end;
outobj = (
outfor = &vf_libOut.."&vf_outFor"n (replace=YES)
outstat = &vf_libOut.."&vf_outStat"n (replace=YES)
outSelect = &vf_libOut.."&vf_outSelect"n (replace=YES)
outmodelinfo = &vf_libOut.."&vf_outModelInfo"n (replace=YES)
of = &vf_libOut..rnnfor (replace=YES)
ofs = &vf_libOut..rnnstat (replace=YES)
ofopt =&vf_libOut..ex1outtnfopt (replace=YES)
)
outlog = &vf_libOut.."&vf_outLog"n
outarray = &vf_libOut..outarray
errorstop = YES
lead = &vf_lead.
;
*define time series ID variable and the time interval;
id &vf_timeID interval = &vf_timeIDInterval
setmissing = &vf_setMissing trimid = LEFT;
*define time series and the corresponding accumulation methods;
%vf_varsTSMODEL;
outarray rfor lfor ufor efor stdefor;
*define the by variables if exist;
%if "&vf_byVars" ne "" %then %do;
by &vf_byVars;
%end;
*using the ATSM, TNF and TSM packages;
require tnf;
require atsm;
require tsm;
*starting user script;
submit;
declare object f(TNF);
declare object of(OUTTNF);
declare object ofs(OUTTNFSTAT);
declare object ofopt(OUTTNFOPT);
rc = f.Initialize();
rc = f.setTarget(&vf_depVar.);
rc = f.SetOption(
'RNNTYPE','LSTM',
'NINPUT', 3,
'NLAYER', 4,
'NNEURONH',30,
'NHOLDOUT',3,
'NORMALIZE', 'STD',
'POSTTRAIN', 'YES',
'LEAD', &vf_lead.,
'SEED', 12345
);
rc = f.SetOptimizer(
'ALGORITHM','ADAM',
'LEARNINGRATE', 0.1,
'BETA1', 0.8,
'BETA2', 0.9,
'LEARNINGPOLICY', 'STEP',
'STEPSIZE', 5,
'GAMMA', 0.5,
'STAGNATION', 20,
'MINIBATCHSIZE',8,
'WARMUPEPOCHS',15,
'MAXEPOCHS', 100
);
rc = f.Run();if rc < 0 then stop;
rc = f.GetForecast('FORECAST', rfor);
rc = f.GetForecast('RESIDUAL', efor);
rc = f.GetForecast('STDERR', stdefor);
rc = f.GetForecast('UPPER', ufor);
rc = f.GetForecast('LOWER', lfor);
rc = of.Collect(f);if rc < 0 then stop;
rc = ofs.Collect(f);if rc < 0 then stop;
rc = ofopt.Collect(f);if rc < 0 then stop;
/* Create external model specification for the RNN model */
declare object rExmSpec(EXMSPEC);
rc = rExmSpec.open();
rc = rExmSpec.setOption('ERROR','efor');
rc = rExmSpec.setOption('STDERR','stdefor');
rc = rExmSpec.setOption('LOWER','lfor');
rc = rExmSpec.setOption('UPPER','ufor');
rc = rExmSpec.setOption('PREDICT','rfor');
rc = rExmSpec.close();
declare object dataFrame(tsdf);
declare object diagnose(diagnose);
declare object diagSpec(diagspec);
declare object inselect(selspec);
declare object forecast(foreng);
/*initialize the tsdf object and assign the time series roles: setup dependent and independent variables*/
rc = dataFrame.initialize();
rc = dataFrame.AddSeries(rfor);
rc = dataFrame.addY(&vf_depVar);
*add independent variables to the tsdf object if there is any;
%if "&vf_indepVars" ne "" %then %do;
%vf_addXTSMODEL(dataFrame);
%end;
/*setup up event information*/
%if "&vf_inEventObj" ne "" or "&vf_events" ne "" %then %do;
declare object ev1(event);
rc = ev1.Initialize();
%vf_addEvents(dataFrame, ev1);
%end;
/*open and setup the diagspec object and enable ESM, IDM, UCM and ARIMAX model class for diagnose;*/
/*setup time series diagnose specifications*/
/*open the diagspec object and enable ESM, IDM, UCM, ARIMAX model class for diagnose; */
rc = diagSpec.open();
%if %UPCASE("&_esmInclude") eq "TRUE" %then %do;
rc = diagSpec.setESM('method', 'BEST');
%end;
%if %UPCASE("&_arimaxInclude") eq "TRUE" %then %do;
rc = diagSpec.setARIMAX('identify', 'BOTH');
%end;
%if %UPCASE("&_idmInclude") eq "TRUE" %then %do;
rc = diagSpec.setIDM('intermittent',
&_intermittencySensitivity);
rc = diagSpec.setIDM('METHOD', "&_idmMethod");
%end;
%else %do;
rc = diagSpec.setIDM('intermittent', 10000);
%end;
%if %UPCASE("&_ucmInclude") eq "TRUE" %then %do;
rc = diagSpec.setUCM();
%end;
rc = diagSpec.close();
/*diagnose time series to generate candidate model list*/
/*set the diagnose object using the diagspec object and run the diagnose process; */
rc = diagnose.initialize(dataFrame);
rc = diagnose.setSpec(diagSpec);
%vf_setObjectOptions(instance=diagnose, object=diagnose)
rc = diagnose.Run();
ndiag = diagnose.nmodels();
/*Run model selection and forecast*/
rc = inselect.Open(1);
rc = inselect.AddFrom(rExmSpec);
rc = inselect.close();
/*initialize the foreng object with the dataframe result and run model selecting and generate forecasts;*/
rc = forecast.initialize(dataFrame);
/* Run the following line of code to diagnose SAS models along with RNN models */
rc = forecast.initialize(diagnose);
rc = forecast.AddFrom(inselect);
%vf_setObjectOptions(instance=forecast, object=foreng)
rc = forecast.Run();
/*collect forecast results*/
declare object outFor(outFor);
declare object outStat(outStat);
declare object outSelect(outSelect);
declare object outModelInfo(outModelInfo);
/*collect the forecast and statistic-of-fit from the forgen object run results; */
rc = outFor.collect(forecast);
rc = outStat.collect(forecast);
rc = outSelect.collect(forecast);
rc = outModelInfo.collect(forecast);
endsubmit;
quit;
*generate outinformation CAS table;
data &vf_libOut.."&vf_outInformation"n;
set work.sforecast_outInformation;
run;
Don't miss out on SAS Innovate - Register now for the FREE Livestream!
Can't make it to Vegas? No problem! Watch our general sessions LIVE or on-demand starting April 17th. Hear from SAS execs, best-selling author Adam Grant, Hot Ones host Sean Evans, top tech journalist Kara Swisher, AI expert Cassie Kozyrkov, and the mind-blowing dance crew iLuminate! Plus, get access to over 20 breakout sessions.
