Watch this Ask the Expert session to learn more about how to optimize accuracy and computational efficiency in your forecasting process, using time-series segmentation and machine learning techniques.
Watch the Webinar
You will learn:
- The difference between time-series and machine learning techniques for forecasting.
- The time-series segmentation techniques available with SAS Visual Forecasting.
- How to apply appropriate modeling strategies to different segments.
- Which segments could benefit the most from machine learning techniques.
- Tips and tricks to further optimize your forecasting process in SAS Visual Forecasting.
The questions from the Q&A segment held at the end of the webinar are listed below and the slides from the webinar are attached.
Q&A
How do I understand and explain what increase in hidden layers means / does?
Increasing hidden layers will make your model more complex. It will take more time to run if you have auto tuning capabilities available, like the one that we have in Panel Series Neural Network node. That was also the winning method of our experiments. You don't need to worry about tuning it yourself like you’ll have to do when it comes to the other neural network-based strategies like the Stacked node that we used. By increasing the complexity this specific time, we managed to increase the accuracy of the results. I wouldn't select more than 3 hidden layers, because then the complication explodes. But experimenting with 1 and up to 3 layers is normally a good rule of thumb to get reliable results. Keep in mind that increasing the number of layers won't always guarantee a more accurate model. Our recommendation is to experiment with the different settings and see if your accuracy is increased in the holdout sample.
If we wanted to have a bit more control of a node, is it possible to use code within a node using SAS model Studio or do we have to switch between the Developer SAS Studio?
There is a lot of control in SAS Visual Forecasting. You can see in some pipelines that the nodes we use have the ‘pluggable’ name before them. There you can open the node, view the code, and modify it or put specific parameters in it, and see what you get – the same applies for the Auto-Forecasting node. Also, we have recently introduced the Distributed Open Source Code node where you can simply and quite natively incorporate your open source code and algorithms that you would like to use. Again, you can put it next to the different nodes inside SAS Visual Forecasting, see the results side by side, and select the best one for your forecasting problem. So, you get unlimited control there. Nothing should stop you from creating a custom node with many different parameters in there to test in your process. The final thing that I want to mention is the custom nodes we have already created and made available via GitHub. If you Google “Gradient Boosting SAS Visual Forecasting GitHub”, you will find the custom node we have made that incorporates the gradient boosting algorithm in the code. You can view the code and switch only the part of the code where you have the gradient boosting algorithm and incorporate algorithms of your choice, like Random Forest or LightGBM, which are also natively available in SAS. You can amend the code, then rerun the node. and see what you get in the results. If you're happy with a custom node that you have created, you can save it to the Exchange, (just one click of a button really) and make it available to other business units to use it out-of-the-box as well. We also have a lot of material about creating custom nodes in SAS Visual Forecasting. Again, if you Google search our papers, you’ll find a fantastic paper around how we built the Gradient Boosting node so you can also create even the UI for this custom node (if you want to go that far). Remember that in SAS Visual Forecasting you have both choice and control to do things the way you like.
Do you have for the list of model evaluation KPIs (RMSE, MAE, RMSE, ...)? What sort of complimentary QA checks on your forecasts since they each have different focus strengths/weaknesses?
It's a big area of research I would say. Use the ‘Resources’ slide of this presentation if you want a great place to start. I would start with the simple blog around forecasting KPIs. This provides an intuitive guide for MAPE, MAE, RMSE and Bias. Very nice discussion with practical examples of what they mean, pros and cons, and what to use where. Then I would continue with the more thorough article, Measuring Forecasting Accuracy Problems and Recommendation, which provides more details on what should be used and in which situations. In SAS we have probably hundreds of metrics available. You can select the ones to actually view in SAS Visual Forecasting, so you don't get lost with the different metrics. You can do your research and then select the ones that you would like to examine. I always start with RMSE, MAE, and their weighted versions, and then you can also select others for the specific problem you've got in hand. But the list is endless, and we always introduce new ones as they are published in the research space.
The webinar is really amazing and thank you for your valuable learning. How can we learn the SAS programming language effectively?
It depends on the background of the user. I think the best possible way to start is by taking the courses we've got in education. I believe that “SAS Programming 1” is completely free as well, so you can start with something like that and then focus on how you actually perform your day-to-day job duties. That's I believe how you can master anything in life. If you start using it and try to solve real world problems, I think it will become very easy going forward. SAS is a coding language which is tailored for non-coders. So, it makes it very easy to use the different steps effectively as they perform a lot of things together and simplify the process. That said, if you're just starting with SAS Visual Forecasting, you don't need to code. If you don't want to do anything manually or configure the settings in great detail, you can just use the different nodes we have available and the drag-and-drop interface. This is an efficient way to get accurate and reliable results.
Why shall I segment my data when I could use an Ensemble node and always select the best model per series?
We recently introduced the Ensemble node that you can insert in your pipelines. The beauty of this node is that it always selects the best model per series. So why should I segment my data? For two reasons. Firstly, machine learning algorithms provide better results when we are dealing with interrelated data. So, making groups that their time series have similar patterns or similar behavior, will most probably lead to better performing machine learning models as segmentation will filter out the noise. Secondly, it's very computationally intensive to run all the algorithms for all your data all the time. Segmentation ensures a much more efficient solution for this problem and as we said, optimizes your process both in terms of accuracy and computational efficiency.
Are there any types of segmentation for time series I could use in SAS?
Yes, we have many different algorithms available. However, you have to do your segmentation in some other place, if you don't want to use the demand classification pipeline, such as SAS Studio or a Jupyter notebook or whatever you would like to use and bring your external segments inside SAS Visual Forecasting and start applying the different strategies in each of the segments. I want to highlight a task in SAS Studio, called Similarity Analysis, which uses Dynamic Time Warping to calculate the distances of the time series and can be very powerful for segmentation solution. When it comes to time series, I would recommend using this task. This task is also made available via a point and click interface in SAS Studio where you select the similarity analysis and you're putting the parameters of your choice. This will create automatically the code for you so give it a shot. Keep in mind that our tests have shown that the predefined segments from the demand classification pipeline, that we have available right out of the box, give better accuracy in the segments in most cases. That doesn’t mean though that there are not different tools out there that you should test and see what works best for your data.
When using the point and click on the Nodes, is it possible to view the SAS code?
Yes, in most of the nodes this is available. For some nodes, the complexity gets very high so we don't have it available. But these are very few. For example, for some neural network-based strategies, we can't see the actual code and what happens behind the scenes. That's why we have a paper available on this topic. In most case though you would be able to see all the code that's generated inside the node and be able to modify it as well.
Can I integrate GenAI pre-trained forecasting models in SAS?
Of course you can, especially if we're just talking about an API call. However, there are still various challenges when it comes to pretrained forecasting models and we could discuss a bit more on why we believe that they are not a great choice to use yet. SAS though provides an open and extensible framework when it comes to data and AI. So, you can bring them in and see how they work. The challenges that I want to mention about pre-trained forecasting models include mostly the lack of diverse and available domain specific data. When it comes forecasting, compared to LLMs, where you have all the text in the world available to train your data, we only have very limited resources when it comes to data sets. The second problem is data leakage, something that we should be cautious about. Currently you see new pretrained forecasting models jumping up every other day, or week. The actual benchmarks though are a bit debatable and when I refer to data leakage, I mean that the data that the models are trained on, are also used in some way for testing them. We also don't have insight on the data used to train these models. However, you could create a pipeline in SAS Studio to format the data appropriately, send it and get the results of a pre-trained model with an API call. This would probably be very costly based on what we have seen overall. But if you want to experiment with that, you can. One thing that we could potentially see a benefit in the future, is use the outputs of pre trained forecasting models as inputs to data specific models, like the machine learning and the statistical ones, that we discussed today. So, get the output from GenAI and use it as an input (independent variable) to your forecasting pipelines and see if this will improve your existing models and by how much. That will be an interesting experiment to run.
Is there a free trial for SAS Visual Forecasting?
SAS Visual Forecasting is included in the trial version of SAS Viya. It's going to be free for 14 days I believe, and most functionality if not all, is available there. Go there, give it a shot. See how you like it.
Recommended Resources
Forecasting with sas free ebook
Using SAS Viya for Time Series Data Handling
Please see additional resources in the attached slide deck.
Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q&A, slides and recordings from other SAS Ask the Expert webinars.
Click image to register for webinar