BethEbersole Tracker

SAS Visual Investigator: It's Not JUST for Fraud

BethEbersole — Fri, 26 Apr 2024 15:04:36 GMT

Recently in the SAS Community Library: SAS' @BethEbersole reveals 4 steps to stop money laundering, solve law-enforcement cases, find missing children and more with SAS Visual Investigator.

Tips and Tricks: Improve Forecast Accuracy: Q&A, Slides, and On-Demand Recording

BethEbersole — Fri, 08 Mar 2024 14:28:43 GMT

Tips and Tricks: Improve Forecast Accuracy Using Interactive Modeling in SAS® Visual Forecasting

Q&A, Slides, and On-Demand Recording

Watch this Ask the Expert session to learn how to improve your forecasting results by adjusting model parameters for individual forecasts in the easy-to-use user interface of SAS Visual Forecasting.

Watch the webinar

You will learn how to use the Interactive Modeling node in SAS Visual Forecasting to:

Tweak individual forecasts to improve accuracy.
Build a model from scratch.
Manually select a champion model.

The questions from the Q&A segment held at the end of the webinar are listed below and the slides from the webinar are attached.

Q&A

How do you know which is the best Model? What factors should I look at?

The best model is going to depend on your domain. We used MAPE, which is commonly used in economics. It's mean absolute percent error. It's very easy to interpret. That's how a lot of economists are going to determine the best model, also keeping in mind simplicity. In your field, you might also consider things like the Akaike Information criterion, Schwartz Bayesian criterion, which is going to basically reward you for a less complex model. In biological science, you might be using the root mean square error. So, this is going to be based entirely on what is your field and you could look at what people have published and what criteria they are using. If you have more detailed questions about your specific situation, you can ping me after the webinar or anytime East Coast time.

Can you see the code generated by the pipeline?

You can download the code for the model, but I can't quickly show that because it is a little bit involved. See my SAS Communities blog at https://communities.sas.com/t5/SAS-Communities-Library/Downloading-Models-from-SAS-Visual-Forecasting/ta-p/831599.

How do we know if it is overfitting or not?

I'm a huge fan of a holdout sample and, frankly, I do it outside of even SAS Visual Forecasting. I like to hold out data completely outside of this process and then try the model I came up with and see if it will generalize to the new data. You can only do that if you have a long enough time period, right? We don't always have that luxury, but in this case, as you can see here, I have 13 years, which is a pretty long time period.

I could hold out the last three years and then even add a holdout within the software. It's kind of comparable to predictive modeling with visual machine learning. You can have a test as well as a validation data set. That’s what I would do if you had the luxury of a long time period. If you don't have the luxury, you could sub sample, try that sub sample set out, and then make sure that it will generalize to the data you pulled out. You could subsample over the whole time period.

Below is a screen shot of how to set the holdout within the auto-forecasting node.

Why doesn’t SAS VF always give me the best results by default?

First let me say that SAS Visual Forecasting is a huge time saver and will give you many excellent and highly accurate models automatically if you have decent time series data for a reasonable length of time. But not every series in your data set might get a great result automatically. I like to use the analogy of when you're having a served dinner buffet at a wedding or a conference and you have three choices - fish, crab cakes or beef tenderloin. They also get to choose their dessert, right? But then you get my cousin Eva, and she's vegan and gluten free. She's going to need a special meal, right? So just like that, some of your data sets are not going to be able to get a good forecast easily. There's something kludgy about them sometimes. You want to put eyes on them, you want to look at the graph, you want to see if maybe there were some errors. One of the things that we commonly dealt with when I was looking at Chesapeake Bay data would be lab changes. We would see a sudden drop or a sudden increase in, for example, nitrates. But it was an artifact of the data. Anytime your automatic forecasting isn't giving a good forecast, you want to dive in with your own eyes and see what's going on. Because it could be some problem with the data, but it could be that you could get a slightly better model if you tweak the model around. I consider these the problem children models that they just don't easily by default give you a good model. And in some cases, actually you're getting the best model that you can get with that data. That's what I tried to show with the petroleum Louisiana naive model, which is a random walk. It might be the best you're going to get with that data set.

Is there a way to apply customization to the model at a higher level of hierarchy? For instance, to all "coal."

Not currently in the Interactive Modeling node. The Interactive Modeling node lets you work with individual time series only. But this is on the road map, and we hope to see it soon.

How do you select the historical data for forecasting?

I'm not sure what you mean. You want the longest time period that you have. I would always start with the longest time period that you have and would only truncate that if, when you look at it, there's big changes. Depending on the field you're in, it may be more and more volatile. If we're talking stock market prices, you may want to truncate it. But you're going to look at the whole time period first and then you may decide you're going to truncate it, if more recent is really more relevant. If you're looking at something like water temperatures, you're going to probably want to use the whole time period. There could be a long-term trend in there that you want to capture. So, it's going to depend on your data, but I would always start with the full time before I started truncating.

Is there a way to choose the domain or field in SAS to get the best forecasting models?

No, that's why, for SAS Visual Forecasting, you need someone who is knowledgeable. You don't need 30,000 statisticians or 20 statisticians. But you need one, at least, forecasting expert to use SAS Visual Forecasting so they would then know the field. They would then know the domain. If you want to go back to an even simpler tool, you could use SAS Visual Analytics. It is for anybody. You don't need anyone with any expertise to use SAS Visual Analytics. That will by default give you the best and would maybe lead you into hiring a forecaster and using SAS Visual Forecasting. It won't default for your domain. You would need someone on your team who knows that.

I missed the beginning of the webinar, are there any "packages/modules," for lack of a better term, that are being used beyond Viya and visual analytics that need to be purchased to do these forecasts?

Everything I showed you is part of SAS Visual Forecasting. You don't have to buy anything extra on top of SAS Visual Forecasting. And this is a Model Studio interface, which is part of SAS Visual Forecasting. So, you get the programming, the Model Studio interface, you get all of that.

Is there a sample size limit?

If you are asking about a) the data size limit, SAS Viya software products are designed to work with extremely large data sets and run the analytics efficiently in parallel, but you will also need to consider a practical/feasible limit based on your compute resources and configuration. If you are instead asking about b) how much data to include in your holdout sample or out of sample region, this would depend on how long your time series data set is, and whether you have seasonal or other cycles in the data. For example, in my data set we had seasonal data by month for 13 years. You could set the holdout region by setting an integer (for example, 12 or 24 months) or by setting a percentage (for example, 10%). With cyclic or seasonal data, you would want the holdout data to encompass at least one full rotation of the seasons. For example, if you have monthly data that is seasonal, you will want to holdout at least one year (12 months). See this link for more information on holdout samples and out-of-sample region in SAS Visual Forecasting. https://go.documentation.sas.com/doc/en/vfcdc/default/vfug/n0mf6wi57q8huin1mroj229k4tur.htm#n09h96b3jv4cecn13vmkibwyx6ro

Is there an ARCH or GARCH option in automatic forecasting?

Although SAS Visual Forecasting auto-forecasting node can consider many types of models (exponential smoothing models, ARIMA and ARIMAX models, intermittent demand models, unobserved component models) it does not consider ARCH or GARCH models.

How would you add an ARCH or GARCH model to the pipeline?

ARCH and GARCH models are supported via SAS/ETS procedures; SAS/ETS is included with SAS Visual Forecasting. These procedures require coding. PROC AUTOREG and PROC VARMAX are the two SAS/ETS procedures you can use to create ARCH and GARCH models. It may be possible to add these via a SAS code node into the pipeline in the future. I will continue to investigate this.

Recommended Resources

SAS Demo: Use SAS Visual Forecasting’s Interactive Modeling Node to Hone in On Accuracy by Beth Ebersole

Interactive Modeling in SAS Visual Forecasting by Joe Katz

Please see additional resources in the attached slide deck.

Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q&A, slides and recordings from other SAS Ask the Expert webinars.

How to Interpret AI and Machine Learning Models Using Shapley Values in SAS Viya

BethEbersole — Fri, 09 Feb 2024 16:30:53 GMT

How to Interpret AI and Machine Learning Models Using Shapley Values in SAS Viya

Deep Dive on the mitigateBias Action: Hyperparameter Tuning

BethEbersole — Mon, 20 Nov 2023 20:42:32 GMT

Deep Dive on the mitigateBias Action: Hyperparameter Tuning

Re: Use SAS to Quickly Simulate and Graph Data from Different Distributions

BethEbersole — Fri, 27 Oct 2023 18:35:14 GMT

Thank you to Rick Wicklin for the correction to the negative binomial pdf code.

Use SAS to Quickly Simulate and Graph Data from Different Distributions

BethEbersole — Fri, 27 Oct 2023 18:34:02 GMT

Use SAS to Quickly Simulate and Graph Data from Different Distributions

How to Use Generalized Additive Models in SAS Viya

BethEbersole — Wed, 20 Sep 2023 15:55:11 GMT

How to Use Generalized Additive Models in SAS Viya

Generative AI and Large Language Models Demystified

BethEbersole — Wed, 09 Aug 2023 00:29:20 GMT

Generative AI and Large Language Models Demystified

Using Recurrent Neural Networks in SAS Visual Forecasting

BethEbersole — Tue, 18 Jul 2023 15:19:00 GMT

Using Recurrent Neural Networks in SAS Visual Forecasting

Recurrent Neural Networks in SAS Viya

BethEbersole — Wed, 24 May 2023 13:41:00 GMT

Recurrent Neural Networks in SAS Viya

Mitigating Bias Using SAS Viya Fair AI Tools

BethEbersole — Fri, 21 Apr 2023 15:13:01 GMT

Mitigating Bias Using SAS Viya Fair AI Tools

AI Ethics Part 2: Trustworthy AI at SAS

BethEbersole — Thu, 16 Mar 2023 21:28:53 GMT

AI Ethics Part 2: Trustworthy AI at SAS

AI and Ethics Part 1: Bots

BethEbersole — Thu, 16 Mar 2023 22:42:45 GMT

AI and Ethics Part 1: Bots

Re: Improve Your Nowcasting in SAS Visual Forecasting Using Real Time Data (by Tammy Jackson)

BethEbersole — Thu, 15 Dec 2022 19:11:15 GMT

I’ve received some comments/discussion offline, so I’d like to add them here. One question was: “What’s the benefit of doing everything (even data pulling) with EXTLANG? Wouldn’t it be easier to download the external datasets using a proc python step in sas studio and then put it in a flow that has the modeling part with proc tsmodel as a second step?”
Response1 (from Tammy Jackson): “For nowcasting, we do want to pull the data in as close to the forecast as possible. For instance, if the user is using search terms as a proxy, then the user might want to look at searches early in the day to schedule for the current day. Certainly for something like the stock market, there are external influences that can cause sudden fluctuations. I think there is an advantage to tying it to the forecast. The other advantage to tying it to the forecast is not having to manage and store the data externally. The most important idea is to push one button. Certainly that can be done if everything is in a single SAS job.”
Response2 (from Tammy Jackson to different forum): “Since the idea of nowcasting is to be able to get a short-term forecast with the most current data to make a decision “now”, I think whatever you do needs to be single step, such as click a button or run a SAS job.
Obviously, either method can be set up to do that. The differences might be:
1) time – how long is the lag between the data collection and the forecast
2) Complexity – if the data is stored externally, then you have to follow the code path to see the connection between the inputs and the data collection
3) Data management – using inline python you don’t have to store the input data.
4) Run time – If the same inputs are being used for many series, you might want to collect the data once for all – in which case, you might want to store the data externally.”
Response 3 (from Javier Delgado): “One point to consider, vis a vis using a PROC PYTHON step versus EXTLANG, and data management, is that PROC PYTHON will download the data to the client whereas EXTLANG will download it to CAS. If client and CAS share a file system, then a PROC PYTHON step may be best if processing multiple BY groups, since each BY group is running the code that downloads the data. If they do not share a file system, then downloading the data on EXTLANG will skip the step of uploading from the client to CAS. If (they don’t share a file system and) you have multiple BY groups you still have the issue of not wanting to download the data multiple times. You can solve this by adding some logic to your code to only download when/where necessary; this adds a bit of complexity and computation, but alas there’s no free lunch. Or free beer in this case."
Response 4 (from Tammy Jackson): "You could use EXTLANG in a separate TSMODEL run to keep the data in CAS, but do a single download that would be available to all bygroups as an AUXDATA file. You still have “external” data management from the TSMODEL side. But you don’t have to cross the CAS boundary."

Improve Your Nowcasting in SAS Visual Forecasting Using Real Time Data (by Tammy Jackson)

BethEbersole — Wed, 14 Dec 2022 17:16:47 GMT

Improve Your Nowcasting in SAS Visual Forecasting Using Real Time Data

Imputing Missing Values is Easy in SAS Model Studio

BethEbersole — Thu, 17 Nov 2022 17:48:56 GMT

Imputing Missing Values is Easy in SAS Model Studio

Understanding SAS Model Studio Forecast Task Options in SAS Visual Forecasting

BethEbersole — Tue, 11 Oct 2022 16:17:41 GMT

Understanding SAS Model Studio Forecast Task Options (DIAGNOSE, SELECT, FIT, UPDATE, FORECAST) in SAS Visual Forecasting

Downloading Models from SAS Visual Forecasting

BethEbersole — Fri, 02 Sep 2022 17:10:59 GMT

Downloading Models from SAS Visual Forecasting

Accessibility Features in SAS Visual Analytics

BethEbersole — Mon, 25 Jul 2022 12:10:15 GMT

Accessibility Features in SAS Visual Analytics

Honing in on Troublesome Time Series: Interactive Modeling in SAS Visual Forecasting

BethEbersole — Fri, 24 Jun 2022 17:43:54 GMT

Honing in on Troublesome Time Series: Interactive Modeling in SAS Visual Forecasting