topic Re: How to handle large time series data when it does not fit in memory? in SAS Forecasting and Econometrics

How to handle large time series data when it does not fit in memory?

alisio_meneses — Mon, 20 Feb 2023 15:56:59 GMT

Hello there!

I am using SAS Viya 3.5, specifically a Model Studio Forecasting project to generate approximately two thousand forecasts for a dataset that has three years of data for a single dependent variable and four BY variables. The project's pipeline contains one Auto-forecasting node, one hierarchical forecasting node, and it all worked fine for a while until the number of unique values for two of the four BY variables doubled. Since then, the hierarchical forecasting node stops with a bunch of warnings and errors in the log while the Auto-Forecasting node still runs fine (screenshot below).

The following errors are present in the log file for the hierarchical forecasting node:

WARNING: Communication failure among server nodes. Journaling communicator repaired.
WARNING: Communication with machine yadayadayada.yada1 has been lost.
WARNING: Communication with machine yadayadayada.yada2 has been lost.
WARNING: Communication with machine yadayadayada.yada3 has been lost.
ERROR: The action cannot be retried because the session has no available workers.
ERROR: The operation was not performed because contact with at least one node was lost before the operation could complete.
ERROR: The action stopped due to errors.

I am not sure, but it seems the warnings and errors are related to resource exhaustion. Regardless, it made me wonder: how to handle very large time-series dataset using SAS Viya when it does not fit in memory?

Additional environment info: SAS Viya 3.5 using MPP Architecture with one CAS controller and three CAS workers.

Re: How to handle large time series data when it does not fit in memory?

JosvanderVelden — Mon, 20 Feb 2023 18:37:13 GMT

SAS Cloud Analytic Services organizes data from tables in blocks. With the exception of SASHDAT files and specialized cases, a copy of in-memory blocks are temporarily stored in file system directories. When the server is installed, one or more of these directories are specified for the CAS_DISK_CACHE environment variable.

To read more: https://communities.sas.com/t5/SAS-Communities-Library/Provisioning-CAS-DISK-CACHE-for-SAS-Viya/ta-p/603689 & https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/casfun/n16qbskv0hwfq1n1lnrqq605p6sv.htm

Re: How to handle large time series data when it does not fit in memory?

alisio_meneses — Mon, 20 Feb 2023 23:09:40 GMT

I see. Thanks for your info and links. I'll have to check if the CAS_DISK_CACHE is set.

Re: How to handle large time series data when it does not fit in memory?

alisio_meneses — Fri, 24 Feb 2023 10:11:28 GMT

After correctly setting CAS_DISK_CACHE, the errors went away. The side effect is a slow pipeline execution. I mean, really slow. But then again, as one of the provided links you states: '....if CAS is relying on persistent storage for its cache, then expect a commensurate slowdown in performance since data loading from physical disk is much slower than from RAM.'

Thank you for your help, @JosvanderVelden .