topic Very Large Dataset - Out of Resources in New SAS User

Very Large Dataset - Out of Resources

JJ_83 — Thu, 22 Apr 2021 14:09:05 GMT

I have a dataset that is 948,036 obs by 221 var. I need each obs to have a day value from 1/1/2008 to 12/31/2017. The output dataset would be 3,462,227,000 obs by 221 var.

data want (compress=yes);
	set have;
    	do day='01JAN2008'd to '31DEC2017'd;
			output;
	end;
	format day mmddyy10.;
run;

When I run this code I get the error below.

I have 16 gb of ram and about 1tb of storage. Is the limitation my device or SAS, and are there any ways to work with datasets this large in SAS?

Re: Very Large Dataset - Out of Resources

ballardw — Thu, 22 Apr 2021 14:15:27 GMT

What SAS environment are you using?

If you are connecting to a SAS server your SAS admin is likely to have a limit on how much space you are allowed to use.

If you are using SAS On demand I believe there are limits on work space.

Why are you duplicating every value in your existing data 36500 times except for that date? Sounds like a not well thought out problem.

Re: Very Large Dataset - Out of Resources

Kurt_Bremser — Thu, 22 Apr 2021 14:23:40 GMT

Take the sizes of the variables (numeric needs 8 byte per variable), add them up, and multiply by the expected number of observations, so you have a rough idea of the disk space needed.

A quick calculation, assuming all 221 variables to be numbers, is this:

221 * 8 * 948036 * 3650

and results in a size of roughly 5.7 TB. Add a little overhead, and you'll need at least 6 TB to store that. But to WORK with it, you'll need at least three times the size, and a really hefty computer to process this much in tolerable time.

What good is this mass of redundancy, anyway?

Re: Very Large Dataset - Out of Resources

yabwon — Thu, 22 Apr 2021 14:35:45 GMT

1) "I need each obs to have a day value from 1/1/2008 to 12/31/2017" - why? is it necessary?

2) even if each variable would be no bigger than 2 bytes, still 1tb may not be enough...

Re: Very Large Dataset - Out of Resources

Reeza — Thu, 22 Apr 2021 15:22:00 GMT

May be a case where you need to loop and deal with each observation individually. More details are needed. Usually the recommendation is to not be loopy but if you are under resourced for your problem, not much choice.

Re: Very Large Dataset - Out of Resources

ballardw — Thu, 22 Apr 2021 15:35:19 GMT

Here's an idea:

Create a very small data set, or use one of the SAS supplied data sets like SASHELP.CLASS , CARS or STOCKS and add maybe 30 dates to the data.

Then show us what you intended to do with your original data using this much reduced data set. We might have some other suggestions.

Re: Very Large Dataset - Out of Resources

mkeintz — Thu, 22 Apr 2021 17:11:57 GMT

I agree with @yabwon.

Your problem may very well be more about how you want to organize data to accomplish your task, than about disk space.

The revealing statement that you want 3,652 days for each incoming obs, multiplying the size of your data set by 3,652, does not have an obvious justification to me. After all, you are repeating the other 220 variables as constants - that's a lot of needless duplication.

What is the intended use of this gigantic data set?