topic Re: ERROR: Insufficient space in file WORK.ACS.DATA in SAS Programming

ERROR: Insufficient space in file WORK.ACS.DATA

thanksforhelp12 — Sun, 27 Jan 2019 01:39:22 GMT

I keep getting this error while attempting to load a large dataset (.sas7bdat) file into the workspace (Library "work" with table name "ACS") so that I can work with it.

No idea how to fix or why this is happening, so I haven't tried much other than clearing my work library and restarting the SAS virtual machine.

I have 200 GB free. I searched around and there were many posts about this happening with SQL, but at least to which I am aware, I am not doing anything with SQL.

Thanks for the help.

Re: ERROR: Insufficient space in file WORK.ACS.DATA

Reeza — Sun, 27 Jan 2019 01:49:06 GMT

@thanksforhelp12 wrote:

I keep getting this error while attempting to load a large dataset (.sas7bdat) file into the workspace (Library "work" with table name "ACS") so that I can work with it.

No idea how to fix or why this is happening, so I haven't tried much other than clearing my work library and restarting the SAS virtual machine.

I have 200 GB free. I searched around and there were many posts about this happening with SQL, but at least to which I am aware, I am not doing anything with SQL.

Thanks for the help.

Are you using SAS University edition? Or a custom installation?

A custom installation only has limits based on what your administration has set.

If you're using SAS UE, there are some limits based on the machine - what do you have RAM and/or cores set to? How big is your 'big' data set. You've set the shared folders to your main hard drive and not a cloud drive with a limit?

Re: ERROR: Insufficient space in file WORK.ACS.DATA

thanksforhelp12 — Sun, 27 Jan 2019 01:56:54 GMT

Thank you for your response and willingness to help. I am indeed using University Edition with shared folder on my local drive.

The dataset is about 1,000,000 rows by 333 columns.

Good thought on the RAM and cores, my actual computer is 8gb/4core (virtual 😎 but I have not looked into the SAS virtual machine settings. I believe the RAM may have been set at 1024 mb. Would altering these setting help the issue you think?

Re: ERROR: Insufficient space in file WORK.ACS.DATA

Reeza — Sun, 27 Jan 2019 01:59:42 GMT

Oh,that's not big at all. Have you redirected your work library as I suggested in other posts?

Re: ERROR: Insufficient space in file WORK.ACS.DATA

thanksforhelp12 — Sun, 27 Jan 2019 02:11:03 GMT

I have not. I tried to search your posts, is this what you are referring to? https://communities.sas.com/t5/SAS-Analytics-U/How-to-clear-up-space-in-WORK-folder/m-p/336740?

If so, any further tips/clarification is much appreciated.

Re: ERROR: Insufficient space in file WORK.ACS.DATA

Reeza — Sun, 27 Jan 2019 02:26:49 GMT

1. Change RAM settings in the virtual machine, I recommend 4GB, but it depends on your system of course. The SAS minimum is 1GB I believe, but 2GB seems more reasonable IMO.

2. Check the number of cores allotted, in the virtual machine, the limit is 2 cores.

3. Redirect your work library to outside of the VM, see instructions below.

You can redirect your work library to a different folder using a USER libname. When a USER library is available, SAS will default data sets to there instead of WORK. This does mean that there is no clean up of temporary data sets when it shuts down though, so this becomes your responsibility. I usually create a temp folder and keep it in there and only use it on projects with size issues.

libname user '/folders/myfolders/temp';

If your system has had several issues with large data sets or crashed while working on large data sets, SAS may not have cleaned up all the files and this can cause problems with the VM. If this happens your best bet is to just delete the VM and reinstall it. DO NOT DELETE your MYFOLDERS folder. You can reconnect to your new installation to it and be fine.

@thanksforhelp12 wrote:

I have not. I tried to search your posts, is this what you are referring to? https://communities.sas.com/t5/SAS-Analytics-U/How-to-clear-up-space-in-WORK-folder/m-p/336740?

If so, any further tips/clarification is much appreciated.

Re: ERROR: Insufficient space in file WORK.ACS.DATA

thanksforhelp12 — Sun, 27 Jan 2019 03:16:55 GMT

Thanks for the continued help.

My command where I load in the data is currently:

data work.ACS;
  set '/folders/myfolders/acs_nsqip_puf12.sas7bdat';
run;

For the above trick to work, do I need to remove the "work." piece?

Edit: I tried that and got this response in the log:

ERROR: Insufficient space in file WORK.'SASTMP-000000047'n.ITEMSTOR.

ERROR: Fatal ODS error has occurred. Unable to continue processing this output destination.

WARNING: No body file. RTF output will not be created.

ERROR: Out of space writing to file /tmp/SAS_work6DED0000097C_localhost.localdomain/#LN00076.

An alternative, potentially, would be if there is some way I can start filtering data from the .sas7bdat file without ever loading it in.

Thanks.

Re: ERROR: Insufficient space in file WORK.ACS.DATA

thanksforhelp12 — Sun, 27 Jan 2019 04:44:16 GMT

Update 2: I installed a new version of the VM...

even though I am only trying to load a subset of the .sas7bdat file into the workspace, it seems it loads the whole thing into the WC000001 library no matter what. These then seemed to build up, even when I cleared them. Is there anyway to avoid this?

Thanks!

Re: ERROR: Insufficient space in file WORK.ACS.DATA

Reeza — Sun, 27 Jan 2019 22:43:47 GMT

Add obs = to read the file in portions? What’s the actual size, I would expect it to be under 1GB so with other changes it should work.

Data want1;
Set yourData(obs=1000);

Data want2;
Set yourdata (firstobs=1001 obs=2000);
Run;

Re: ERROR: Insufficient space in file WORK.ACS.DATA

Tom — Sun, 27 Jan 2019 23:41:55 GMT

Why are you copying the dataset into the WORK library?

Why not just run whatever analysis you need on the dataset where it is?

Re: ERROR: Insufficient space in file WORK.ACS.DATA

Vince_SAS — Mon, 28 Jan 2019 13:11:43 GMT

@thanksforhelp12 wrote:

Edit: I tried that and got this response in the log:

ERROR: Insufficient space in file WORK.'SASTMP-000000047'n.ITEMSTOR.

ERROR: Fatal ODS error has occurred. Unable to continue processing this output destination.

WARNING: No body file. RTF output will not be created.

ERROR: Out of space writing to file /tmp/SAS_work6DED0000097C_localhost.localdomain/#LN00076.

This error indicates that ODS fails to create an RTF file. Is it your intention to create an RTF file? If not, then turn off that option (and also the option to create PDF, if that is also on).

Vince DelGobbo

SAS R&D

Re: ERROR: Insufficient space in file WORK.ACS.DATA

thanksforhelp12 — Wed, 30 Jan 2019 14:34:03 GMT

How would one do this? This would certainly be ideal. Even when I run a "where" command after the set command above in order to just select the small subset of the large data file (which is 3GB+ for those asking), it seems to load the entire file into the WC000001 library. Is there a better way to load in the data?

Thanks.

Re: ERROR: Insufficient space in file WORK.ACS.DATA

Tom — Wed, 30 Jan 2019 15:23:51 GMT

All the datastep you posted does is make a duplicate copy of the data. It is not DOING anything with it.

What do you want to DO with the data?

Do you want to run a regression?

data reg data= '/folders/myfolders/acs_nsqip_puf12.sas7bdat';
  model response=input1 input2;
run;

Re: ERROR: Insufficient space in file WORK.ACS.DATA

Vince_SAS — Wed, 30 Jan 2019 15:33:58 GMT

I don't understand how this code shown earlier:

data work.ACS;
  set '/folders/myfolders/acs_nsqip_puf12.sas7bdat';
run;

Is causing this error message, also shown earlier:

ERROR: Insufficient space in file WORK.'SASTMP-000000047'n.ITEMSTOR.
ERROR: Fatal ODS error has occurred. Unable to continue processing this output destination.
WARNING: No body file. RTF output will not be created.

Is there more code than what has been already posted?

Vince DelGobbo

SAS R&D

Re: ERROR: Insufficient space in file WORK.ACS.DATA

thanksforhelp12 — Wed, 30 Jan 2019 17:28:54 GMT

I would like to select a subset of the data for further analyses.

data ACS;
set '/folders/myfolders/file.sas7bdat';
where Column1 = "X" OR Column2 = "Y";
run;

When I do this, even though I am selecting just a subset, it seems to create a copy of the entire big file in my WC000001 library. Either immediately then or after I subset a different datafile, the insufficient space error is thrown.

Re: ERROR: Insufficient space in file WORK.ACS.DATA

thanksforhelp12 — Wed, 30 Jan 2019 17:27:46 GMT

I just reviewed my file and I have absolutely nothing referencing RTF anywhere in it. Nonetheless, it is possible there may have been something there before.

Re: ERROR: Insufficient space in file WORK.ACS.DATA

Vince_SAS — Wed, 30 Jan 2019 17:44:59 GMT

Look in your SAS Studio Preferences. There are options to create RTF and PDF under the Results section. RTF should only be created when you display procedure results, but we don't see evidence of that in the code that you posted.

Vince DelGobbo

SAS R&D