'Insufficient memory' or 'No disk space is available' errors in large size datasets

[bhandara]

Hi,

I have often encountered ‘Insufficient Memory' or ‘No disk space is available’ errors in SAS whenever I try to analyze large size datasets in SAS (>100 GB). I use 15 TB external hard drive and connect SAS directly to the external drive using libname. I have 16 GB of RAM, Microsoft Windows 10 Enterprise OS, Intel® Core™ i76500U CPU 2.5 GHz processor. In regards to my internal HD storage, I have around 75 GB left in C: and around 281 GB left in E:.

Let me know the steps to be taken to sort this issue which I only face while using large size datasets.

Regards,
Avi

[jimbarbour]

Where is your WORK directory pointed and where is your Utility space allocated to?  Even if your primary Libname is allocated to your 15 Terabyte external drive, if you run out of WORK space, your job will will get 'No disk space is available' error messages.  

Can you run this and post the log please?

PROC	OPTIONS	OPTION=(UTILLOC WORK);
RUN;

The above should tell you where your WORK Libname points.  If it's not to the large external drive, then you can allocate a USER library that points to the external drive, and SAS should use that for it's work space.

The above should also tell you where your location for Utility files is.  I would think that it also should be set to the large external 15 TB drive when your processing really big files.

The memory issue is a separate issue.  What kind of analysis are you doing?  Exactly which SAS command is executing when the memory issue occurs?  Can you share your code?

Jim

[ChrisNZ]

These 2 messages are unrelated (unless the disk fills because of paging, but I won't go there).

'Insufficient memory' is about RAM. 

You can increase the RAM in the computer, or use different processing algorithms, or split the data into smaller chunks, or increase the size of the paging file.

'No disk space is available'  is about disk. 

Check which disk fills up, what the processes do when this happens, and what the files are that take all this space. To increase disk space, you can reduce data set size (the SPDE engine with binary compression is extremely efficient) or reduce temp/utility file size by compressing the utility folder.

Note that disks are orders of magnitude slower than RAM, so when disks are heavily used, run times surge.

[nazmul · Accepted solution]

As other posts have pointed out, your insufficient memory is related to RAM. Have you tried cleaning up temporary files? In the past, I had encountered similar issues while executing the BoardEx database in SAS. This database has a size of 400+GB. I was able to solve the problem by upgrading RAM from 8 GB to 32 GB. As you may see on other posts (posted below) and also posted by Chris above, SAS insufficient memory points out to RAM. It has nothing to do with your space in the internal and external hard drives. RAM is used for temporary storage. The hard drive is capable of storing information after the computer turns off, but RAM is not. Regardless of the fact that you have large internal or external hard drives, SAS matrices are kept in RAM. Memory is automatically allocated when needed.

Even though removing temporary files frequently could be an option, it may not much help for you since you need significant memory for executing each SAS code. Thus, I recommend you upgrade your configuration to 32 GB or more RAM size.

If you are using SAS visual analytics, a minimum of 16 GB is required for each core. Thus, if you have two cores, you would need 32 GB of RAM. If you operate SAS in four cores, you would need 64 GB of RAM.

https://support.sas.com/documentation/installcenter/en/ikvisanlytofrsr/68904/HTML/default/index.html...

Below is another thread that discusses a similar issue and states that “Insufficient memory points to RAM, not a disk.”

https://communities.sas.com/t5/General-SAS-Programming/Insufficient-memory-log-error-identify-what-c...

Regarding the no disk space available problem, you have two options. The first and preferred option is that you allocate a USER library to your external drive, which has 15 TB of space. This is also recommended above by Jim. The second option and less preferable one- is to increase the size of your internal hard drive. I don’t prefer the second option.

Regards,

Md

[bhandara]

Thank you Jim, Chris and Md.  The information you all shared is helpful.