topic Re: SAS Enterprise not enough memory in SAS Enterprise Guide

SAS Enterprise not enough memory

AnnaNZ — Tue, 16 May 2017 04:28:51 GMT

I am importing large txt files (10 - 12 gb per file) using SAS Enterprise and export them into an external shared drive, but run into memory issues.

Is there a setting to allocate more memory to SAS Enterprise? Sometime the import crashes SAS, sometimes the export, all in all import and export take tremendously long, as I can only inport 3 files (a total of 26GB) per day - if at all.

Windows 10, 8RAM (not enough?), 151GB on C drive where SAS is installed, 1TB external memor

Many thanks for any help,

Anna

Re: SAS Enterprise not enough memory

SASKiwi — Tue, 16 May 2017 07:09:30 GMT

Can you post your actual error messages? Are you running out of memory or is it disk space? Where is SAS actually running? On your PC (local server) or on a remote server?

Re: SAS Enterprise not enough memory

AnnaNZ — Tue, 16 May 2017 08:14:36 GMT

Hi SASKiwi, thanks for coming back to this post.

The message I receive is:

The SAS System stopped processing this step because of insufficient memory.

Both SAS Enterprise and 9.4 are running on the computer and are located in the C drive, which has 150GB available. I shuffled everything out of there to create as much memory/ space as possible.

So I don´t quite get what the problem is.

CPU too small? Not enough memory allocated to SAS work ?

Many thanks for your help,

Anna

Re: SAS Enterprise not enough memory

Kurt_Bremser — Tue, 16 May 2017 10:07:38 GMT

Please post the code ("little running man" icon) and the log - including all code of the step that fails (use {i}).

PS A data step that imports plain text never runs out of memory on its own, as it only deals with one line of data at a time.

If you use proc import, it might be that guessingrows is set too high and SAS runs out of memory while prefetching for the guess.

With this kind of data, I'd never use proc import, but write the data step according to the file specification.

Re: SAS Enterprise not enough memory

CaseySmith — Thu, 18 May 2017 14:59:55 GMT

Also wanted to note, in case you are using the EG point-and-click import/export features... we generally recommend using SAS code (ex. proc import/export, DATA step) for importing/exporting very large files, since SAS code (server-side) is usually more efficient than EG's Import Wizard and point-and-click Export (client-side) with large data.

Re: SAS Enterprise not enough memory

AnnaNZ — Mon, 22 May 2017 02:42:41 GMT

Thank you very much, I did try to import using SAS9.4 instead - same result.

I found out that the limitation comes from the I/O issue _ amont of data coming in vs amount of data going out. I would have to add more I/O resources to improve the performance, such as running SAS in more server boxes which will require more licenses...

Sorry, diverted to other software to deal with big data. But once the data is v=cut into smaller pieces again, I can work in SAS again 🙂

Thank you for everybody's input and thoughts.

Anna

Re: SAS Enterprise not enough memory

AnnaNZ — Thu, 08 Jun 2017 03:40:09 GMT

an update:

the papers below helped me to navigate through my problems in importing and dealing with extremely large data. I thought I share them.

Many thanks,

http://www2.sas.com/proceedings/sugi27/p023-27.pdf

http://www.lexjansen.com/nesug/nesug11/ld/ld02.pdf