Desktop productivity for business analysts and programmers

SAS Enterprise not enough memory

Accepted Solution Solved
Reply
Contributor
Posts: 58
Accepted Solution

SAS Enterprise not enough memory

I am importing large txt files (10 - 12 gb per file) using SAS Enterprise and export them into an external shared drive, but run into memory issues.

Is there a setting to allocate more memory to SAS Enterprise? Sometime the import crashes SAS, sometimes the export, all in all import and export take tremendously long, as I can only inport 3 files (a total of 26GB) per day - if at all.

 

Windows 10, 8RAM (not enough?), 151GB on C drive where SAS is installed, 1TB external memor

Many thanks for any help,

Anna

 


Accepted Solutions
Solution
‎05-21-2017 10:43 PM
SAS Super FREQ
Posts: 306

Re: SAS Enterprise not enough memory

Also wanted to note, in case you are using the EG point-and-click import/export features...  we generally recommend using SAS code (ex. proc import/export, DATA step) for importing/exporting very large files, since SAS code (server-side) is usually more efficient than EG's Import Wizard and point-and-click Export (client-side) with large data.

View solution in original post


All Replies
Super User
Posts: 3,233

Re: SAS Enterprise not enough memory

Can you post your actual error messages? Are you running out of memory or is it disk space? Where is SAS actually running? On your PC (local server) or on a remote server?

Contributor
Posts: 58

Re: SAS Enterprise not enough memory

Hi SASKiwi, thanks for coming back to this post. 


The message I receive is:
 
The SAS System stopped processing this step because of insufficient memory.
 
Both SAS Enterprise and 9.4 are running on the computer and are located in the C drive, which has 150GB available. I shuffled everything out of there to create as much memory/ space as possible. 
So I don´t quite get what the problem is. 
 
CPU too small? Not enough memory allocated to SAS work ?
Many thanks for your help, 
Anna
Super User
Posts: 7,422

Re: SAS Enterprise not enough memory

[ Edited ]

Please post the code ("little running man" icon) and the log - including all code of the step that fails (use {i}).

 

PS A data step that imports plain text never runs out of memory on its own, as it only deals with one line of data at a time.

If you use proc import, it might be that guessingrows is set too high and SAS runs out of memory while prefetching for the guess.

With this kind of data, I'd never use proc import, but write the data step according to the file specification.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Solution
‎05-21-2017 10:43 PM
SAS Super FREQ
Posts: 306

Re: SAS Enterprise not enough memory

Also wanted to note, in case you are using the EG point-and-click import/export features...  we generally recommend using SAS code (ex. proc import/export, DATA step) for importing/exporting very large files, since SAS code (server-side) is usually more efficient than EG's Import Wizard and point-and-click Export (client-side) with large data.

Contributor
Posts: 58

Re: SAS Enterprise not enough memory

Thank you very much, I did try to import using SAS9.4 instead - same result.

I found out that the limitation comes from the  I/O issue _ amont of data coming in vs amount of data going out. I would have to add more I/O resources to improve the performance, such as running SAS in more server boxes which will require more licenses...

 

Sorry, diverted to other software to deal with big data. But once the data is v=cut into smaller pieces again, I can work in SAS again :-)

 

Thank you for everybody's input and thoughts.

Anna

 

 

 

Contributor
Posts: 58

Re: SAS Enterprise not enough memory

an update:

the papers below helped me to navigate through my problems in importing and dealing with extremely large data. I thought I share them.

Many thanks,

 

http://www2.sas.com/proceedings/sugi27/p023-27.pdf

 

http://www.lexjansen.com/nesug/nesug11/ld/ld02.pdf

☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
  • 6 replies
  • 411 views
  • 1 like
  • 4 in conversation