BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
AnnaNZ
Quartz | Level 8

I am importing large txt files (10 - 12 gb per file) using SAS Enterprise and export them into an external shared drive, but run into memory issues.

Is there a setting to allocate more memory to SAS Enterprise? Sometime the import crashes SAS, sometimes the export, all in all import and export take tremendously long, as I can only inport 3 files (a total of 26GB) per day - if at all.

 

Windows 10, 8RAM (not enough?), 151GB on C drive where SAS is installed, 1TB external memor

Many thanks for any help,

Anna

 

1 ACCEPTED SOLUTION

Accepted Solutions
CaseySmith
SAS Employee

Also wanted to note, in case you are using the EG point-and-click import/export features...  we generally recommend using SAS code (ex. proc import/export, DATA step) for importing/exporting very large files, since SAS code (server-side) is usually more efficient than EG's Import Wizard and point-and-click Export (client-side) with large data.


Register today and join us virtually on June 16!
sasglobalforum.com | #SASGF

View now: on-demand content for SAS users

View solution in original post

6 REPLIES 6
SASKiwi
PROC Star

Can you post your actual error messages? Are you running out of memory or is it disk space? Where is SAS actually running? On your PC (local server) or on a remote server?

AnnaNZ
Quartz | Level 8

Hi SASKiwi, thanks for coming back to this post. 


The message I receive is:
 
The SAS System stopped processing this step because of insufficient memory.
 
Both SAS Enterprise and 9.4 are running on the computer and are located in the C drive, which has 150GB available. I shuffled everything out of there to create as much memory/ space as possible. 
So I don´t quite get what the problem is. 
 
CPU too small? Not enough memory allocated to SAS work ?
Many thanks for your help, 
Anna
Kurt_Bremser
Super User

Please post the code ("little running man" icon) and the log - including all code of the step that fails (use {i}).

 

PS A data step that imports plain text never runs out of memory on its own, as it only deals with one line of data at a time.

If you use proc import, it might be that guessingrows is set too high and SAS runs out of memory while prefetching for the guess.

With this kind of data, I'd never use proc import, but write the data step according to the file specification.

CaseySmith
SAS Employee

Also wanted to note, in case you are using the EG point-and-click import/export features...  we generally recommend using SAS code (ex. proc import/export, DATA step) for importing/exporting very large files, since SAS code (server-side) is usually more efficient than EG's Import Wizard and point-and-click Export (client-side) with large data.


Register today and join us virtually on June 16!
sasglobalforum.com | #SASGF

View now: on-demand content for SAS users

AnnaNZ
Quartz | Level 8

Thank you very much, I did try to import using SAS9.4 instead - same result.

I found out that the limitation comes from the  I/O issue _ amont of data coming in vs amount of data going out. I would have to add more I/O resources to improve the performance, such as running SAS in more server boxes which will require more licenses...

 

Sorry, diverted to other software to deal with big data. But once the data is v=cut into smaller pieces again, I can work in SAS again 🙂

 

Thank you for everybody's input and thoughts.

Anna

 

 

 

AnnaNZ
Quartz | Level 8

an update:

the papers below helped me to navigate through my problems in importing and dealing with extremely large data. I thought I share them.

Many thanks,

 

http://www2.sas.com/proceedings/sugi27/p023-27.pdf

 

http://www.lexjansen.com/nesug/nesug11/ld/ld02.pdf

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

SAS Enterprise Guide vs. SAS Studio

What’s the difference between SAS Enterprise Guide and SAS Studio? How are they similar? Just ask SAS’ Danny Modlin.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 6 replies
  • 5453 views
  • 1 like
  • 4 in conversation