I am importing large txt files (10 - 12 gb per file) using SAS Enterprise and export them into an external shared drive, but run into memory issues.
Is there a setting to allocate more memory to SAS Enterprise? Sometime the import crashes SAS, sometimes the export, all in all import and export take tremendously long, as I can only inport 3 files (a total of 26GB) per day - if at all.
Windows 10, 8RAM (not enough?), 151GB on C drive where SAS is installed, 1TB external memor
Many thanks for any help,
Anna
Also wanted to note, in case you are using the EG point-and-click import/export features... we generally recommend using SAS code (ex. proc import/export, DATA step) for importing/exporting very large files, since SAS code (server-side) is usually more efficient than EG's Import Wizard and point-and-click Export (client-side) with large data.
Register today and join us virtually on June 16!
sasglobalforum.com | #SASGF
View now: on-demand content for SAS users
Can you post your actual error messages? Are you running out of memory or is it disk space? Where is SAS actually running? On your PC (local server) or on a remote server?
Hi SASKiwi, thanks for coming back to this post.
Please post the code ("little running man" icon) and the log - including all code of the step that fails (use {i}).
PS A data step that imports plain text never runs out of memory on its own, as it only deals with one line of data at a time.
If you use proc import, it might be that guessingrows is set too high and SAS runs out of memory while prefetching for the guess.
With this kind of data, I'd never use proc import, but write the data step according to the file specification.
Also wanted to note, in case you are using the EG point-and-click import/export features... we generally recommend using SAS code (ex. proc import/export, DATA step) for importing/exporting very large files, since SAS code (server-side) is usually more efficient than EG's Import Wizard and point-and-click Export (client-side) with large data.
Register today and join us virtually on June 16!
sasglobalforum.com | #SASGF
View now: on-demand content for SAS users
Thank you very much, I did try to import using SAS9.4 instead - same result.
I found out that the limitation comes from the I/O issue _ amont of data coming in vs amount of data going out. I would have to add more I/O resources to improve the performance, such as running SAS in more server boxes which will require more licenses...
Sorry, diverted to other software to deal with big data. But once the data is v=cut into smaller pieces again, I can work in SAS again 🙂
Thank you for everybody's input and thoughts.
Anna
an update:
the papers below helped me to navigate through my problems in importing and dealing with extremely large data. I thought I share them.
Many thanks,
http://www2.sas.com/proceedings/sugi27/p023-27.pdf
Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.
Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.
What’s the difference between SAS Enterprise Guide and SAS Studio? How are they similar? Just ask SAS’ Danny Modlin.
Find more tutorials on the SAS Users YouTube channel.