I'm using Proc Export to export a large (53gb, and compression option = yes) .sas7bdat to .csv format.
I'm confused that SAS begins to hog all of my RAM: I went from 2-3 of 16 gb being used to 15.7 of 16 gb used.
Any ideas why SAS requires so much RAM to save a .csv? Also, I know there are multiple ways to save .sas7bdat's to .csv, Is there another way to save this file that is less RAM intensive?
No need to use EXPORT to generate a CSV file from SAS dataset. Just write it using a normal data step. This will not use much memory.
set mydataset ;
file 'myfile.csv' dsd lrecl=2000000 ;
put (_all_) (:) ;
If you want to write the column headers there are a number of solutions. One that I find useful is to use PROC EXPORT, but use OBS=0 dataset option to write just the column headers and then use the above data step with the MOD option on the FILE statement to append the data.
I use proc export as well to get a csv data. But sometimes I don't get to see all the columns exported. There are not too many rows though. I have about 3000 columns. Do you think the data step would help to sort my problem as well?
Instead of using EXPORT, your datastep approach is very fast even for the big files too. but i was stumped at the later part of yours
"One that I find useful is to use PROC EXPORT, but use OBS=0 dataset option to write just the column headers and then use the above data step with the MOD option on the FILE statement to append the data"
Could you able to show me with an example if possible how to use MOD option to append the data?
Could be. I believe that PROC EXPORT will only create CSV files with maximum line length of 32,767. So for 3000 variables if the average value is longer than 10 you will get truncation or wrapping of lines. If you write your own data step (or recall the one that PROC EXPORT generates and modify it) you can use a larger value for the LRECL option on the FILE statement.
LRECL is Logical RECord Length. It is how many characters to allow per line of data in a text file. So if in one observation the sum of lengths of the displayed values for all variables (including the commas and any quoting required to mask embedded commas or quotes) is longer than the LRECL specified in the FILE statement then SAS will wrap the lines and cause your CSV file to be confused. I am not sure what the max value that SAS supports for LRECL, but there is normally no harm in setting it much longer than you expect to use. Unless you set RECFM (RECord ForMat) to F (Fixed).
The number of observations (rows or lines) is not limited, other than perhaps by your disk space.
This was driving me crazy too. Here's is the official SAS statement:
"Beginning in SAS 9.2 PROC EXPORT and the EXPORT WIZARD limit the record length of an external file to 32767. At this time there is not a way to increase the LRECL= value to be greater than 32767."
There are two workarounds, I think using ods for my data is insane so hopefully the macro will work.
Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.
If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website.
Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.
Find more tutorials on the SAS Users YouTube channel.