Hi Everyone,
I have a PC with 32GB ram.
I wonder if the setting I got below maximize the use of Ram yet?
If not, how can I change the setting?
Thank you,
HHCFX
Group=MEMORY
SORTSIZE=268435456
Size parameter for sort
SUMSIZE=0 Upper limit for data-dependent memory usage during summarization
MAXMEMQUERY=0 Maximum amount of memory returned when inquiring as to available space
MEMBLKSZ=16777216 Size of memory blocks allocated to support MEMLIB and MEMCACHE options.
MEMMAXSZ=2147483648
Maximum amount of memory allocated to support MEMLIB and MEMCACHE options.
LOADMEMSIZE=0 Suggested memory limit for loaded SAS executables
MEMSIZE=2147483648
Specifies the limit on the total amount of memory to be used by the SAS
System
REALMEMSIZE=0 Limit on the total amount of real memory to be used by the SAS System
Usually you should only increase the MEMSIZE / SORTSIZE options' values from the default if your SAS jobs are running out of memory for processing or sorting. Increasing these won't necessarily reduce job run times.
So I see 1 way to change MEMSIZE is: Start -> Run -> sas.exe -memsize 30G
Also the OPTIONS way but I try this one and it doesn't work.
proc options memsize=3047483648;
run;
How can I fix that proc?
Thank you.
HHCFX
To make things easier, you may want to consider having your own custom SAS config file. That way you can tweak settings quickly, especially for testing (edit the file and restart SAS to pick up any changes). My main SAS workstation has an i7-6700 processor, 32GB RAM, and a 512GB SSD (SanDisk X400) and I've been pretty happy with it for development work.
My SAS desktop shortcut is:
"C:\Program Files\SASHome\SASFoundation\9.4\sas.exe" -CONFIG "C:\users\dave\sas94.cfg"
In the sas94.cfg file I include the default shipping UTF-8 config followed by my custom settings:
/* * sas94.cfg * * This config file sets up the SAS environment for Dave - UTF8. * */ /* Include default SAS config file */ -config "C:\Program Files\SASHome\SASFoundation\9.4\nls\u8\sasv9.cfg" /* Update Autocall macro path */ -insert sasautos "C:\src\macros" /* Autoexec location */ -autoexec "C:\src\autoexec_win.sas" /* Memlib Memory options */ -MEMLIB -MEMCACHE 4 -MEMMAXSZ 32G -MEMSIZE MAX -SORTSIZE 1G /* General options */ -FONT 'Consolas' 10 -awstitle "SAS 9.4M5 UTF-8" -noenhancededitor -nodmsexp -nofmterr -comamid tcp -noxwait -pagesize max -dmslogsize max -dmsoutsize max -errors 10 -altlog "C:\temp\altutf.log" -work "C:\Temp\sastemp" /* File I/O */ -BUFNO 200 -BUFSIZE 64K -UBUFSIZE 64K -UBUFNO 20 -IBUFNO 64 -IBUFSIZE MAX -CBUFNO MAX -tbufsize 512k -catcache max -compress yes /* Oracle options */ -DBIDIRECTEXEC -sql_ip_trace note /* Email options */ -EMAILSYS SMTP -EMAILHOST mail.yourhost.com /* Stats and messages */ -STIMER -FULLSTIMER -MSGLEVEL I -mautolocdisplay -source2 -oplist -nosplash
I set the memory sizes to the max, and also turn on MEMLIB and MEMCACHE which utilizes RAM for work files and caching. Depending on your file sizes, you may or may not be able to fit everything in memory and may have to turn those options off. The rest of the options are not directly related to your question so I thought I'd leave them in there for completeness and discussion 🙂 I actually have several config files in case I need to run with other encodings (ie Latin1) or without the memlib/memcache, etc.
So I run this proc to find the config file.
proc options option=config;run;
Mine is i7 3770 32G 512G SSD.
I play around with
-MEMCACHE 4 -MEMMAXSZ 32G -MEMSIZE MAX -SORTSIZE 1G
Clearly, making MEMSIZE MAX and MEMMAXSZ 32G
make huge improvement in data step. In this step, I simply make a big file from my existing file. It cut time form:
CPU from 19.53 to 16.66
Real time from 31.11 to 17.84
I am not sure why real time reduce a lot while CPU reduce not as much
SORTSIZE benefit is not clear. Leaving at the 256MB or change to 0.5G or 1G doesn't impact datastep or Proc sort much.
I wonder why setting it to 1G doesn't help at all?
-MEMCACHE 4
Turns out to be disaster, in the "Windows Task Manager", I saw the Ram usage rocket to 30 G and stay flat there for really long time.
Data step take longer time. (with out that MEMCACHE, ram usage max about 28G).
Thank you so much.
HHCFX
7 data combine_file; set Ret_buy_fix_tp_notrail_h4 Ret_buy_fix_tp_notrail_h4
7 ! Ret_buy_fix_tp_notrail_h4; run;
NOTE: There were 549101 observations read from the data set WORK.RET_BUY_FIX_TP_NOTRAIL_H4.
NOTE: There were 549101 observations read from the data set WORK.RET_BUY_FIX_TP_NOTRAIL_H4.
NOTE: There were 549101 observations read from the data set WORK.RET_BUY_FIX_TP_NOTRAIL_H4.
NOTE: The data set WORK.COMBINE_FILE has 1647303 observations and 568 variables.
NOTE: DATA statement used (Total process time):
real time 17.84 seconds
cpu time 16.66 seconds
8
9
10 proc sort data=combine_file ; by decending rate Q_date tradetime;run;
NOTE: There were 1647303 observations read from the data set WORK.COMBINE_FILE.
NOTE: The data set WORK.COMBINE_FILE has 1647303 observations and 568 variables.
NOTE: PROCEDURE SORT used (Total process time):
real time 38.22 seconds
cpu time 32.27 seconds
@hhchenfx did you also introduce the File I/O options @DaveHorne had in his configuration?
For simple datasteps if you are not hitting errors mentioning being out of memory that just adjusting the memory limit by itself would not affect the runtime.
Its more likely your improved datastep run times came from improved I/O tuning rather than increasing the memory limit.
All I did is to change these 4 things.
I didn't touch the I/O at all.
My SAS is running now so I will try the I/O later.
@hhchenfx seeing the log and that the data set has 568 variables, that's probably the biggest impact on your performance in this particular case (especially in the sort step). If you don't need all those variables in your analysis, then keeping (or dropping) variables in the data step can help.
If you do need to keep all the variables, what are their total length (observation length)? You can get the observation length from proc contents and bump the bufsize value (in my config I have the bufsize at 64K but you could definitely go higher so that you can fit more observations into a page of memory).
Since you have a fast processor, try turning on the compress option as well. That may not speed up processing a great deal but it should save space for data sets stored on disk (and save I/O reading them).
Closing the time difference between CPU and real time is the goal; keep the cpu busier than the disk drive (I/O).
Thank you for your suggestion.
It is the first time I look into this matter!
Below is the content of my data I normally work with.
So what do you think the right Bufsize for me. By the way, what is the cost of bumping Bufsize too hight?
Thank you,
HHCFX
SAS Output
Data Set Name | INDI2018.RET_BUY_FIX_TP_NOTRAIL_H4 | Observations | 549101 |
---|---|---|---|
Member Type | DATA | Variables | 568 |
Engine | V9 | Indexes | 0 |
Created | Monday, February 05, 2018 02:53:54 AM | Observation Length | 4544 |
Last Modified | Monday, February 05, 2018 02:53:54 AM | Deleted Observations | 0 |
Protection | Compressed | NO |
SAS Output
Engine/Host Dependent Information | |
---|---|
Data Set Page Size | 16384 |
Number of Data Set Pages | 183038 |
First Data Page | 5 |
Max Obs per Page | 3 |
Obs in First Data Page | 3 |
Regarding bufsize, here is a link that might interest you.
https://communities.sas.com/t5/Base-SAS-Programming/Using-bufsize-and-bufno/m-p/401170#M97302
Regarding compressing such wide tables,I recommend you try the SPDE engine with binary compression. You'll probably reduce the table size to a few percent points of the uncompressed size (i.e. > 90% compression ratio).
Is there a way to have SAS recognize an increased MEMSIZE for the 32bit version of SAS? I have plenty of RAM and have set up the config file the way you suggested and that's the only thing that is defaulting backwards (it will let me have 4GB of RAM).
2^32 = 4,294,967,296 = 4 GB.
That's a limitation of the architecture, and one reason the world has moved to 64 bits.
Thanks!
@DaveHorne Do you actually use MEMCACHE Dave? I find it very inconvenient to use.
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
Learn how use the CAT functions in SAS to join values from multiple variables into a single value.
Find more tutorials on the SAS Users YouTube channel.