BookmarkSubscribeRSS Feed
hhchenfx
Barite | Level 11

Hi Everyone,

I have a PC with 32GB ram.

I wonder if the setting I got below maximize the use of Ram yet?

If not, how can I change the setting?

Thank you,

HHCFX

 

Group=MEMORY

 SORTSIZE=268435456

                   Size parameter for sort

 SUMSIZE=0         Upper limit for data-dependent memory usage during summarization

 MAXMEMQUERY=0     Maximum amount of memory returned when inquiring as to available space

 MEMBLKSZ=16777216 Size of memory blocks allocated to support MEMLIB and MEMCACHE options.

 MEMMAXSZ=2147483648

                   Maximum amount of memory allocated to support MEMLIB and MEMCACHE options.

 LOADMEMSIZE=0     Suggested memory limit for loaded SAS executables

 MEMSIZE=2147483648

                   Specifies the limit on the total amount of memory to be used by the SAS

                   System

 REALMEMSIZE=0     Limit on the total amount of real memory to be used by the SAS System

19 REPLIES 19
LinusH
Tourmaline | Level 20
No you are not.
Increase MEMSIZE and SORTSIZE options.
Data never sleeps
SASKiwi
PROC Star

Usually you should only increase the MEMSIZE / SORTSIZE options' values from the default if your SAS jobs are running out of memory for processing or sorting. Increasing these won't necessarily reduce job run times.

hhchenfx
Barite | Level 11

So I see 1 way to change MEMSIZE is: Start -> Run -> sas.exe -memsize 30G

 

Also the OPTIONS way but I try this one and it doesn't work.

 

proc options memsize=3047483648;
run;

 

How can I fix that proc?

 

Thank you.

 

HHCFX

DaveHorne
SAS Employee

To make things easier, you may want to consider having your own custom SAS config file.  That way you can tweak settings quickly, especially for testing (edit the file and restart SAS to pick up any changes).  My main SAS workstation has an i7-6700 processor, 32GB RAM, and a 512GB SSD (SanDisk X400) and I've been pretty happy with it for development work.

 

My SAS desktop shortcut is:

"C:\Program Files\SASHome\SASFoundation\9.4\sas.exe" -CONFIG "C:\users\dave\sas94.cfg"

In the sas94.cfg file I include the default shipping UTF-8 config followed by my custom settings:

 

/*
 * sas94.cfg
 *
 * This config file sets up the SAS environment for Dave - UTF8.
 *
 */

/* Include default SAS config file */

-config "C:\Program Files\SASHome\SASFoundation\9.4\nls\u8\sasv9.cfg"

/* Update Autocall macro path */

-insert sasautos "C:\src\macros"

/* Autoexec location */

-autoexec "C:\src\autoexec_win.sas"

/* Memlib Memory options */

-MEMLIB
-MEMCACHE 4
-MEMMAXSZ 32G 
-MEMSIZE MAX
-SORTSIZE 1G

/* General options */

-FONT 'Consolas' 10
-awstitle "SAS 9.4M5 UTF-8"
-noenhancededitor 
-nodmsexp
-nofmterr 
-comamid tcp 
-noxwait
-pagesize max 
-dmslogsize max
-dmsoutsize max
-errors 10
-altlog "C:\temp\altutf.log"
-work "C:\Temp\sastemp"

/* File I/O */

-BUFNO 200
-BUFSIZE 64K
-UBUFSIZE 64K
-UBUFNO 20
-IBUFNO 64
-IBUFSIZE MAX
-CBUFNO MAX
-tbufsize 512k
-catcache max
-compress yes

/* Oracle options */
-DBIDIRECTEXEC
-sql_ip_trace note

/* Email options */

-EMAILSYS SMTP 
-EMAILHOST mail.yourhost.com 

/* Stats and messages */

-STIMER
-FULLSTIMER
-MSGLEVEL I
-mautolocdisplay 
-source2 
-oplist
-nosplash

I set the memory sizes to the max, and also turn on MEMLIB and MEMCACHE which utilizes RAM for work files and caching.  Depending on your file sizes, you may or may not be able to fit everything in memory and may have to turn those options off.  The rest of the options are not directly related to your question so I thought I'd leave them in there for completeness and discussion 🙂  I actually have several config files in case I need to run with other encodings (ie Latin1) or without the memlib/memcache, etc.

hhchenfx
Barite | Level 11

So I run this proc to find the config file.

proc options option=config;run;

 

Mine is i7 3770 32G 512G SSD.

 

I play around with 

-MEMCACHE 4
-MEMMAXSZ 32G 
-MEMSIZE MAX
-SORTSIZE 1G

Clearly, making MEMSIZE MAX and MEMMAXSZ 32G

make huge improvement in data step. In this step, I simply make a big file from my existing file. It cut time form:

CPU from 19.53 to 16.66

Real time from 31.11 to 17.84

   I am not sure why real time reduce a lot while CPU reduce not as much

 

SORTSIZE benefit is not clear. Leaving at the 256MB or change to 0.5G or 1G doesn't impact datastep or Proc sort much.

I wonder why setting it to 1G doesn't help at all?

 

 

-MEMCACHE 4

Turns out to be disaster, in the "Windows Task Manager", I saw the Ram usage rocket to 30 G and stay flat there for really long time.

Data step take longer time. (with out that MEMCACHE, ram usage max about 28G).

 

Thank you so much.

 

HHCFX

 

 

 

7 data combine_file; set Ret_buy_fix_tp_notrail_h4 Ret_buy_fix_tp_notrail_h4
7 ! Ret_buy_fix_tp_notrail_h4; run;

NOTE: There were 549101 observations read from the data set WORK.RET_BUY_FIX_TP_NOTRAIL_H4.
NOTE: There were 549101 observations read from the data set WORK.RET_BUY_FIX_TP_NOTRAIL_H4.
NOTE: There were 549101 observations read from the data set WORK.RET_BUY_FIX_TP_NOTRAIL_H4.
NOTE: The data set WORK.COMBINE_FILE has 1647303 observations and 568 variables.
NOTE: DATA statement used (Total process time):
real time 17.84 seconds
cpu time 16.66 seconds


8
9
10 proc sort data=combine_file ; by decending rate Q_date tradetime;run;

NOTE: There were 1647303 observations read from the data set WORK.COMBINE_FILE.
NOTE: The data set WORK.COMBINE_FILE has 1647303 observations and 568 variables.
NOTE: PROCEDURE SORT used (Total process time):
real time 38.22 seconds
cpu time 32.27 seconds

SimonDawson
SAS Employee

@hhchenfx did you also introduce the File I/O options @DaveHorne had in his configuration?

For simple datasteps if you are not hitting errors mentioning being out of memory that just adjusting the memory limit by itself would not affect the runtime.

Its more likely your improved datastep run times came from improved I/O tuning rather than increasing the memory limit.

hhchenfx
Barite | Level 11

All I did is to change these 4 things.

I didn't touch the I/O at all.

My SAS is running now so I will try the I/O later.

 

DaveHorne
SAS Employee

@hhchenfx seeing the log and that the data set has 568 variables, that's probably the biggest impact on your performance in this particular case (especially in the sort step).  If you don't need all those variables in your analysis, then keeping (or dropping) variables in the data step can help. 

 

If you do need to keep all the variables, what are their total length (observation length)?  You can get the observation length from proc contents and bump the bufsize value (in my config I have the bufsize at 64K but you could definitely go higher so that you can fit more observations into a page of memory).

 

Since you have a fast processor, try turning on the compress option as well.  That may not speed up processing a great deal but it should save space for data sets stored on disk (and save I/O reading them).

 

Closing the time difference between CPU and real time is the goal; keep the cpu busier than the disk drive (I/O).

hhchenfx
Barite | Level 11

Thank you for your suggestion.

It is the first time I look into this matter!

 

Below is the content of my data I normally work with. 

 

So what do you think the right Bufsize for me. By the way, what is the cost of bumping Bufsize too hight?

 

Thank you,

 

HHCFX

 

SAS Output

Data Set Name INDI2018.RET_BUY_FIX_TP_NOTRAIL_H4 Observations 549101
Member Type DATA Variables 568
Engine V9 Indexes 0
Created Monday, February 05, 2018 02:53:54 AM Observation Length 4544
Last Modified Monday, February 05, 2018 02:53:54 AM Deleted Observations 0
Protection   Compressed NO

SAS Output

Engine/Host Dependent Information
Data Set Page Size 16384
Number of Data Set Pages 183038
First Data Page 5
Max Obs per Page 3
Obs in First Data Page 3
ChrisNZ
Tourmaline | Level 20

Regarding bufsize, here is a link that might interest you.

https://communities.sas.com/t5/Base-SAS-Programming/Using-bufsize-and-bufno/m-p/401170#M97302

 

Regarding compressing such wide tables,I recommend you try the SPDE engine with binary compression. You'll probably reduce the table size to a few percent points of the uncompressed size (i.e. > 90% compression ratio).

 

 

 

drb4511
Obsidian | Level 7

Is there a way to have SAS recognize an increased MEMSIZE for the 32bit version of SAS? I have plenty of RAM and have set up the config file the way you suggested and that's the only thing that is defaulting backwards (it will let me have 4GB of RAM).

ChrisNZ
Tourmaline | Level 20

@drb4511

2^32 = 4,294,967,296 = 4 GB.
That's a limitation of the architecture, and one reason the world has moved to 64 bits.

 

drb4511
Obsidian | Level 7

Thanks!

ChrisNZ
Tourmaline | Level 20

@DaveHorne Do you actually use MEMCACHE Dave? I find it very inconvenient to use.

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 19 replies
  • 7901 views
  • 2 likes
  • 7 in conversation