DATA Step, Macro, Functions and more

RAM usages // MEMSIZE SORTSIZE

Reply
Super Contributor
Posts: 503

RAM usages // MEMSIZE SORTSIZE

[ Edited ]

Hi Everyone,

I have a PC with 32GB ram.

I wonder if the setting I got below maximize the use of Ram yet?

If not, how can I change the setting?

Thank you,

HHCFX

 

Group=MEMORY

 SORTSIZE=268435456

                   Size parameter for sort

 SUMSIZE=0         Upper limit for data-dependent memory usage during summarization

 MAXMEMQUERY=0     Maximum amount of memory returned when inquiring as to available space

 MEMBLKSZ=16777216 Size of memory blocks allocated to support MEMLIB and MEMCACHE options.

 MEMMAXSZ=2147483648

                   Maximum amount of memory allocated to support MEMLIB and MEMCACHE options.

 LOADMEMSIZE=0     Suggested memory limit for loaded SAS executables

 MEMSIZE=2147483648

                   Specifies the limit on the total amount of memory to be used by the SAS

                   System

 REALMEMSIZE=0     Limit on the total amount of real memory to be used by the SAS System

Super User
Posts: 5,826

Re: RAM usages

No you are not.
Increase MEMSIZE and SORTSIZE options.
Data never sleeps
Super User
Posts: 3,767

Re: RAM usages

Usually you should only increase the MEMSIZE / SORTSIZE options' values from the default if your SAS jobs are running out of memory for processing or sorting. Increasing these won't necessarily reduce job run times.

Super Contributor
Posts: 503

Re: RAM usages

So I see 1 way to change MEMSIZE is: Start -> Run -> sas.exe -memsize 30G

 

Also the OPTIONS way but I try this one and it doesn't work.

 

proc options memsize=3047483648;
run;

 

How can I fix that proc?

 

Thank you.

 

HHCFX

SAS Employee
Posts: 19

Re: RAM usages

To make things easier, you may want to consider having your own custom SAS config file.  That way you can tweak settings quickly, especially for testing (edit the file and restart SAS to pick up any changes).  My main SAS workstation has an i7-6700 processor, 32GB RAM, and a 512GB SSD (SanDisk X400) and I've been pretty happy with it for development work.

 

My SAS desktop shortcut is:

"C:\Program Files\SASHome\SASFoundation\9.4\sas.exe" -CONFIG "C:\users\dave\sas94.cfg"

In the sas94.cfg file I include the default shipping UTF-8 config followed by my custom settings:

 

/*
 * sas94.cfg
 *
 * This config file sets up the SAS environment for Dave - UTF8.
 *
 */

/* Include default SAS config file */

-config "C:\Program Files\SASHome\SASFoundation\9.4\nls\u8\sasv9.cfg"

/* Update Autocall macro path */

-insert sasautos "C:\src\macros"

/* Autoexec location */

-autoexec "C:\src\autoexec_win.sas"

/* Memlib Memory options */

-MEMLIB
-MEMCACHE 4
-MEMMAXSZ 32G 
-MEMSIZE MAX
-SORTSIZE 1G

/* General options */

-FONT 'Consolas' 10
-awstitle "SAS 9.4M5 UTF-8"
-noenhancededitor 
-nodmsexp
-nofmterr 
-comamid tcp 
-noxwait
-pagesize max 
-dmslogsize max
-dmsoutsize max
-errors 10
-altlog "C:\temp\altutf.log"
-work "C:\Temp\sastemp"

/* File I/O */

-BUFNO 200
-BUFSIZE 64K
-UBUFSIZE 64K
-UBUFNO 20
-IBUFNO 64
-IBUFSIZE MAX
-CBUFNO MAX
-tbufsize 512k
-catcache max
-compress yes

/* Oracle options */
-DBIDIRECTEXEC
-sql_ip_trace note

/* Email options */

-EMAILSYS SMTP 
-EMAILHOST mail.yourhost.com 

/* Stats and messages */

-STIMER
-FULLSTIMER
-MSGLEVEL I
-mautolocdisplay 
-source2 
-oplist
-nosplash

I set the memory sizes to the max, and also turn on MEMLIB and MEMCACHE which utilizes RAM for work files and caching.  Depending on your file sizes, you may or may not be able to fit everything in memory and may have to turn those options off.  The rest of the options are not directly related to your question so I thought I'd leave them in there for completeness and discussion :-)  I actually have several config files in case I need to run with other encodings (ie Latin1) or without the memlib/memcache, etc.

Super Contributor
Posts: 503

Re: RAM usages

Posted in reply to DaveHorne

So I run this proc to find the config file.

proc options option=config;run;

 

Mine is i7 3770 32G 512G SSD.

 

I play around with 

-MEMCACHE 4
-MEMMAXSZ 32G 
-MEMSIZE MAX
-SORTSIZE 1G

Clearly, making MEMSIZE MAX and MEMMAXSZ 32G

make huge improvement in data step. In this step, I simply make a big file from my existing file. It cut time form:

CPU from 19.53 to 16.66

Real time from 31.11 to 17.84

   I am not sure why real time reduce a lot while CPU reduce not as much

 

SORTSIZE benefit is not clear. Leaving at the 256MB or change to 0.5G or 1G doesn't impact datastep or Proc sort much.

I wonder why setting it to 1G doesn't help at all?

 

 

-MEMCACHE 4

Turns out to be disaster, in the "Windows Task Manager", I saw the Ram usage rocket to 30 G and stay flat there for really long time.

Data step take longer time. (with out that MEMCACHE, ram usage max about 28G).

 

Thank you so much.

 

HHCFX

 

 

 

7 data combine_file; set Ret_buy_fix_tp_notrail_h4 Ret_buy_fix_tp_notrail_h4
7 ! Ret_buy_fix_tp_notrail_h4; run;

NOTE: There were 549101 observations read from the data set WORK.RET_BUY_FIX_TP_NOTRAIL_H4.
NOTE: There were 549101 observations read from the data set WORK.RET_BUY_FIX_TP_NOTRAIL_H4.
NOTE: There were 549101 observations read from the data set WORK.RET_BUY_FIX_TP_NOTRAIL_H4.
NOTE: The data set WORK.COMBINE_FILE has 1647303 observations and 568 variables.
NOTE: DATA statement used (Total process time):
real time 17.84 seconds
cpu time 16.66 seconds


8
9
10 proc sort data=combine_file ; by decending rate Q_date tradetime;run;

NOTE: There were 1647303 observations read from the data set WORK.COMBINE_FILE.
NOTE: The data set WORK.COMBINE_FILE has 1647303 observations and 568 variables.
NOTE: PROCEDURE SORT used (Total process time):
real time 38.22 seconds
cpu time 32.27 seconds

SAS Employee
Posts: 56

Re: RAM usages

@hhchenfx did you also introduce the File I/O options @DaveHorne had in his configuration?

For simple datasteps if you are not hitting errors mentioning being out of memory that just adjusting the memory limit by itself would not affect the runtime.

Its more likely your improved datastep run times came from improved I/O tuning rather than increasing the memory limit.

Super Contributor
Posts: 503

Re: RAM usages

Posted in reply to SimonDawson

All I did is to change these 4 things.

I didn't touch the I/O at all.

My SAS is running now so I will try the I/O later.

 

SAS Employee
Posts: 19

Re: RAM usages

@hhchenfx seeing the log and that the data set has 568 variables, that's probably the biggest impact on your performance in this particular case (especially in the sort step).  If you don't need all those variables in your analysis, then keeping (or dropping) variables in the data step can help. 

 

If you do need to keep all the variables, what are their total length (observation length)?  You can get the observation length from proc contents and bump the bufsize value (in my config I have the bufsize at 64K but you could definitely go higher so that you can fit more observations into a page of memory).

 

Since you have a fast processor, try turning on the compress option as well.  That may not speed up processing a great deal but it should save space for data sets stored on disk (and save I/O reading them).

 

Closing the time difference between CPU and real time is the goal; keep the cpu busier than the disk drive (I/O).

Super Contributor
Posts: 503

Re: RAM usages

Posted in reply to DaveHorne

Thank you for your suggestion.

It is the first time I look into this matter!

 

Below is the content of my data I normally work with. 

 

So what do you think the right Bufsize for me. By the way, what is the cost of bumping Bufsize too hight?

 

Thank you,

 

HHCFX

 

SAS Output

Data Set Name INDI2018.RET_BUY_FIX_TP_NOTRAIL_H4 Observations 549101
Member Type DATA Variables 568
Engine V9 Indexes 0
Created Monday, February 05, 2018 02:53:54 AM Observation Length 4544
Last Modified Monday, February 05, 2018 02:53:54 AM Deleted Observations 0
Protection   Compressed NO

SAS Output

Engine/Host Dependent Information
Data Set Page Size 16384
Number of Data Set Pages 183038
First Data Page 5
Max Obs per Page 3
Obs in First Data Page 3
PROC Star
Posts: 2,224

Re: RAM usages

Regarding bufsize, here is a link that might interest you.

https://communities.sas.com/t5/Base-SAS-Programming/Using-bufsize-and-bufno/m-p/401170#M97302

 

Regarding compressing such wide tables,I recommend you try the SPDE engine with binary compression. You'll probably reduce the table size to a few percent points of the uncompressed size (i.e. > 90% compression ratio).

 

 

 

Occasional Contributor
Posts: 11

Re: RAM usages

Posted in reply to DaveHorne

Is there a way to have SAS recognize an increased MEMSIZE for the 32bit version of SAS? I have plenty of RAM and have set up the config file the way you suggested and that's the only thing that is defaulting backwards (it will let me have 4GB of RAM).

PROC Star
Posts: 2,224

Re: RAM usages

@drb4511

2^32 = 4,294,967,296 = 4 GB.
That's a limitation of the architecture, and one reason the world has moved to 64 bits.

 

Occasional Contributor
Posts: 11

Re: RAM usages

Thanks!

PROC Star
Posts: 2,224

Re: RAM usages

Posted in reply to DaveHorne

@DaveHorne Do you actually use MEMCACHE Dave? I find it very inconvenient to use.

Ask a Question
Discussion stats
  • 19 replies
  • 363 views
  • 2 likes
  • 7 in conversation