topic RAM usages // MEMSIZE SORTSIZE in SAS Programming

RAM usages // MEMSIZE SORTSIZE

hhchenfx — Thu, 22 Feb 2018 03:04:01 GMT

Hi Everyone,

I have a PC with 32GB ram.

I wonder if the setting I got below maximize the use of Ram yet?

If not, how can I change the setting?

Thank you,

HHCFX

Group=MEMORY

SORTSIZE=268435456

Size parameter for sort

SUMSIZE=0 Upper limit for data-dependent memory usage during summarization

MAXMEMQUERY=0 Maximum amount of memory returned when inquiring as to available space

MEMBLKSZ=16777216 Size of memory blocks allocated to support MEMLIB and MEMCACHE options.

MEMMAXSZ=2147483648

Maximum amount of memory allocated to support MEMLIB and MEMCACHE options.

LOADMEMSIZE=0 Suggested memory limit for loaded SAS executables

MEMSIZE=2147483648

Specifies the limit on the total amount of memory to be used by the SAS

System

REALMEMSIZE=0 Limit on the total amount of real memory to be used by the SAS System

Re: RAM usages

LinusH — Wed, 21 Feb 2018 07:16:46 GMT

No you are not.
Increase MEMSIZE and SORTSIZE options.

Re: RAM usages

SASKiwi — Wed, 21 Feb 2018 08:11:04 GMT

Usually you should only increase the MEMSIZE / SORTSIZE options' values from the default if your SAS jobs are running out of memory for processing or sorting. Increasing these won't necessarily reduce job run times.

Re: RAM usages

hhchenfx — Wed, 21 Feb 2018 16:03:46 GMT

So I see 1 way to change MEMSIZE is: Start -> Run -> sas.exe -memsize 30G

Also the OPTIONS way but I try this one and it doesn't work.

proc options memsize=3047483648;
run;

How can I fix that proc?

Thank you.

HHCFX

Re: RAM usages

DaveHorne — Wed, 21 Feb 2018 21:52:49 GMT

To make things easier, you may want to consider having your own custom SAS config file. That way you can tweak settings quickly, especially for testing (edit the file and restart SAS to pick up any changes). My main SAS workstation has an i7-6700 processor, 32GB RAM, and a 512GB SSD (SanDisk X400) and I've been pretty happy with it for development work.

My SAS desktop shortcut is:

"C:\Program Files\SASHome\SASFoundation\9.4\sas.exe" -CONFIG "C:\users\dave\sas94.cfg"

In the sas94.cfg file I include the default shipping UTF-8 config followed by my custom settings:

/*
 * sas94.cfg
 *
 * This config file sets up the SAS environment for Dave - UTF8.
 *
 */

/* Include default SAS config file */

-config "C:\Program Files\SASHome\SASFoundation\9.4\nls\u8\sasv9.cfg"

/* Update Autocall macro path */

-insert sasautos "C:\src\macros"

/* Autoexec location */

-autoexec "C:\src\autoexec_win.sas"

/* Memlib Memory options */

-MEMLIB
-MEMCACHE 4
-MEMMAXSZ 32G 
-MEMSIZE MAX
-SORTSIZE 1G

/* General options */

-FONT 'Consolas' 10
-awstitle "SAS 9.4M5 UTF-8"
-noenhancededitor 
-nodmsexp
-nofmterr 
-comamid tcp 
-noxwait
-pagesize max 
-dmslogsize max
-dmsoutsize max
-errors 10
-altlog "C:\temp\altutf.log"
-work "C:\Temp\sastemp"

/* File I/O */

-BUFNO 200
-BUFSIZE 64K
-UBUFSIZE 64K
-UBUFNO 20
-IBUFNO 64
-IBUFSIZE MAX
-CBUFNO MAX
-tbufsize 512k
-catcache max
-compress yes

/* Oracle options */
-DBIDIRECTEXEC
-sql_ip_trace note

/* Email options */

-EMAILSYS SMTP 
-EMAILHOST mail.yourhost.com 

/* Stats and messages */

-STIMER
-FULLSTIMER
-MSGLEVEL I
-mautolocdisplay 
-source2 
-oplist
-nosplash

I set the memory sizes to the max, and also turn on MEMLIB and MEMCACHE which utilizes RAM for work files and caching. Depending on your file sizes, you may or may not be able to fit everything in memory and may have to turn those options off. The rest of the options are not directly related to your question so I thought I'd leave them in there for completeness and discussion 🙂 I actually have several config files in case I need to run with other encodings (ie Latin1) or without the memlib/memcache, etc.

Re: RAM usages

hhchenfx — Thu, 22 Feb 2018 03:03:15 GMT

So I run this proc to find the config file.

proc options option=config;run;

Mine is i7 3770 32G 512G SSD.

I play around with

-MEMCACHE 4
-MEMMAXSZ 32G 
-MEMSIZE MAX
-SORTSIZE 1G

Clearly, making MEMSIZE MAX and MEMMAXSZ 32G

make huge improvement in data step. In this step, I simply make a big file from my existing file. It cut time form:

CPU from 19.53 to 16.66

Real time from 31.11 to 17.84

I am not sure why real time reduce a lot while CPU reduce not as much

SORTSIZE benefit is not clear. Leaving at the 256MB or change to 0.5G or 1G doesn't impact datastep or Proc sort much.

I wonder why setting it to 1G doesn't help at all?

-MEMCACHE 4

Turns out to be disaster, in the "Windows Task Manager", I saw the Ram usage rocket to 30 G and stay flat there for really long time.

Data step take longer time. (with out that MEMCACHE, ram usage max about 28G).

Thank you so much.

HHCFX

7 data combine_file; set Ret_buy_fix_tp_notrail_h4 Ret_buy_fix_tp_notrail_h4
7 ! Ret_buy_fix_tp_notrail_h4; run;

NOTE: There were 549101 observations read from the data set WORK.RET_BUY_FIX_TP_NOTRAIL_H4.
NOTE: There were 549101 observations read from the data set WORK.RET_BUY_FIX_TP_NOTRAIL_H4.
NOTE: There were 549101 observations read from the data set WORK.RET_BUY_FIX_TP_NOTRAIL_H4.
NOTE: The data set WORK.COMBINE_FILE has 1647303 observations and 568 variables.
NOTE: DATA statement used (Total process time):
real time 17.84 seconds
cpu time 16.66 seconds

8
9
10 proc sort data=combine_file ; by decending rate Q_date tradetime;run;

NOTE: There were 1647303 observations read from the data set WORK.COMBINE_FILE.
NOTE: The data set WORK.COMBINE_FILE has 1647303 observations and 568 variables.
NOTE: PROCEDURE SORT used (Total process time):
real time 38.22 seconds
cpu time 32.27 seconds

Re: RAM usages

SimonDawson — Thu, 22 Feb 2018 03:46:37 GMT

@hhchenfx did you also introduce the File I/O options @DaveHorne had in his configuration?

For simple datasteps if you are not hitting errors mentioning being out of memory that just adjusting the memory limit by itself would not affect the runtime.

Its more likely your improved datastep run times came from improved I/O tuning rather than increasing the memory limit.

Re: RAM usages

hhchenfx — Thu, 22 Feb 2018 04:23:14 GMT

All I did is to change these 4 things.

I didn't touch the I/O at all.

My SAS is running now so I will try the I/O later.

Re: RAM usages // MEMSIZE SORTSIZE

ChrisNZ — Thu, 22 Feb 2018 11:01:02 GMT

You seem to be confusing MEMSIZE and REALMEMSIZE.

REALMEMSIZE should always be less than RAM (you can let SAS decide, but it can better to set it yourself).

MEMSIZE can be more as it includes paging space.

See here for a diagram.

Re: RAM usages

DaveHorne — Thu, 22 Feb 2018 14:26:40 GMT

@hhchenfx seeing the log and that the data set has 568 variables, that's probably the biggest impact on your performance in this particular case (especially in the sort step). If you don't need all those variables in your analysis, then keeping (or dropping) variables in the data step can help.

If you do need to keep all the variables, what are their total length (observation length)? You can get the observation length from proc contents and bump the bufsize value (in my config I have the bufsize at 64K but you could definitely go higher so that you can fit more observations into a page of memory).

Since you have a fast processor, try turning on the compress option as well. That may not speed up processing a great deal but it should save space for data sets stored on disk (and save I/O reading them).

Closing the time difference between CPU and real time is the goal; keep the cpu busier than the disk drive (I/O).

Re: RAM usages

hhchenfx — Thu, 22 Feb 2018 15:59:16 GMT

Thank you for your suggestion.

It is the first time I look into this matter!

Below is the content of my data I normally work with.

So what do you think the right Bufsize for me. By the way, what is the cost of bumping Bufsize too hight?

Thank you,

HHCFX

SAS Output

Data Set Name	INDI2018.RET_BUY_FIX_TP_NOTRAIL_H4	Observations	549101
Member Type	DATA	Variables	568
Engine	V9	Indexes	0
Created	Monday, February 05, 2018 02:53:54 AM	Observation Length	4544
Last Modified	Monday, February 05, 2018 02:53:54 AM	Deleted Observations	0
Protection		Compressed	NO

SAS Output

Engine/Host Dependent Information
Data Set Page Size	16384
Number of Data Set Pages	183038
First Data Page	5
Max Obs per Page	3
Obs in First Data Page	3

Re: RAM usages

ChrisNZ — Thu, 22 Feb 2018 20:04:59 GMT

Regarding bufsize, here is a link that might interest you.

https://communities.sas.com/t5/Base-SAS-Programming/Using-bufsize-and-bufno/m-p/401170#M97302

Regarding compressing such wide tables,I recommend you try the SPDE engine with binary compression. You'll probably reduce the table size to a few percent points of the uncompressed size (i.e. > 90% compression ratio).

Re: RAM usages

drb4511 — Fri, 18 May 2018 02:19:55 GMT

Is there a way to have SAS recognize an increased MEMSIZE for the 32bit version of SAS? I have plenty of RAM and have set up the config file the way you suggested and that's the only thing that is defaulting backwards (it will let me have 4GB of RAM).

Re: RAM usages

ChrisNZ — Fri, 18 May 2018 02:52:24 GMT

@DaveHorne Do you actually use MEMCACHE Dave? I find it very inconvenient to use.

Re: RAM usages

ChrisNZ — Fri, 18 May 2018 03:08:10 GMT

@drb4511

2^32 = 4,294,967,296 = 4 GB.
That's a limitation of the architecture, and one reason the world has moved to 64 bits.

Re: RAM usages

DaveHorne — Fri, 18 May 2018 12:56:15 GMT

@ChrisNZ, yes, I use it quite a lot. I run over a thousand jobs a day scheduled on Windows servers and at least 95% of them use the MEMLIB/MEMCACHE options (enabled in our "default" config). Being able to run in-memory saves a lot of I/O processing. I even tested it with pointing WORK to a RAMDISK and the MEMLIB option was still faster (I imagine it's been optimized for that). A few of our jobs use upwards of 96-128GB of RAM. Most fit inside of 48GB or less. It's all scripted so it's fairly easy to turn on/off as needed (I point to a different config without the memory options in those few cases).

I do have MEMLIB/MEMCACHE turned on in the default config on my workstation (it's 32GB RAM, i7-6700, 500GB SSD storage, running SAS 9.4M5, Windows10). I tend to do smaller unit testing on the workstation and run the "big" jobs on the servers. I don't think the MEMLIB options are a huge benefit on my PC, especially with a fast processor and SSD storage (but I like pushing it to the limit if I can :). Let me know if you'd like more details.

Re: RAM usages

drb4511 — Fri, 18 May 2018 15:42:43 GMT

Thanks!

Re: RAM usages

ChrisNZ — Sat, 19 May 2018 04:48:12 GMT

@DaveHorne

I wonder why you use MEMLIB/MEMCACHE as if they were a conjoined pair.

Here is I how I see the SAS in-memory data options:

- I often use SASFILE for read-only data as it doesn't require changing tables names or defining a new library

- MEMLIB is less used; it is useful for dynamic data or for creating/processing temporary tables.

- MEMCACHE is less useful, and I avoid it because if one is not vigilant, MEMCACHE can lock RAM space until the SAS session is teminated.

Do you have a different experience with MEMCACHE?

MEMLIB speeds are typically several times faster than SSD, but the newfangled M2 drives may just make MEMLIB redundant. I haven't tested this yet.

Maybe should this discussion not be polluting this thread.

Re: RAM usages

DaveHorne — Tue, 22 May 2018 15:29:26 GMT

Hi @ChrisNZ, I didn't mean to imply they were a conjoined pair, I tend to use them both to maximize memory usage (i.e. on servers where we have a large amount of RAM).

With jobs that have really large data sets or work files we do sprinkle proc deletes in the code to free up RAM as the job runs. And sometimes we skip the in-memory options for large data over 200GB. (I've had some success with SGIO and multi-gigabyte data sets)

I agree with you, the latest M.2 SSD's are very promising and I can wait to get my hands on one to try out! Definitely would love to hear your experience if you get one to test with.

Cheers,
Dave

Re: RAM usages

ChrisNZ — Tue, 22 May 2018 21:45:14 GMT

@DaveHorne I'll let you know if can free the time to try M.2 drives, and please do likewise. 🙂

Yes SGIO is definitely an option to be considered for speeding up sequential access on large data sets.

Here is a benchmark in case you are interested in hard(ish) figures.