RAM usages

[DaveHorne]

@ChrisNZ, yes, I use it quite a lot.  I run over a thousand jobs a day scheduled on Windows servers and at least 95% of them use the MEMLIB/MEMCACHE options (enabled in our "default" config).  Being able to run in-memory saves a lot of I/O processing.  I even tested it with pointing WORK to a RAMDISK and the MEMLIB option was still faster (I imagine it's been optimized for that).  A few of our jobs use upwards of 96-128GB of RAM.  Most fit inside of 48GB or less.  It's all scripted so it's fairly easy to turn on/off as needed (I point to a different config without the memory options in those few cases).

I do have MEMLIB/MEMCACHE turned on in the default config on my workstation (it's 32GB RAM, i7-6700, 500GB SSD storage, running SAS 9.4M5, Windows10).  I tend to do smaller unit testing on the workstation and run the "big" jobs on the servers.  I don't think the MEMLIB options are a huge benefit on my PC, especially with a fast processor and SSD storage (but I like pushing it to the limit if I can :).  Let me know if you'd like more details.

[ChrisNZ]

@DaveHorne 

I wonder why you use MEMLIB/MEMCACHE as if they were a conjoined pair.

Here is I how I see the SAS in-memory data options:

- I often use SASFILE for read-only data as it doesn't require changing tables names or defining a new library

- MEMLIB is less used; it is useful for dynamic data or for creating/processing temporary tables. 

- MEMCACHE is less useful, and I avoid it because if one is not vigilant, MEMCACHE can lock RAM space until the SAS session is teminated.

Do you have a different experience with MEMCACHE?

MEMLIB speeds are typically several times faster than SSD, but the newfangled M2 drives may just make MEMLIB redundant. I haven't tested this yet.

Maybe should this discussion not be polluting this thread.

[DaveHorne]

Hi @ChrisNZ, I didn't mean to imply they were a conjoined pair, I tend to use them both to maximize memory usage (i.e. on servers where we have a large amount of RAM).

With jobs that have really large data sets or work files we do sprinkle proc deletes in the code to free up RAM as the job runs. And sometimes we skip the in-memory options for large data over 200GB. (I've had some success with SGIO and multi-gigabyte data sets)

I agree with you, the latest M.2 SSD's are very promising and I can wait to get my hands on one to try out! Definitely would love to hear your experience if you get one to test with.

Cheers,
Dave

[ChrisNZ]

@DaveHorne I'll let you know if can free the time to try M.2 drives, and please do likewise. 🙂

Yes SGIO is definitely an option to be considered for speeding up sequential access on large data sets.

Here is a benchmark in case you are interested in hard(ish) figures.

[ChrisNZ]

You seem to be confusing MEMSIZE and REALMEMSIZE.

REALMEMSIZE should always be less than RAM (you can let SAS decide, but it can better to set it yourself).

MEMSIZE can be more as it includes paging space.

See here for a diagram.