topic Re: Swapping to disk when MEMMAXSZ cap is reached in SAS Programming

Swapping to disk when MEMMAXSZ cap is reached

Martin1 — Wed, 06 Sep 2017 14:19:01 GMT

I have just started experimenting with one of the in memory options for SAS. The in memory libname.

I start by setting my MEMMAXSZ=2147483648 and then setting MEMSIZE = 4294967296.

Then I declare the libname like:

libname turbo "c:\inmem" memlib;

Then I try loading a dataset of approximately 3 GB into the “turbo” library. This could be like

Data turbo.TEST_DATA_MARHA;

Set big_data_3_gb;

Run;

This means that we will exceed the allocated RAM size. This results to my surprise in the following error:

ERROR: Insufficient space in file TURBO.TEST_DATA_MARHA.DATA.

ERROR: File TURBO.TEST_DATA_MARHA.DATA is damaged. I/O processing did not complete.

I was actually expecting that SAS would swap to the diskpath ”c:\inmem” and finalize the dataset here. Now my question to you guys – is it possible to do so? (I know that it will hurt performance and so on)

If this is not possible – why do I have to give it a path when declaring the libname?

Im running 9.3 TS1M2 on a server.

Best regards

Martin

Re: Swapping to disk when MEMMAXSZ cap is reached

ChrisNZ — Fri, 08 Sep 2017 00:00:17 GMT

The path you give ( "c:\inmem" in your case) is a dummy. It is not used.

So the dataset must fit in the space you define with MEMMAXSZ.

Note that REALMEMSIZE (the space used by procedures) + MEMMAXSZ (the space used by RAM libraries) must fit side by side in MEMSIZE.

And REALMEMSIZE must be larger than SORTSIZE or SUMSIZE (the space used by some procedures like proc means or proc sort).

If you want to read a lot more about this, click on the link below, a section is dedicated to in-memory data allocation (SASFILE, MEMLIB, MEMCACHE, etc).

Re: Swapping to disk when MEMMAXSZ cap is reached

ChrisNZ — Fri, 08 Sep 2017 00:07:06 GMT

As for splitting the dataset, the V9 engine doesn't do this.

The SPDE engine does though.

But you can't use this engine and the memlib option on the same libname statement.

A workaround is to define a RAM disk in the OS and create a multi-path spde library in SAS.

Also, try compressing your data, it might just fit in 2GB...

Re: Swapping to disk when MEMMAXSZ cap is reached

Martin1 — Mon, 11 Sep 2017 07:09:50 GMT

Sadly this is not the perfect solution, but that is not on you - thank you for you help claryfying the problem and giving me an alternative.

Good day to you 🙂