topic SAS IML Memory Issue in SAS/IML Software and Matrix Computations

SAS IML Memory Issue

Vivian — Mon, 05 Mar 2012 11:50:28 GMT

Hi, all:

I have been a IML user for years.

However, I recently encountered a very weird problem when using IML on the linux system.

Basically I wrote a sas program for a new estimation method created with macros and IML, and then did some simulations on linux system.

The SAS IML was failed with the error message as follows:

ERROR: (execution) Unable to allocate sufficient memory. At least 3115064 more bytes required.

operation : CHOOSE at line 578 column 167

operands : _TEM1001, *LIT1083, dwn

_TEM1001 624 rows 624 cols (numeric)

*LIT1083 1 row 1 col (numeric)

dwn 624 rows 624 cols (numeric)

For the above message, the corresponding codes in my program is "dwn=choose(wn=.,0,wn);" where wn is a 624*624 matrix.

Well, the point is that I did the same simulations (same codes on same server, I promise) last Nov, but everything was fine without memory problem.

Here is the information about the sas version and the linux system I used.

Last Nov. :

NOTE: SAS (r) Proprietary Software 9.3 (TS1M0)

NOTE: This session is executing on the Linux 2.6.18-238.9.1.el5 (LIN X64) platform.

Recently:

NOTE: SAS (r) Proprietary Software 9.3 (TS1M0)

NOTE: This session is executing on the Linux 2.6.18-274.7.1.el5 (LIN X64) platform.

Does the 'out of memory' issue really result from the system difference?

Any clue?

I have been struggling with this problem for a while.

I would appreciate anyone who can help out.

Thanks,

Vivian

SAS IML Memory Issue

Rick_SAS — Mon, 05 Mar 2012 13:40:10 GMT

Strange. SAS didn't change, but you are getting different behavior. I have no idea what has changed on your system,

but remember that memory is shared among all applications. If you have a big PDF file open, an email program running, or other applications, those are all using memory, too.

However, maybe I can solve the immediate problem of getting your program to run. Instead of

dwn=choose(wn=.,0,wn);

try this:

dwn = wn; /* copy wn */

idx = loc(dwn=.); /* find missing */

if ncol(idx)>0 then dwn[idx]=0; /* replace with zeros */

The above code can be even more memory-efficient if you are willing to overwrite the original copy of wn:

idx = loc(wn=.);

if ncol(idx)>0 then wn[idx]=0; /* overwrite wn */

Of course, this might just move the out-of-memory error to another line of the program.

SAS IML Memory Issue

Vivian — Mon, 05 Mar 2012 15:03:34 GMT

Thanks a lot for your prompt reply.

As I have used SAS for many years, it is also my first time to have such problem.

I also have no idea what has been chenged in our system.

The staff at our school computer center in fact could not figure out what happened.

Regarding the out-of-memory problem, I did ever edited the codes and unfortunately, like you exppected, it moved the error to another line of the problem.

I even tried to release some memory and still, it didn't work.

That is why I have been struggling for such a log time.

Another interesting thing is that I submitted the same SAS program on a computer with window 7.

Everything works well without memory problem, even with the function CHOOSE.

So, no solution, right?

Anyway, I still appreciate your response.

Oh, by the way, as I have read through your book and started using IML Studio for a while, I have a question in mind.

Can IML Studio work together with some macro codes, like we ususally do in regular IML (PROC IML)?

Thank you very much!!

Vivian

SAS IML Memory Issue

Rick_SAS — Mon, 05 Mar 2012 15:33:22 GMT

You can contact technical support to see if they have any advice on the out-of-memory situation.

Regarding macros, SAS/IML Studio programs are not parsed by the macro facility, so it doesn't support macros in the usual way. However, you can

1) call macros within a SUBMIT/ENDSUBMIT block. These are executed in the SAS environment.

2) Use the SYMGET and SYMPUT family of functions to get or set macro variables from within a SAS/IML Studio program. For example, the following statements set a macro variable (The '@' operator is shorthand for 'submit this single SAS statement.'), then reads in the macro and assigns it to an IML variable:

@%let g=5; /* submit global stmt */

five = symgetn("g");

print five;

See p. 99 of my book for more examples.