topic Re: Unable to allocate sufficient memory in SAS/IML Software and Matrix Computations

Unable to allocate sufficient memory

R_Fraser — Thu, 14 May 2015 06:31:18 GMT

Apparently SAS is not coping well with my very large matrix set-up here. Is there a way to do this computation more efficiently?

B = t(X)*diag(w)*X;

ERROR: (execution) Unable to allocate sufficient memory. At least 2147483647 more bytes required.

operation : DIAG at line 8841 column 14

operands : w

w 365703 rows 1 col (numeric)

Thanks,

Raphael

Re: Unable to allocate sufficient memory

jakarman — Thu, 14 May 2015 09:15:20 GMT

The first challenge is understanding your computing environment. What machine type are you running on processors/memory OS version SAS version?

As SAS is having a lot of memory options, how are these being set? That are you environment limitation as set by you..

There is one row w: of 370k numeric (each is occupying 8 bytes) that is 2Mb depending of the size of X you are needing that size being multiplied.
How big is your resulting request? The diag function will create a diagonal matric needing space for 2Mb*2Mb = 4Tb. Well you do not need to wait another 7! days for that.
http://blogs.sas.com/content/iml/2014/04/28/how-much-ram-do-i-need-to-store-that-matrix.html

Re: Unable to allocate sufficient memory

Rick_SAS — Thu, 14 May 2015 10:37:14 GMT

Yes, see this tip on how to avoid multiplying by a large diagonal matrix. The second half of the article shows how to use ordinary elemtnwise multiplication to avoid forming the large matrix.

Re: Unable to allocate sufficient memory

Ksharp — Thu, 14 May 2015 13:01:55 GMT

Using the following options to make your available memory bigger.

PROC IML SYMSIZE=n1 WORKSIZE=(n2) ;

Re: Unable to allocate sufficient memory

R_Fraser — Thu, 14 May 2015 17:04:15 GMT

Thank you all! Just as I suspected, there is a more efficient way.