Statistical programming, matrix languages, and more

Unable to allocate sufficient memory

Accepted Solution Solved
Reply
Contributor
Posts: 20
Accepted Solution

Unable to allocate sufficient memory

Apparently SAS is not coping well with my very large matrix set-up here. Is there a way to do this computation more efficiently?

B = t(X)*diag(w)*X;

ERROR: (execution) Unable to allocate sufficient memory. At least 2147483647 more bytes required.

operation : DIAG at line 8841 column 14

operands  : w

w 365703 rows      1 col     (numeric)

Thanks,

Raphael


Accepted Solutions
Solution
‎05-14-2015 06:37 AM
SAS Super FREQ
Posts: 3,615

Re: Unable to allocate sufficient memory

Yes, see this tip on how to avoid multiplying by a large diagonal matrix. The second half of the article shows how to use ordinary elemtnwise multiplication to avoid forming the large matrix.

View solution in original post


All Replies
Valued Guide
Posts: 3,208

Re: Unable to allocate sufficient memory

The first challenge is understanding your computing environment. What machine type are you running on processors/memory OS version SAS version?

As SAS is having a lot of memory options, how are these being set? That are you environment limitation as set by you.. 

There is one row w: of 370k numeric (each is occupying 8 bytes) that is 2Mb depending of the size of X you are needing that size being multiplied.
How big is your resulting request?    The diag function will create a diagonal matric needing space for 2Mb*2Mb  = 4Tb. Well you do not need to wait another 7! days for that. 
http://blogs.sas.com/content/iml/2014/04/28/how-much-ram-do-i-need-to-store-that-matrix.html

---->-- ja karman --<-----
Solution
‎05-14-2015 06:37 AM
SAS Super FREQ
Posts: 3,615

Re: Unable to allocate sufficient memory

Yes, see this tip on how to avoid multiplying by a large diagonal matrix. The second half of the article shows how to use ordinary elemtnwise multiplication to avoid forming the large matrix.

Contributor
Posts: 20

Re: Unable to allocate sufficient memory

Thank you all! Just as I suspected, there is a more efficient way.

Super User
Posts: 9,854

Re: Unable to allocate sufficient memory

Using the following options to make your available memory bigger.

PROC IML   SYMSIZE=n1    WORKSIZE=(n2)  ;

🔒 This topic is solved and locked.

Need further help from the community? Please ask a new question.

Discussion stats
  • 4 replies
  • 1952 views
  • 6 likes
  • 4 in conversation