Apparently SAS is not coping well with my very large matrix set-up here. Is there a way to do this computation more efficiently?
B = t(X)*diag(w)*X;
ERROR: (execution) Unable to allocate sufficient memory. At least 2147483647 more bytes required.
operation : DIAG at line 8841 column 14
operands : w
w 365703 rows 1 col (numeric)
Thanks,
Raphael
Yes, see this tip on how to avoid multiplying by a large diagonal matrix. The second half of the article shows how to use ordinary elemtnwise multiplication to avoid forming the large matrix.
The first challenge is understanding your computing environment. What machine type are you running on processors/memory OS version SAS version?
As SAS is having a lot of memory options, how are these being set? That are you environment limitation as set by you..
There is one row w: of 370k numeric (each is occupying 8 bytes) that is 2Mb depending of the size of X you are needing that size being multiplied.
How big is your resulting request? The diag function will create a diagonal matric needing space for 2Mb*2Mb = 4Tb. Well you do not need to wait another 7! days for that.
http://blogs.sas.com/content/iml/2014/04/28/how-much-ram-do-i-need-to-store-that-matrix.html
Yes, see this tip on how to avoid multiplying by a large diagonal matrix. The second half of the article shows how to use ordinary elemtnwise multiplication to avoid forming the large matrix.
Thank you all! Just as I suspected, there is a more efficient way.
Using the following options to make your available memory bigger.
PROC IML SYMSIZE=n1 WORKSIZE=(n2) ;
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
Learn how to run multiple linear regression models with and without interactions, presented by SAS user Alex Chaplin.
Find more tutorials on the SAS Users YouTube channel.