Hello everyone,
I am running into a problem when manipulating an enormous matrix with proc iml. I am on SAS 9.4 TS 1M1. My codes are as followed:
proc iml;
use dnet_new_1;
read all var{director_id connected_director_id yearb yeare} into x;
close dnet_new_1;
xrow=nrow(x);
z=shape({.},xrow,5);
z[,1:4]=x;
z[1,5]=1;
do i=2 to xrow;
if x[i,1]^=x[i-1,1] | x[i,2]^=x[i-1,2] then z[i,5]=1;
else if x[i,3]-x[i-1,4]<=1 then z[i,5]=z[i-1,5];
else z[i,5]=z[i-1,5]+1;
end;
create dnet_new2_1 from z;
append from z;
close dnet_new2_1;
quit;
The matrix has 59,248,496 rows and 5 columns. I've already used the MEMSIZE option to allocate all possible system memory. It reads the following after I run "proc options option=memsize value;":
Option Value Information For SAS Option MEMSIZE
Value: 33523975680
Scope: SAS Session
How option value set: SAS Session Startup Command Line
Nevertheless, I receive the following error message: "ERROR: (execution) Unable to allocate sufficient memory. At least 2147483647 more bytes required. ".
Any advice will be appreciated.
Two good references for computing with big matrices are
The first reference tells you that a 60 million x 5 matrix requires about 2.2 GB of RAM.
The second tells you that on Windows PCs, SAS/IML 14.1 (shipped with SAS 9.4M3) supports matrices up to 2^31 elements (about 2 billion elements or 16GB). However, you are running SAS 9.4M1. On that release (for Windows), any one matrix is limited to 2GB. Your matrix exceeds 2 GB.
Your computation is only using one row at a time (and a LAG value) so you can easily do this computation in the DATA step.
If you want to stay in in SAS/IML, you can read the data in chunks (for example, 30 million at a time), as shown in the third reference.
HOWEVER, THE EASIEST way to proceed is to let z = j(xrow, 1, .), rather tha embed x into the first 4 columns of z. You can then compute the elements of z (formerly the 5th column) as a vector. Write z to a data set and then merge it with the original data.
Two good references for computing with big matrices are
The first reference tells you that a 60 million x 5 matrix requires about 2.2 GB of RAM.
The second tells you that on Windows PCs, SAS/IML 14.1 (shipped with SAS 9.4M3) supports matrices up to 2^31 elements (about 2 billion elements or 16GB). However, you are running SAS 9.4M1. On that release (for Windows), any one matrix is limited to 2GB. Your matrix exceeds 2 GB.
Your computation is only using one row at a time (and a LAG value) so you can easily do this computation in the DATA step.
If you want to stay in in SAS/IML, you can read the data in chunks (for example, 30 million at a time), as shown in the third reference.
HOWEVER, THE EASIEST way to proceed is to let z = j(xrow, 1, .), rather tha embed x into the first 4 columns of z. You can then compute the elements of z (formerly the 5th column) as a vector. Write z to a data set and then merge it with the original data.
Thank you Rick for responding to my very first post to the SAS community. Your advice and blogs are so helpful. I really appreciate that.
Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.
If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website.
Learn how to run multiple linear regression models with and without interactions, presented by SAS user Alex Chaplin.
Find more tutorials on the SAS Users YouTube channel.