Working on big matrix with proc iml

[fengyibj]

Hello everyone,

I am running into a problem when manipulating an enormous matrix with proc iml. I am on SAS 9.4 TS 1M1. My codes are as followed:

proc iml;

use dnet_new_1;
read all var{director_id connected_director_id yearb yeare} into x;
close dnet_new_1;

xrow=nrow(x);

z=shape({.},xrow,5);
z[,1:4]=x;
z[1,5]=1;

do i=2 to xrow;
if x[i,1]^=x[i-1,1] | x[i,2]^=x[i-1,2] then z[i,5]=1;
else if x[i,3]-x[i-1,4]<=1 then z[i,5]=z[i-1,5];
else z[i,5]=z[i-1,5]+1;
end;

create dnet_new2_1 from z;
append from z;
close dnet_new2_1;
quit;

The matrix has 59,248,496 rows and 5 columns. I've already used the MEMSIZE option to allocate all possible system memory. It reads the following after I run "proc options option=memsize value;":

Option Value Information For SAS Option MEMSIZE
Value: 33523975680
Scope: SAS Session
How option value set: SAS Session Startup Command Line

Nevertheless, I receive the following error message: "ERROR: (execution) Unable to allocate sufficient memory. At least 2147483647 more bytes required. ".

Any advice will be appreciated. 

[Rick_SAS · Accepted solution]

Two good references for computing with big matrices are 

•  How much RAM do I need to store that matrix?
•  Large matrices in SAS/IML 14.1
• Reading big data in the SAS/IML language

The first reference tells you that a 60 million x 5 matrix requires about 2.2 GB of RAM.

The second tells you that on Windows PCs, SAS/IML 14.1 (shipped with SAS 9.4M3) supports matrices up to 2^31 elements (about 2  billion elements or 16GB). However, you are running SAS 9.4M1.  On that release (for Windows), any one matrix is limited to 2GB. Your matrix exceeds 2 GB.

Your computation is only using one row at a time (and a LAG value) so you can easily do this computation in the DATA step. 

If you want to stay in in SAS/IML, you can read the data in chunks (for example, 30 million at a time), as shown in the third reference.

HOWEVER, THE EASIEST way to proceed is to let z = j(xrow, 1, .), rather tha embed x into the first 4 columns of z. You can then compute the elements of z (formerly the 5th column) as a vector. Write z to a data set and then merge it with the original data.

[fengyibj]

Thank you Rick for responding to my very first post to the SAS community. Your advice and blogs are so helpful. I really appreciate that.