Statistical programming, matrix languages, and more

Working on big matrix with proc iml

Accepted Solution Solved
Reply
New Contributor
Posts: 2
Accepted Solution

Working on big matrix with proc iml

Hello everyone,

 

I am running into a problem when manipulating an enormous matrix with proc iml. I am on SAS 9.4 TS 1M1. My codes are as followed:

 

proc iml;

use dnet_new_1;
read all var{director_id connected_director_id yearb yeare} into x;
close dnet_new_1;

xrow=nrow(x);

z=shape({.},xrow,5);
z[,1:4]=x;
z[1,5]=1;

do i=2 to xrow;
if x[i,1]^=x[i-1,1] | x[i,2]^=x[i-1,2] then z[i,5]=1;
else if x[i,3]-x[i-1,4]<=1 then z[i,5]=z[i-1,5];
else z[i,5]=z[i-1,5]+1;
end;

create dnet_new2_1 from z;
append from z;
close dnet_new2_1;
quit;

 

The matrix has 59,248,496 rows and 5 columns. I've already used the MEMSIZE option to allocate all possible system memory. It reads the following after I run "proc options option=memsize value;":

 

Option Value Information For SAS Option MEMSIZE
Value: 33523975680
Scope: SAS Session
How option value set: SAS Session Startup Command Line

 

Nevertheless, I receive the following error message: "ERROR: (execution) Unable to allocate sufficient memory. At least 2147483647 more bytes required. ".

 

Any advice will be appreciated. 


Accepted Solutions
Solution
‎01-07-2018 10:10 PM
SAS Super FREQ
Posts: 4,023

Re: Working on big matrix with proc iml

Two good references for computing with big matrices are 

The first reference tells you that a 60 million x 5 matrix requires about 2.2 GB of RAM.

 

The second tells you that on Windows PCs, SAS/IML 14.1 (shipped with SAS 9.4M3) supports matrices up to 2^31 elements (about 2  billion elements or 16GB). However, you are running SAS 9.4M1.  On that release (for Windows), any one matrix is limited to 2GB. Your matrix exceeds 2 GB.

 

Your computation is only using one row at a time (and a LAG value) so you can easily do this computation in the DATA step. 

 

If you want to stay in in SAS/IML, you can read the data in chunks (for example, 30 million at a time), as shown in the third reference.

HOWEVER, THE EASIEST way to proceed is to let z = j(xrow, 1, .), rather tha embed x into the first 4 columns of z. You can then compute the elements of z (formerly the 5th column) as a vector. Write z to a data set and then merge it with the original data.

View solution in original post


All Replies
Solution
‎01-07-2018 10:10 PM
SAS Super FREQ
Posts: 4,023

Re: Working on big matrix with proc iml

Two good references for computing with big matrices are 

The first reference tells you that a 60 million x 5 matrix requires about 2.2 GB of RAM.

 

The second tells you that on Windows PCs, SAS/IML 14.1 (shipped with SAS 9.4M3) supports matrices up to 2^31 elements (about 2  billion elements or 16GB). However, you are running SAS 9.4M1.  On that release (for Windows), any one matrix is limited to 2GB. Your matrix exceeds 2 GB.

 

Your computation is only using one row at a time (and a LAG value) so you can easily do this computation in the DATA step. 

 

If you want to stay in in SAS/IML, you can read the data in chunks (for example, 30 million at a time), as shown in the third reference.

HOWEVER, THE EASIEST way to proceed is to let z = j(xrow, 1, .), rather tha embed x into the first 4 columns of z. You can then compute the elements of z (formerly the 5th column) as a vector. Write z to a data set and then merge it with the original data.

New Contributor
Posts: 2

Re: Working on big matrix with proc iml

Thank you Rick for responding to my very first post to the SAS community. Your advice and blogs are so helpful. I really appreciate that.

☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
  • 2 replies
  • 222 views
  • 0 likes
  • 2 in conversation