turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Find a Community

Topic Options

- RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

01-07-2018 03:10 PM

Hello everyone,

I am running into a problem when manipulating an enormous matrix with proc iml. I am on SAS 9.4 TS 1M1. My codes are as followed:

proc iml;

use dnet_new_1;

read all var{director_id connected_director_id yearb yeare} into x;

close dnet_new_1;

xrow=nrow(x);

z=shape({.},xrow,5);

z[,1:4]=x;

z[1,5]=1;

do i=2 to xrow;

if x[i,1]^=x[i-1,1] | x[i,2]^=x[i-1,2] then z[i,5]=1;

else if x[i,3]-x[i-1,4]<=1 then z[i,5]=z[i-1,5];

else z[i,5]=z[i-1,5]+1;

end;

create dnet_new2_1 from z;

append from z;

close dnet_new2_1;

quit;

The matrix has 59,248,496 rows and 5 columns. I've already used the MEMSIZE option to allocate all possible system memory. It reads the following after I run "proc options option=memsize value;":

Option Value Information For SAS Option MEMSIZE

Value: 33523975680

Scope: SAS Session

How option value set: SAS Session Startup Command Line

Nevertheless, I receive the following error message: "ERROR: (execution) Unable to allocate sufficient memory. At least 2147483647 more bytes required. ".

Any advice will be appreciated.

Accepted Solutions

Solution

01-07-2018
10:10 PM

- Mark as New
- Bookmark
- Subscribe
- RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to fengyibj

01-07-2018 07:19 PM

Two good references for computing with big matrices are

- How much RAM do I need to store that matrix?
- Large matrices in SAS/IML 14.1
- Reading big data in the SAS/IML language

The first reference tells you that a 60 million x 5 matrix requires about 2.2 GB of RAM.

The second tells you that on Windows PCs, SAS/IML 14.1 (shipped with SAS 9.4M3) supports matrices up to 2^31 elements (about 2 billion elements or 16GB). However, you are running SAS 9.4M1. On that release (for Windows), any one matrix is limited to 2GB. Your matrix exceeds 2 GB.

Your computation is only using one row at a time (and a LAG value) so you can easily do this computation in the DATA step.

If you want to stay in in SAS/IML, you can read the data in chunks (for example, 30 million at a time), as shown in the third reference.

HOWEVER, THE EASIEST way to proceed is to let z = j(xrow, 1, .), rather tha embed x into the first 4 columns of z. You can then compute the elements of z (formerly the 5th column) as a vector. Write z to a data set and then merge it with the original data.

All Replies

Solution

01-07-2018
10:10 PM

- Mark as New
- Bookmark
- Subscribe
- RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to fengyibj

01-07-2018 07:19 PM

Two good references for computing with big matrices are

- How much RAM do I need to store that matrix?
- Large matrices in SAS/IML 14.1
- Reading big data in the SAS/IML language

The first reference tells you that a 60 million x 5 matrix requires about 2.2 GB of RAM.

The second tells you that on Windows PCs, SAS/IML 14.1 (shipped with SAS 9.4M3) supports matrices up to 2^31 elements (about 2 billion elements or 16GB). However, you are running SAS 9.4M1. On that release (for Windows), any one matrix is limited to 2GB. Your matrix exceeds 2 GB.

Your computation is only using one row at a time (and a LAG value) so you can easily do this computation in the DATA step.

If you want to stay in in SAS/IML, you can read the data in chunks (for example, 30 million at a time), as shown in the third reference.

HOWEVER, THE EASIEST way to proceed is to let z = j(xrow, 1, .), rather tha embed x into the first 4 columns of z. You can then compute the elements of z (formerly the 5th column) as a vector. Write z to a data set and then merge it with the original data.

- Mark as New
- Bookmark
- Subscribe
- RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Rick_SAS

01-07-2018 10:14 PM

Thank you Rick for responding to my very first post to the SAS community. Your advice and blogs are so helpful. I really appreciate that.