Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Options

- RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page

🔒 This topic is **solved** and **locked**.
Need further help from the community? Please
sign in and ask a **new** question.

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

Posted 01-07-2018 03:10 PM
(1445 views)

Hello everyone,

I am running into a problem when manipulating an enormous matrix with proc iml. I am on SAS 9.4 TS 1M1. My codes are as followed:

proc iml;

use dnet_new_1;

read all var{director_id connected_director_id yearb yeare} into x;

close dnet_new_1;

xrow=nrow(x);

z=shape({.},xrow,5);

z[,1:4]=x;

z[1,5]=1;

do i=2 to xrow;

if x[i,1]^=x[i-1,1] | x[i,2]^=x[i-1,2] then z[i,5]=1;

else if x[i,3]-x[i-1,4]<=1 then z[i,5]=z[i-1,5];

else z[i,5]=z[i-1,5]+1;

end;

create dnet_new2_1 from z;

append from z;

close dnet_new2_1;

quit;

The matrix has 59,248,496 rows and 5 columns. I've already used the MEMSIZE option to allocate all possible system memory. It reads the following after I run "proc options option=memsize value;":

Option Value Information For SAS Option MEMSIZE

Value: 33523975680

Scope: SAS Session

How option value set: SAS Session Startup Command Line

Nevertheless, I receive the following error message: "ERROR: (execution) Unable to allocate sufficient memory. At least 2147483647 more bytes required. ".

Any advice will be appreciated.

1 ACCEPTED SOLUTION

Accepted Solutions

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

Two good references for computing with big matrices are

- How much RAM do I need to store that matrix?
- Large matrices in SAS/IML 14.1
- Reading big data in the SAS/IML language

The first reference tells you that a 60 million x 5 matrix requires about 2.2 GB of RAM.

The second tells you that on Windows PCs, SAS/IML 14.1 (shipped with SAS 9.4M3) supports matrices up to 2^31 elements (about 2 billion elements or 16GB). However, you are running SAS 9.4M1. On that release (for Windows), any one matrix is limited to 2GB. Your matrix exceeds 2 GB.

Your computation is only using one row at a time (and a LAG value) so you can easily do this computation in the DATA step.

If you want to stay in in SAS/IML, you can read the data in chunks (for example, 30 million at a time), as shown in the third reference.

HOWEVER, THE EASIEST way to proceed is to let z = j(xrow, 1, .), rather tha embed x into the first 4 columns of z. You can then compute the elements of z (formerly the 5th column) as a vector. Write z to a data set and then merge it with the original data.

2 REPLIES 2

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

Two good references for computing with big matrices are

- How much RAM do I need to store that matrix?
- Large matrices in SAS/IML 14.1
- Reading big data in the SAS/IML language

The first reference tells you that a 60 million x 5 matrix requires about 2.2 GB of RAM.

The second tells you that on Windows PCs, SAS/IML 14.1 (shipped with SAS 9.4M3) supports matrices up to 2^31 elements (about 2 billion elements or 16GB). However, you are running SAS 9.4M1. On that release (for Windows), any one matrix is limited to 2GB. Your matrix exceeds 2 GB.

Your computation is only using one row at a time (and a LAG value) so you can easily do this computation in the DATA step.

If you want to stay in in SAS/IML, you can read the data in chunks (for example, 30 million at a time), as shown in the third reference.

HOWEVER, THE EASIEST way to proceed is to let z = j(xrow, 1, .), rather tha embed x into the first 4 columns of z. You can then compute the elements of z (formerly the 5th column) as a vector. Write z to a data set and then merge it with the original data.

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

**SAS Innovate 2025** is scheduled for May 6-9 in Orlando, FL. Sign up to be **first to learn** about the agenda and registration!

Multiple Linear Regression in SAS

Learn how to run multiple linear regression models with and without interactions, presented by SAS user Alex Chaplin.

Find more tutorials on the SAS Users YouTube channel.