topic Re: Splitting a matrix in SAS/IML Software and Matrix Computations

Splitting a matrix

Venkibhu1 — Wed, 10 Jan 2018 11:02:31 GMT

Dear All,

The below code is not executing successfully as the matrix size is large. The program is ending up with the below error

ERROR: (execution) Unable to allocate sufficient memory.

operation : LENGTH at line 1395 column 11
operands : _TEM1001
_TEM1001 33 rows 400000 cols (character, size 80)

Is there anyway to find and split the matrix based on the size?

dm "output" clear;
data one;
array cha{*} $18. c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 c11 c12 c13 c14 c15 c16 c17 c18 c19 c20 c21 c22 c23 c24 c25 c26 c27 c28 c29 c30 c31 c32 c33;
do j = 1 to 400000;
seed = 100;
do i = 1 to hbound(cha);
rannum = ranpoi(seed,50);
if rannum = 0 then rannum = 1;
cha{i} = repeat("a",rannum);
end;
output;
end;
run;

proc iml;
use one;
read all var _CHAR_ into y[colname=cNames];
close one;
c = length(y`);
k = cnames`||char(c[,<>]);
print k;
quit;

Thanks in Advance,

Venkat.

Re: Splitting a matrix

Rick_SAS — Wed, 10 Jan 2018 11:29:31 GMT

I'm on my way out the door, but look at the following links for dealing with big matrices:

Re: Splitting a matrix

IanWakeling — Wed, 10 Jan 2018 12:00:06 GMT

I think you are nearly running out of memory when you create y`, and then when you try to create c that is the final straw. Since c needs less space than y`, I think it should work if you avoid forming y`, For example:

proc iml;
use one;
read all var _CHAR_ into y[colname=cNames];
close one;
c = length(y);
print cnames;
k = t( cnames//char(c[<>,]) );
print k;
quit;

Re: Splitting a matrix

Rick_SAS — Wed, 10 Jan 2018 13:46:25 GMT

The size of y will depend on the length of the character variables in the data set. A variable with length=256 takes twice as much RAM as one that has length=128.

Look at the MEMSIZE option (second link) to allocate more memory. Since you are computing a maximum, you can also break the data into chunks, as shown in the third link. To build on Ian's program:


proc iml;
use one nObs N;
chunkSize = N/ 10;
max = .;
do data;
   read next (chunkSize) var _CHAR_ into y[colname=cNames];
   c = length(y);
   if max=. then max = c[<>, ];
   else max = (max // c[<>, ])[<>, ];
end;
close one;

k = t( cnames// char(max) );
print k;