Statistical programming, matrix languages, and more

Splitting a matrix

Reply
New Contributor
Posts: 4

Splitting a matrix

Dear All,

 

The below code is not executing successfully as the matrix size is large. The program is ending up with the below error

 

ERROR: (execution) Unable to allocate sufficient memory.

operation : LENGTH at line 1395 column 11
operands : _TEM1001
_TEM1001 33 rows 400000 cols (character, size 80)

 

Is there anyway to find and split the matrix based on the size?

 

dm "output" clear;
data one;
array cha{*} $18. c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 c11 c12 c13 c14 c15 c16 c17 c18 c19 c20 c21 c22 c23 c24 c25 c26 c27 c28 c29 c30 c31 c32 c33;
do j = 1 to 400000;
seed = 100;
do i = 1 to hbound(cha);
rannum = ranpoi(seed,50);
if rannum = 0 then rannum = 1;
cha{i} = repeat("a",rannum);
end;
output;
end;
run;

proc iml;
use one;
read all var _CHAR_ into y[colname=cNames];
close one;
c = length(y`);
k = cnames`||char(c[,<>]);
print k;
quit;

 

Thanks in Advance,

 

Venkat.

SAS Super FREQ
Posts: 4,274

Re: Splitting a matrix

Posted in reply to Venkibhu1

I'm on my way out the door, but look at the following links for dealing with big matrices: 

 

Regular Contributor
Posts: 168

Re: Splitting a matrix

I think you are nearly running out of memory when you create y`, and then when you try to create c that is the final straw.  Since c needs less space than y`, I think it should work if you avoid forming y`,   For example:

 

proc iml;
use one;
read all var _CHAR_ into y[colname=cNames];
close one;
c = length(y);
print cnames;
k = t( cnames//char(c[<>,]) );
print k;
quit;
SAS Super FREQ
Posts: 4,274

Re: Splitting a matrix

Posted in reply to IanWakeling

The size of y will depend on the length of the character variables in the data set.  A variable with length=256 takes twice as much RAM as one that has length=128. 

 

Look at the MEMSIZE option (second link) to allocate more memory. Since you are computing a maximum, you can also break the data into chunks, as shown in the third link. To build on Ian's program:

 


proc iml;
use one nObs N;
chunkSize = N/ 10;
max = .;
do data;
   read next (chunkSize) var _CHAR_ into y[colname=cNames];
   c = length(y);
   if max=. then max = c[<>, ];
   else max = (max // c[<>, ])[<>, ];
end;
close one;

k = t( cnames// char(max) );
print k;

 

 

Ask a Question
Discussion stats
  • 3 replies
  • 309 views
  • 4 likes
  • 3 in conversation