memory allocation with matrix operations

[jan_t_lagen]

This could be something that is trivial or trivial to look up but my experience with with the subject is lacking to the point that i don't know where i would start so i'm hoping someone could provide information on where to look (if not a straightforward answer).

I have an issue with insufficient memory when i'm trying to execute my code. The way i understand it this makes no sense since i have a loop with temporary variables  and the problem starts when the number of iterations in said loop are to high. In other words the code works for a low number of iterations but not for a large number.

The only explanation i can think of is that the vector that i save the answers in is the same size as the number of iterations but compared to the rest of the calculations i would think this to be negligible in size, especially for such small numbers that i'm sending in.  

Below i show the code i'm using. I'm not sure how much it will help but what i'm looking for is an answer why this code would work for 5000 iterations but not 10 000; which i know is specific to what i'm using but i'm hoping that there is a more general answer as to why the memory runs out.  

m = 10000; /*size of port.*/

S = 13;

X_I = j(m,S,0);
do j=1 to m;
X_I[j,sector_id[j]] = 1;
end;

n = 5000;
norm1 = j(S,1);
norm2 = j(m,1);
L = j(n,(S+1));

do j=1 to n;

norm1 = RANDNORMAL(S, 0, 1); /*sector factors*/
norm2 = RANDNORMAL(m, 0, 1); /*idosyncratic factors*/

X_vector = (a_inter*norm1); /*vector with shock of each sector*/
X_s = X_I*X_vector; /*Only keep the right sector for each exposure*/

A = p1#X_s+p2#norm2; /*Simulated asset value*/
Ind = A < b; /*Indicator function*/

L_k = w#LGD#Ind; /*simulated loss of iteration j*/
L[j,1] = sum(L_k);

tmp = REPEAT(L_k,1,S);
L_i = X_I#(tmp);
L[j,2:(S+1)] = L_i[+,];

end;

[Rick_SAS · Accepted solution]

1. The SAS log should say which statement is failing. Please post the portion of the log that reports the error.

2. On what iteration does the allocation fail? To find out, submit  PRINT J; after the error occurs.

3. What is your operating system? If Windows, 32-bit or 64-bit? What is your version of SAS?

Have you tried increasing the memory of your SAS process, as described in the article "Large matrices in SAS/IML"?

Incidentally, the RANDNORMAL function is for generating multivariate normal data. It shouldn't affect your program, but it is more efficient to generate univariate normal vectors by using RANDGEN, like this:

call randgen(norm1, "Normal", 0, 1); /*sector factors*/
call randgen(norm2, "Normal", 0, 1); /*idosyncratic factors*/

[jan_t_lagen]

Thanks for the tip, i'll use RANDGEN from now on!

It fails on the repeat of L_k. This is what the log says:

ERROR: (execution) Unable to allocate sufficient memory.

operation : REPEAT at line 139 column 16
operands : L_k, *LIT1052, S
L_k 10000 rows 1 col (numeric)

5 The SAS System 14:41 Monday, March 26, 2018

*LIT1052 1 row 1 col (numeric)

1

S 1 row 1 col (numeric)

13

Like i said before this confuses me since this variable is rewritten in each loop so it should not take up more memory at any point (at least i think so).  And I think that the fact that my variable with the answers in (L) contains more information should be negligible.  

This is a minor part of my project so if there is no not a easy fix i think I'll leave it and accept that these calculations cannot be done using my current setup. In that case, thank you for your help!

EDIT:

I don't know why i did not try it before but it seems that X_I#L_k does the same thing as what i'm trying to achieve with the repeat function. It seems to be working using this. although this does not actually answer the question but i guess it is way more efficient to do it this way.

[Rick_SAS]

Could you please answer (2) and (3) so that I can think about the root cause?

Also, run
   proc options option=memsize value; run;

and send the results from the log.

Thanks.

[jan_t_lagen]

2: It fails at random iterations. When I implemented the change I described in my EDIT it seems like I could run it longer before it runs out of memory but it does none the less.

3: I wish I was more tech savvy so that I could answer the question but the only thing I can give you is that I run it on a server and not on my computer locally 

[IanWakeling]

What you say in the EDIT is correct.    X_I#L_k will do exactly the same thing, without the need to form the matrix tmp using the REPEAT function. This is how elementwise multiplication between a matrix and a vector works in IML.

Now that REPEAT is no longer needed, where does IML fail now?

[jan_t_lagen]

It still fails but after more itterations. It fails at the calcualtion of A. Specifically p2#norm2. But i only tried this once so im not sure that it will always fail there. 

[Rick_SAS]

A program usually does not give errors at "random iterations" when the size of the matrices are not changing. Please post the EXACT program that you are running. 

To find your version of SAS, the operating system, and how you are connecting to the server, run the following statements:

%put &=SYSVLONG;
%put &=SYSSCP;
%put &=SYSSCPL;
%put &=_CLIENTAPP;
proc options option=memsize value; run;

Copy and paste the results from the SAS log. For example, the output might look like this:

 26 %put &=SYSVLONG;
SYSVLONG=9.04.01M5P091317
27 %put &=SYSSCP;
SYSSCP=WIN
28 %put &=SYSSCPL;
SYSSCPL=X64_7PRO
29 %put &=_CLIENTAPP;
_CLIENTAPP='SAS Enterprise Guide'

30 proc options option=memsize value; run;

SAS (r) Proprietary Software Release 9.4 TS1M5

Option Value Information For SAS Option MEMSIZE
Value: 12884901888
Scope: SAS Session
How option value set: Config File
Config file name:
C:\PROGRA~1\SASHome\SASFOU~1\9.4\sasv9.cfg