I believe what you are seeing is that parsing and compiling a SAS program requires memory.
Recall that a macro call is a PREPROCESSOR. It generates test, which are usually SAS statements. In the '%macro loop' example, the macro generates a program that contains 300 MILLION statements. It doesn't matter what the statements are, what matters is that there are 300 million statements.
This huge program is then sent to PROC IML for parsing and execution. The parsing is performed in C. Parsing the program results in a (huge!) C structure that represents the program. That structure is then sent to the procedure for execution. If the amount of memory exceeds the MEMSIZE option in your SAS session, SAS will report an out-of-memory error, which is what is happening in your example.
Your example isn't limited to PROC IML. You can see the same behavior in the DATA step if you use macro to generate a program that contains hundreds of millions of lines. I ran the following macro code to generate huge programs in both the DATA step and PROC IML.
options fullstimer;
options nosource;
%macro IMLtest(numStmts);
%put Test IML: numStmts=%sysfunc(putn(&numStmts,comma9.));
proc iml;
a = 1;
%do i=1 %to &numStmts;
b = a;
%end;
quit;
%mend;
%put ----- START IML TESTS -----;
%IMLtest(10000);
%IMLtest(100000);
%IMLtest(500000);
%IMLtest(1000000);
%macro DStest(numStmts);
%put Test DATA Step: numStmts=%sysfunc(putn(&numStmts,comma9.));
data _NULL_;
a = 1;
%do i=1 %to &numStmts;
b = a;
%end;
run;
%mend;
%put ----- START DATA STEP TESTS -----;
%DStest(10000);
%DStest(100000);
%DStest(500000);
%DStest(1000000);
To make the programs comparable, I use 'a=1' in IML and in the DATA step. If 'a' is a vector in IML, then IML will use a little more memory, but it is really the length of the program (known at compile time) that is causing this issue, not the run-time memory.
On my PC version of SAS, the memory usage for each PROC is as follows:
data VizMemory;
length Proc $10;
input Proc numStmts Memory;
label numStmts="Number of Statements" Memory="Memory (k)";
format numStmts Memory comma10.;
datalines;
DATA_STEP 10000 3907
DATA_STEP 100000 37787
DATA_STEP 500000 180044
DATA_STEP 1000000 359641
IML 1000 254
IML 10000 1428
IML 100000 12698
IML 500000 62705
IML 1000000 125271
;
title "Memory Usage to Parse and Compile SAS Programs";
proc sgplot data=VizMemory;
series x=numStmts y=Memory / group=Proc markers;
xaxis grid;
yaxis grid;
run;
The memory usage scales linearly with the number of SAS statements in the programs. The DATA step actually uses more memory than PROC IML to parse and execute a similar program.
So, in conclusion, you are seeing the result of generating a program that is hundreds of millions of lines long. Those text statements are parsed, which converts the text to a C structure, which requires considerable memory. (Think about a linked list that has 300 million items, and each item has information about the statement and the symbols.) My advice is to use the loops and conditional logic in IML to write your program, rather than using the macro preprocessor to generate repeated statements.
... View more
